Keras 3 API documentation / KerasNLP / Models / FNet / FNetMaskedLM model

FNetMaskedLM model

[source]

FNetMaskedLM class

keras_nlp.models.FNetMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end FNet model for the masked language modeling task.

This model will train FNet on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() constructor.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind.

Arguments

Example usage:

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model.
masked_lm = keras_nlp.models.FNetMaskedLM.from_preset(
    "f_net_base_en",
)
masked_lm.fit(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
masked_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
masked_lm.backbone.trainable = False
# Fit again.
masked_lm.fit(x=features, batch_size=2)

Preprocessed integer data.

# Create a preprocessed dataset where 0 is the mask token.
features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "segment_ids": np.array([[0, 0, 0, 1, 1, 1, 0, 0]] * 2),
    "mask_positions": np.array([[2, 4]] * 2)
}
# Labels are the original masked values.
labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.FNetMaskedLM.from_preset(
    "f_net_base_en",
    preprocessor=None,
)
masked_lm.fit(x=features, y=labels, batch_size=2)

[source]

from_preset method

FNetMaskedLM.from_preset()

Instantiate FNetMaskedLM model from preset architecture and weights.

Arguments

  • preset: string. Must be one of "f_net_base_en", "f_net_large_en".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.

Examples

# Load architecture and weights from preset
model = FNetMaskedLM.from_preset("f_net_base_en")

# Load randomly initialized model from preset architecture
model = FNetMaskedLM.from_preset(
    "f_net_base_en",
    load_weights=False
)
Preset name Parameters Description
f_net_base_en 82.86M 12-layer FNet model where case is maintained. Trained on the C4 dataset.
f_net_large_en 236.95M 24-layer FNet model where case is maintained. Trained on the C4 dataset.

backbone property

keras_nlp.models.FNetMaskedLM.backbone

A keras.Model instance providing the backbone sub-model.


preprocessor property

keras_nlp.models.FNetMaskedLM.preprocessor

A keras.layers.Layer instance used to preprocess inputs.