MaskedLM

[source]

MaskedLM class

keras_nlp.models.MaskedLM()

Base class for masked language modeling tasks.

MaskedLM tasks wrap a keras_nlp.models.Backbone and a keras_nlp.models.Preprocessor to create a model that can be used for unsupervised fine-tuning with a masked language modeling loss.

When calling fit(), all input will be tokenized, and random tokens in the input sequence will be masked. These positions of these masked tokens will be fed as an additional model input, and the original value of the tokens predicted by the model outputs.

All MaskedLM tasks include a from_preset() constructor which can be used to load a pre-trained config and weights.

Example

# Load a Bert MaskedLM with pre-trained weights.
masked_lm = keras_nlp.models.MaskedLM.from_preset(
    "bert_base_en",
)
masked_lm.fit(train_ds)

[source]

from_preset method

MaskedLM.from_preset(preset, load_weights=True, **kwargs)

Instantiate a keras_nlp.models.Task from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as a one of:

1. a built in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Task subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

This constructor can be called in one of two ways. Either from a task specific base class like keras_nlp.models.CausalLM.from_preset(), or from a model class like keras_nlp.models.BertClassifier.from_preset(). If calling from the a base class, the subclass of the returning object will be inferred from the config in the preset directory.

Arguments

• preset: string. A built in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
• load_weights: bool. If True, the weights will be loaded into the model architecture. If False, the weights will be randomly initialized.

Examples

# Load a Gemma generative task.
causal_lm = keras_nlp.models.CausalLM.from_preset(
    "gemma_2b_en",
)

# Load a Bert classification task.
model = keras_nlp.models.Classifier.from_preset(
    "bert_base_en",
    num_classes=2,
)

[source]

compile method

MaskedLM.compile(optimizer="auto", loss="auto", weighted_metrics="auto", **kwargs)

Configures the MaskedLM task for training.

The MaskedLM task extends the default compilation signature of keras.Model.compile with defaults for optimizer, loss, and weighted_metrics. To override these defaults, pass any value to these arguments during compilation.

Note that because training inputs include padded tokens which are excluded from the loss, it is almost always a good idea to compile with weighted_metrics and not metrics.

Arguments

• optimizer: "auto", an optimizer name, or a keras.Optimizer instance. Defaults to "auto", which uses the default optimizer for the given model and task. See keras.Model.compile and keras.optimizers for more info on possible optimizer values.
• loss: "auto", a loss name, or a keras.losses.Loss instance. Defaults to "auto", where a keras.losses.SparseCategoricalCrossentropy loss will be applied for the token classification MaskedLM task. See keras.Model.compile and keras.losses for more info on possible loss values.
• weighted_metrics: "auto", or a list of metrics to be evaluated by the model during training and testing. Defaults to "auto", where a keras.metrics.SparseCategoricalAccuracy will be applied to track the accuracy of the model at guessing masked token values. See keras.Model.compile and keras.metrics for more info on possible weighted_metrics values.
• **kwargs: See keras.Model.compile for a full list of arguments supported by the compile method.

[source]

save_to_preset method

MaskedLM.save_to_preset(preset_dir)

Save task to a preset directory.

Arguments

• preset_dir: The path to the local model preset directory.

preprocessor property

keras_nlp.models.MaskedLM.preprocessor

A keras_nlp.models.Preprocessor layer used to preprocess input.

backbone property

keras_nlp.models.MaskedLM.backbone

A keras_nlp.models.Backbone model with the core architecture.