Keras 3 API documentation / KerasNLP / Models / Roberta / RobertaMaskedLM model

RobertaMaskedLM model

[source]

RobertaMaskedLM class

keras_nlp.models.RobertaMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end RoBERTa model for the masked language modeling task.

This model will train RoBERTa on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() method.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.

Arguments

Examples

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model.
masked_lm = keras_nlp.models.RobertaMaskedLM.from_preset(
    "roberta_base_en",
)
masked_lm.fit(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
masked_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
masked_lm.backbone.trainable = False
# Fit again.
masked_lm.fit(x=features, batch_size=2)

Preprocessed integer data.

# Create a preprocessed dataset where 0 is the mask token.
features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1]] * 2),
    "mask_positions": np.array([[2, 4]] * 2)
}
# Labels are the original masked values.
labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.RobertaMaskedLM.from_preset(
    "roberta_base_en",
    preprocessor=None,
)

masked_lm.fit(x=features, y=labels, batch_size=2)

[source]

from_preset method

RobertaMaskedLM.from_preset()

Instantiate RobertaMaskedLM model from preset architecture and weights.

Arguments

  • preset: string. Must be one of "roberta_base_en", "roberta_large_en".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.

Examples

# Load architecture and weights from preset
model = RobertaMaskedLM.from_preset("roberta_base_en")

# Load randomly initialized model from preset architecture
model = RobertaMaskedLM.from_preset(
    "roberta_base_en",
    load_weights=False
)
Preset name Parameters Description
roberta_base_en 124.05M 12-layer RoBERTa model where case is maintained.Trained on English Wikipedia, BooksCorpus, CommonCraw, and OpenWebText.
roberta_large_en 354.31M 24-layer RoBERTa model where case is maintained.Trained on English Wikipedia, BooksCorpus, CommonCraw, and OpenWebText.

backbone property

keras_nlp.models.RobertaMaskedLM.backbone

A keras.Model instance providing the backbone sub-model.


preprocessor property

keras_nlp.models.RobertaMaskedLM.preprocessor

A keras.layers.Layer instance used to preprocess inputs.