Keras 3 API documentation / KerasNLP / Models / XLMRoberta / XLMRobertaMaskedLM model

XLMRobertaMaskedLM model


XLMRobertaMaskedLM class

keras_nlp.models.XLMRobertaMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end XLM-RoBERTa model for the masked language modeling task.

This model will train XLM-RoBERTa on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() method.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.


Example usage:

Raw string inputs and pretrained backbone.

# Create a dataset with raw string features. Labels are inferred.
features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model
# on an MLM task.
masked_lm = keras_nlp.models.XLMRobertaMaskedLM.from_preset(
), batch_size=2)

Re-compile (e.g., with a new learning rate)


masked_lm.compile( loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), optimizer=keras.optimizers.Adam(5e-5), jit_compile=True, ) Access backbone programmatically (e.g., to change trainable)


masked_lm.backbone.trainable = False Fit again

., batch_size=2)

Preprocessed integer data.
__Create a preprocessed dataset where 0 is the mask token__


features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1]] * 2),
    "mask_positions": np.array([[2, 4]] * 2)
__Labels are the original masked values__


labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.XLMRobertaMaskedLM.from_preset(
), y=labels, batch_size=2)


from_preset method


Instantiate XLMRobertaMaskedLM model from preset architecture and weights.


  • preset: string. Must be one of "xlm_roberta_base_multi", "xlm_roberta_large_multi".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.


# Load architecture and weights from preset
model = XLMRobertaMaskedLM.from_preset("xlm_roberta_base_multi")

# Load randomly initialized model from preset architecture
model = XLMRobertaMaskedLM.from_preset(
Preset name Parameters Description
xlm_roberta_base_multi 277.45M 12-layer XLM-RoBERTa model where case is maintained. Trained on CommonCrawl in 100 languages.
xlm_roberta_large_multi 558.84M 24-layer XLM-RoBERTa model where case is maintained. Trained on CommonCrawl in 100 languages.

backbone property


A keras.Model instance providing the backbone sub-model.

preprocessor property


A keras.layers.Layer instance used to preprocess inputs.