Keras 3 API documentation / KerasNLP / Models / XLMRoberta / XLMRobertaMaskedLM model

XLMRobertaMaskedLM model

[source]

XLMRobertaMaskedLM class

keras_nlp.models.XLMRobertaMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end XLM-RoBERTa model for the masked language modeling task.

This model will train XLM-RoBERTa on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() method.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.

Arguments

Example usage:

Raw string inputs and pretrained backbone.

# Create a dataset with raw string features. Labels are inferred.
features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model
# on an MLM task.
masked_lm = keras_nlp.models.XLMRobertaMaskedLM.from_preset(
    "xlm_roberta_base_multi",
)
masked_lm.fit(x=features, batch_size=2)

Re-compile (e.g., with a new learning rate)

.

masked_lm.compile( loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), optimizer=keras.optimizers.Adam(5e-5), jit_compile=True, ) Access backbone programmatically (e.g., to change trainable)

.

masked_lm.backbone.trainable = False Fit again

.

masked_lm.fit(x=features, batch_size=2)

Preprocessed integer data.
```python
__Create a preprocessed dataset where 0 is the mask token__

.

features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1]] * 2),
    "mask_positions": np.array([[2, 4]] * 2)
}
__Labels are the original masked values__

.

labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.XLMRobertaMaskedLM.from_preset(
    "xlm_roberta_base_multi",
    preprocessor=None,
)

masked_lm.fit(x=features, y=labels, batch_size=2)

[source]

from_preset method

XLMRobertaMaskedLM.from_preset()

Instantiate XLMRobertaMaskedLM model from preset architecture and weights.

Arguments

  • preset: string. Must be one of "xlm_roberta_base_multi", "xlm_roberta_large_multi".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.

Examples

# Load architecture and weights from preset
model = XLMRobertaMaskedLM.from_preset("xlm_roberta_base_multi")

# Load randomly initialized model from preset architecture
model = XLMRobertaMaskedLM.from_preset(
    "xlm_roberta_base_multi",
    load_weights=False
)
Preset name Parameters Description
xlm_roberta_base_multi 277.45M 12-layer XLM-RoBERTa model where case is maintained. Trained on CommonCrawl in 100 languages.
xlm_roberta_large_multi 558.84M 24-layer XLM-RoBERTa model where case is maintained. Trained on CommonCrawl in 100 languages.

backbone property

keras_nlp.models.XLMRobertaMaskedLM.backbone

A keras.Model instance providing the backbone sub-model.


preprocessor property

keras_nlp.models.XLMRobertaMaskedLM.preprocessor

A keras.layers.Layer instance used to preprocess inputs.