Keras 3 API documentation / KerasNLP / Models / DistilBert / DistilBertMaskedLM model

DistilBertMaskedLM model


DistilBertMaskedLM class

keras_nlp.models.DistilBertMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end DistilBERT model for the masked language modeling task.

This model will train DistilBERT on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() constructor.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.


Example usage:

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model.
masked_lm = keras_nlp.models.DistilBertMaskedLM.from_preset(
), batch_size=2)

# Re-compile (e.g., with a new learning rate).
# Access backbone programmatically (e.g., to change `trainable`).
masked_lm.backbone.trainable = False
# Fit again., batch_size=2)

Preprocessed integer data.

# Create preprocessed batch where 0 is the mask token.
features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1]] * 2),
    "mask_positions": np.array([[2, 4]] * 2)
# Labels are the original masked values.
labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.DistilBertMaskedLM.from_preset(
), y=labels, batch_size=2)


from_preset method


Instantiate DistilBertMaskedLM model from preset architecture and weights.


  • preset: string. Must be one of "distil_bert_base_en_uncased", "distil_bert_base_en", "distil_bert_base_multi".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.


# Load architecture and weights from preset
model = DistilBertMaskedLM.from_preset("distil_bert_base_en_uncased")

# Load randomly initialized model from preset architecture
model = DistilBertMaskedLM.from_preset(
Preset name Parameters Description
distil_bert_base_en_uncased 66.36M 6-layer DistilBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus using BERT as the teacher model.
distil_bert_base_en 65.19M 6-layer DistilBERT model where case is maintained. Trained on English Wikipedia + BooksCorpus using BERT as the teacher model.
distil_bert_base_multi 134.73M 6-layer DistilBERT model where case is maintained. Trained on Wikipedias of 104 languages

backbone property


A keras.Model instance providing the backbone sub-model.

preprocessor property


A keras.layers.Layer instance used to preprocess inputs.