Keras 3 API documentation / KerasNLP / Models / Albert / AlbertMaskedLM model

AlbertMaskedLM model

[source]

AlbertMaskedLM class

keras_nlp.models.AlbertMaskedLM(backbone, preprocessor=None, **kwargs)

An end-to-end ALBERT model for the masked language modeling task.

This model will train ALBERT on a masked language modeling task. The model will predict labels for a number of masked tokens in the input data. For usage of this model with pre-trained weights, see the from_preset() method.

This model can optionally be configured with a preprocessor layer, in which case inputs can be raw string features during fit(), predict(), and evaluate(). Inputs will be tokenized and dynamically masked during training and evaluation. This is done by default when creating the model with from_preset().

Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind.

Arguments

Example usage:

Raw string data.

features = ["The quick brown fox jumped.", "I forgot my homework."]

# Pretrained language model.
masked_lm = keras_nlp.models.AlbertMaskedLM.from_preset(
    "albert_base_en_uncased",
)
masked_lm.fit(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
masked_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
masked_lm.backbone.trainable = False
# Fit again.
masked_lm.fit(x=features, batch_size=2)

Preprocessed integer data.

# Create preprocessed batch where 0 is the mask token.
features = {
    "token_ids": np.array([[1, 2, 0, 4, 0, 6, 7, 8]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1]] * 2),
    "mask_positions": np.array([[2, 4]] * 2),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 0, 0, 0]] * 2),
}
# Labels are the original masked values.
labels = [[3, 5]] * 2

masked_lm = keras_nlp.models.AlbertMaskedLM.from_preset(
    "albert_base_en_uncased",
    preprocessor=None,
)
masked_lm.fit(x=features, y=labels, batch_size=2)

[source]

from_preset method

AlbertMaskedLM.from_preset()

Instantiate AlbertMaskedLM model from preset architecture and weights.

Arguments

  • preset: string. Must be one of "albert_base_en_uncased", "albert_large_en_uncased", "albert_extra_large_en_uncased", "albert_extra_extra_large_en_uncased".
  • load_weights: Whether to load pre-trained weights into model. Defaults to True.

Examples

# Load architecture and weights from preset
model = AlbertMaskedLM.from_preset("albert_base_en_uncased")

# Load randomly initialized model from preset architecture
model = AlbertMaskedLM.from_preset(
    "albert_base_en_uncased",
    load_weights=False
)
Preset name Parameters Description
albert_base_en_uncased 11.68M 12-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus.
albert_large_en_uncased 17.68M 24-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus.
albert_extra_large_en_uncased 58.72M 24-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus.
albert_extra_extra_large_en_uncased 222.60M 12-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus.

backbone property

keras_nlp.models.AlbertMaskedLM.backbone

A keras.Model instance providing the backbone sub-model.


preprocessor property

keras_nlp.models.AlbertMaskedLM.preprocessor

A keras.layers.Layer instance used to preprocess inputs.