ModernBertMaskedLM model

[source]

ModernBertMaskedLM class

keras_hub.models.ModernBertMaskedLM(
    backbone, preprocessor=None, dtype=None, **kwargs
)

ModernBERT Masked LM task model.

The Masked LM model provides a prediction head for the Masked Language Modeling task. It is composed of a keras_hub.models.ModernBertBackbone and a prediction head which projects the backbone's hidden states back to the vocabulary space.

This model can be used for pre-training or fine-tuning on a specific corpus.

Arguments

Examples

import keras_hub
import numpy as np

# Pre-trained backbone and preprocessor
tokenizer = keras_hub.models.ModernBertTokenizer(
    vocabulary="vocab.json",
    merges="merges.txt",
)
preprocessor = keras_hub.models.ModernBertMaskedLMPreprocessor(
    tokenizer=tokenizer,
    sequence_length=128,
)
backbone = keras_hub.models.ModernBertBackbone(
    vocabulary_size=50368,
    hidden_dim=768,
    intermediate_dim=1152,
    num_layers=22,
    num_heads=12,
)

# Instantiate the MaskedLM task model
masked_lm = keras_hub.models.ModernBertMaskedLM(
    backbone=backbone,
    preprocessor=preprocessor,
)

# Predict on raw text strings
raw_data = ["The quick brown fox [MASK] over the lazy dog."]
predictions = masked_lm.predict(raw_data)

[source]

from_preset method

ModernBertMaskedLM.from_preset(preset, load_weights=True, **kwargs)

Instantiate a keras_hub.models.Task from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

  1. a built-in preset identifier like 'bert_base_en'
  2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
  3. a Hugging Face handle like 'hf://user/bert_base_en'
  4. a path to a local preset directory like './bert_base_en'

For any Task subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

This constructor can be called in one of two ways. Either from a task specific base class like keras_hub.models.CausalLM.from_preset(), or from a model class like keras_hub.models.BertTextClassifier.from_preset(). If calling from the a base class, the subclass of the returning object will be inferred from the config in the preset directory.

Arguments

  • preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
  • load_weights: bool. If True, saved weights will be loaded into the model architecture. If False, all weights will be randomly initialized.

Examples

# Load a Gemma generative task.
causal_lm = keras_hub.models.CausalLM.from_preset(
    "gemma_2b_en",
)

# Load a Bert classification task.
model = keras_hub.models.TextClassifier.from_preset(
    "bert_base_en",
    num_classes=2,
)
Preset Parameters Description
modern_bert_base_en 149.01M 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers.
modern_bert_embed_base 149.01M 22-layer ModernBERT Base sentence embedding model from Nomic AI (nomic-ai/modernbert-embed-base). Produces 768-dimensional, L2-normalized, mean-pooled embeddings for inputs of up to 8192 tokens. Prefix queries with 'search_query: ' and documents with 'search_document: '.
gte_modern_bert_base 149.01M 22-layer ModernBERT Base sentence embedding model from Alibaba-NLP (Alibaba-NLP/gte-modernbert-base). Produces 768-dimensional [CLS]-pooled embeddings that are not normalized, so compare them with cosine similarity. The model supports inputs of up to 8192 tokens.
modern_bert_large_en 394.78M 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers.

backbone property

keras_hub.models.ModernBertMaskedLM.backbone

A keras_hub.models.Backbone model with the core architecture.


preprocessor property

keras_hub.models.ModernBertMaskedLM.preprocessor

A keras_hub.models.Preprocessor layer used to preprocess input.