ModernBertMaskedLMPreprocessor classkeras_hub.models.ModernBertMaskedLMPreprocessor(
tokenizer,
sequence_length=512,
truncate="round_robin",
mask_selection_rate=0.15,
mask_selection_length=96,
mask_token_rate=0.8,
random_token_rate=0.1,
**kwargs
)
ModernBERT Masked LM preprocessor.
This preprocessor tokenizes and prepares inputs for masked language
modeling with ModernBERT. The masking, packing, and serialization logic
is inherited from MaskedLMPreprocessor.
Arguments
keras_hub.models.ModernBertTokenizer instance.MaskedLMPreprocessor.Examples
import keras_hub
preprocessor = keras_hub.models.ModernBertMaskedLMPreprocessor.from_preset(
"modern_bert_base_en"
)
# Tokenize and mask a single sentence.
preprocessor("The quick brown fox jumped.")
# Tokenize and mask a batch of sentences.
preprocessor(["The quick brown fox jumped.", "Call me Ishmael."])
# Use in a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset).
import tensorflow as tf
features = ["The quick brown fox jumped.", "Call me Ishmael."]
ds = tf.data.Dataset.from_tensor_slices(features)
ds = ds.map(preprocessor, num_parallel_calls=tf.data.AUTOTUNE)
from_preset methodModernBertMaskedLMPreprocessor.from_preset(
preset, config_file="preprocessor.json", **kwargs
)
Instantiate a keras_hub.models.Preprocessor from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''./bert_base_en'For any Preprocessor subclass, you can run cls.presets.keys() to
list all built-in presets available on the class.
As there are usually multiple preprocessing classes for a given model,
this method should be called on a specific subclass like
keras_hub.models.BertTextClassifierPreprocessor.from_preset().
Arguments
Examples
# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.CausalLMPreprocessor.from_preset(
"gemma_2b_en",
)
# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.TextClassifierPreprocessor.from_preset(
"bert_base_en",
)
| Preset | Parameters | Description |
|---|---|---|
| modern_bert_base_en | 149.01M | 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |
| modern_bert_embed_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Nomic AI (nomic-ai/modernbert-embed-base). Produces 768-dimensional, L2-normalized, mean-pooled embeddings for inputs of up to 8192 tokens. Prefix queries with 'search_query: ' and documents with 'search_document: '. |
| gte_modern_bert_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Alibaba-NLP (Alibaba-NLP/gte-modernbert-base). Produces 768-dimensional [CLS]-pooled embeddings that are not normalized, so compare them with cosine similarity. The model supports inputs of up to 8192 tokens. |
| modern_bert_large_en | 394.78M | 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |
tokenizer propertykeras_hub.models.ModernBertMaskedLMPreprocessor.tokenizer
The tokenizer used to tokenize strings.