ModernBertTextEmbedderPreprocessor classkeras_hub.models.ModernBertTextEmbedderPreprocessor(
tokenizer, sequence_length=256, truncate="round_robin", **kwargs
)
A ModernBERT preprocessing layer which tokenizes and packs inputs for sentence embedding.
This preprocessing layer will do three things:
tokenizer.keras_hub.layers.MultiSegmentPacker
with the appropriate "[CLS]", "[SEP]" and "[PAD]" tokens, i.e.,
adding a single "[CLS]" at the start of the entire sequence and a
"[SEP]" at the end of each segment. This matches the Hugging Face
ModernBERT template: [CLS] A [SEP] for single inputs and
[CLS] A [SEP] B [SEP] for pairs."token_ids" and "padding_mask"
that can be passed directly to a ModernBERT text embedder. ModernBERT
has no segment embeddings, so no "segment_ids" are returned.This layer can be used directly with tf.data.Dataset.map to preprocess
string data in the (x, y, sample_weight) format used by
keras.Model.fit.
Arguments
keras_hub.models.ModernBertTokenizer instance.256. When loading from a Hugging Face sentence-transformers
checkpoint, this is overridden by the length sentence-transformers
uses: max_seq_length from sentence_bert_config.json (e.g.
8192 for nomic-ai/modernbert-embed-base), or otherwise the
model's maximum length (e.g. 8192 for
Alibaba-NLP/gte-modernbert-base). Every input is padded to this
length, so for faster inference on short texts, pass a smaller
sequence_length to this class's from_preset() or set the
sequence_length attribute after loading.sequence_length. The value can be either
round_robin or waterfall:"round_robin": Available space is assigned one token at a
time in a round-robin fashion to the inputs that still need
some, until the limit is reached."waterfall": The allocation of the budget is done using a
"waterfall" algorithm that allocates quota in a
left-to-right manner and fills up the buckets until we run
out of budget. It supports an arbitrary number of segments.Call arguments
Examples
Directly calling the layer on data.
preprocessor = (
keras_hub.models.ModernBertTextEmbedderPreprocessor.from_preset(
"hf://nomic-ai/modernbert-embed-base",
sequence_length=128,
)
)
# Tokenize and pack a single sentence.
preprocessor("search_query: What is TSNE?")
# Tokenize a batch of single sentences.
preprocessor([
"search_query: What is TSNE?",
"search_document: TSNE is a dimensionality reduction algorithm.",
])
Mapping with tf.data.Dataset.
preprocessor = (
keras_hub.models.ModernBertTextEmbedderPreprocessor.from_preset(
"hf://nomic-ai/modernbert-embed-base",
sequence_length=128,
)
)
first = tf.constant(["The quick brown fox jumped.", "Call me Ishmael."])
# Map unlabeled single sentences.
ds = tf.data.Dataset.from_tensor_slices(first)
ds = ds.map(preprocessor, num_parallel_calls=tf.data.AUTOTUNE)
from_preset methodModernBertTextEmbedderPreprocessor.from_preset(
preset, config_file="preprocessor.json", **kwargs
)
Instantiate a keras_hub.models.Preprocessor from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''./bert_base_en'For any Preprocessor subclass, you can run cls.presets.keys() to
list all built-in presets available on the class.
As there are usually multiple preprocessing classes for a given model,
this method should be called on a specific subclass like
keras_hub.models.BertTextClassifierPreprocessor.from_preset().
Arguments
Examples
# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.CausalLMPreprocessor.from_preset(
"gemma_2b_en",
)
# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.TextClassifierPreprocessor.from_preset(
"bert_base_en",
)
| Preset | Parameters | Description |
|---|---|---|
| modern_bert_base_en | 149.01M | 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |
| modern_bert_embed_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Nomic AI (nomic-ai/modernbert-embed-base). Produces 768-dimensional, L2-normalized, mean-pooled embeddings for inputs of up to 8192 tokens. Prefix queries with 'search_query: ' and documents with 'search_document: '. |
| gte_modern_bert_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Alibaba-NLP (Alibaba-NLP/gte-modernbert-base). Produces 768-dimensional [CLS]-pooled embeddings that are not normalized, so compare them with cosine similarity. The model supports inputs of up to 8192 tokens. |
| modern_bert_large_en | 394.78M | 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |
tokenizer propertykeras_hub.models.ModernBertTextEmbedderPreprocessor.tokenizer
The tokenizer used to tokenize strings.