ModernBertTokenizer classkeras_hub.tokenizers.ModernBertTokenizer(vocabulary=None, merges=None, **kwargs)
ModernBERT byte-level BPE tokenizer.
This tokenizer configures the special token defaults required for the ModernBERT architecture, mapping padding, mask, and sequence boundary tokens to their specific unsplittable representations.
Whitespace immediately preceding [MASK] is absorbed into the token, to
match HuggingFace, which declares it as AddedToken(..., lstrip=True).
Arguments
None.BytePairTokenizer class.Examples
import keras_hub
# Load tokenizer directly from a preset configuration
tokenizer = keras_hub.models.ModernBertTokenizer.from_preset(
"modern_bert_base_en"
)
# Encode raw text strings to integer ID tokens
token_ids = tokenizer("The quick brown fox.")
from_preset methodModernBertTokenizer.from_preset(preset, config_file="tokenizer.json", **kwargs)
Instantiate a keras_hub.models.Tokenizer from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''./bert_base_en'For any Tokenizer subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Tokenizer.from_preset(), or from
a model class like keras_hub.models.GemmaTokenizer.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a preset tokenizer.
tokenizer = keras_hub.tokenizer.Tokenizer.from_preset("bert_base_en")
# Tokenize some input.
tokenizer("The quick brown fox tripped.")
# Detokenize some input.
tokenizer.detokenize([5, 6, 7, 8, 9])
| Preset | Parameters | Description |
|---|---|---|
| modern_bert_base_en | 149.01M | 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |
| modern_bert_embed_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Nomic AI (nomic-ai/modernbert-embed-base). Produces 768-dimensional, L2-normalized, mean-pooled embeddings for inputs of up to 8192 tokens. Prefix queries with 'search_query: ' and documents with 'search_document: '. |
| gte_modern_bert_base | 149.01M | 22-layer ModernBERT Base sentence embedding model from Alibaba-NLP (Alibaba-NLP/gte-modernbert-base). Produces 768-dimensional [CLS]-pooled embeddings that are not normalized, so compare them with cosine similarity. The model supports inputs of up to 8192 tokens. |
| modern_bert_large_en | 394.78M | 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers. |