EmbeddingGemma2Tokenizer classkeras_hub.tokenizers.EmbeddingGemma2Tokenizer(
proto, has_vision_tokens=True, has_audio_tokens=False, has_video_tokens=True, **kwargs
)
EmbeddingGemma2 tokenizer.
This tokenizer is a subclass of Gemma4Tokenizer and shares the same
vocabulary and functionality.
Arguments
string path to a SentencePiece proto file, or a
bytes object with a serialized SentencePiece proto. See the
SentencePiece repository
for more details on the format.Examples
# Unbatched input.
tokenizer = keras_hub.models.EmbeddingGemma2Tokenizer.from_preset(
"embedding_gemma2"
)
tokenizer("The quick brown fox jumped.")
# Detokenization.
tokenizer.detokenize(tokenizer("The quick brown fox jumped."))
from_preset methodEmbeddingGemma2Tokenizer.from_preset(preset, config_file="tokenizer.json", **kwargs)
Instantiate a keras_hub.models.Tokenizer from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''./bert_base_en'For any Tokenizer subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Tokenizer.from_preset(), or from
a model class like keras_hub.models.GemmaTokenizer.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a preset tokenizer.
tokenizer = keras_hub.tokenizer.Tokenizer.from_preset("bert_base_en")
# Tokenize some input.
tokenizer("The quick brown fox tripped.")
# Detokenize some input.
tokenizer.detokenize([5, 6, 7, 8, 9])
| Preset | Parameters | Description |
|---|---|---|
| embedding_gemma2 | 744.37M | EmbeddingGemma 2: 744M-parameter, 24-layer multimodal embedding model built on Gemma 4. Embeds text, images, video and audio into a 768-dimensional space. |