EmbeddingGemma2Backbone model

[source]

EmbeddingGemma2Backbone class

keras_hub.models.EmbeddingGemma2Backbone(
    vocabulary_size,
    image_size,
    num_layers,
    num_query_heads,
    num_key_value_heads,
    hidden_dim,
    intermediate_dim,
    head_dim,
    use_sliding_window_attention=True,
    sliding_window_size=512,
    sliding_window_pattern=6,
    layer_types=None,
    global_head_dim=None,
    local_rope_scaling_factor=1.0,
    global_rope_scaling_factor=1.0,
    vision_encoder=None,
    audio_encoder=None,
    layer_norm_epsilon=1e-06,
    dropout=0,
    embedding_dim=None,
    num_global_key_value_heads=None,
    hidden_size_per_layer_input=0,
    global_rope_wavelength=None,
    local_rope_wavelength=None,
    global_rope_partial_rotary_factor=1.0,
    dtype=None,
    **kwargs
)

EmbeddingGemma 2 core network with hyperparameters.

This backbone implements the EmbeddingGemma 2 encoder: the Gemma 4 text stack run with fully bidirectional attention, followed by a dense embedding_projection applied to every token's final hidden state. Mean pooling and L2 normalisation live in keras_hub.models.EmbeddingGemma2TextEmbedder.

Images, video frames and audio clips are encoded by the optional vision_encoder (Gemma4VisionEncoder) and audio_encoder (Gemma4AudioEncoder); their soft tokens are interleaved into the text embedding sequence at the placeholder positions before the encoder blocks run.

The default constructor gives a fully customised, randomly initialised model. To load preset weights use the from_preset constructor.

Arguments

  • vocabulary_size: int. The size of the token vocabulary.
  • image_size: int. The spatial resolution of images (height = width). Stored as a config value for serialization purposes only; it does not affect the backbone's forward pass. Image patching and resizing are handled by keras_hub.layers.EmbeddingGemma2ImageConverter before data reaches the backbone. The vision_encoder has its own image_size parameter that controls position embedding sizes.
  • num_layers: int. Number of transformer decoder layers.
  • num_query_heads: int. Number of query heads per attention layer.
  • num_key_value_heads: int. Number of key/value heads (GQA).
  • hidden_dim: int. Hidden state dimension at the end of each layer.
  • intermediate_dim: int. First dense layer output dimension in each FFW sub-block.
  • head_dim: int. Per-head dimension in the encoder attention.
  • use_sliding_window_attention: bool. Whether to use sliding-window attention on the local layers. Defaults to True.
  • sliding_window_size: int. Size of the local attention window. Defaults to 512.
  • sliding_window_pattern: int. Repeat period of the local/global attention pattern. The last layer in each group of this many consecutive layers uses global attention; all others use local (sliding-window) attention. Defaults to 6.
  • layer_types: list of str or None. Explicit specification of the attention type for every layer sequentially (e.g. "full_attention", "sliding_attention"). When None, type sequence is derived from sliding_window_pattern. Defaults to None.
  • global_head_dim: int or None. Per-head dimension used specifically for global attention layers. When None, head_dim is used for all layers. Defaults to None.
  • local_rope_scaling_factor: float. RoPE scaling factor for local layers. Defaults to 1.0.
  • global_rope_scaling_factor: float. RoPE scaling factor for global layers. Defaults to 1.0.
  • global_rope_partial_rotary_factor: float. Fraction of each head dimension that receives rotary position embeddings in global attention layers. Only the first int(factor * head_dim) dimensions are rotated; the remainder are left unchanged (NoPE). Local layers always use full RoPE (factor = 1.0). Defaults to 1.0.
  • vision_encoder: keras_hub.models.Gemma4VisionEncoder or None. When None the model processes no images.
  • audio_encoder: keras_hub.models.Gemma4AudioEncoder or None. When None the model processes no audio.
  • layer_norm_epsilon: float. Epsilon for all RMS norms. Defaults 1e-6.
  • dropout: float. Dropout probability. Defaults to 0.
  • embedding_dim: int. Output dimension of the final dense embedding_projection applied to every token's hidden state.
  • num_global_key_value_heads: int or None. When set, global attention layers use this many K/V heads instead of num_key_value_heads (separate K and V projections are kept). Defaults to None.
  • hidden_size_per_layer_input: int. Size of the per-token, per-layer conditioning vector that gates each decoder layer's output. Set to 0 to disable. Defaults to 0.
  • __ dtype__: string or keras.mixed_precision.DTypePolicy. Compute dtype. Defaults to None.

Examples

# Load a pretrained EmbeddingGemma 2 backbone.
model = keras_hub.models.EmbeddingGemma2Backbone.from_preset(
    "embedding_gemma2"
)

# Randomly initialised text-only backbone with a custom config.
model = keras_hub.models.EmbeddingGemma2Backbone(
    vocabulary_size=262144,
    image_size=896,
    num_layers=24,
    num_query_heads=4,
    num_key_value_heads=2,
    hidden_dim=512,
    intermediate_dim=2048,
    head_dim=256,
    embedding_dim=768,
    dtype="float32",
)

[source]

from_preset method

EmbeddingGemma2Backbone.from_preset(preset, load_weights=True, **kwargs)

Instantiate a keras_hub.models.Backbone from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as a one of:

  1. a built-in preset identifier like 'bert_base_en'
  2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
  3. a Hugging Face handle like 'hf://user/bert_base_en'
  4. a ModelScope handle like 'modelscope://user/bert_base_en'
  5. a path to a local preset directory like './bert_base_en'

This constructor can be called in one of two ways. Either from the base class like keras_hub.models.Backbone.from_preset(), or from a model class like keras_hub.models.GemmaBackbone.from_preset(). If calling from the base class, the subclass of the returning object will be inferred from the config in the preset directory.

For any Backbone subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

Arguments

  • preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
  • load_weights: bool. If True, the weights will be loaded into the model architecture. If False, the weights will be randomly initialized.

Examples

# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
    "gemma_2b_en",
)

# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
    "bert_base_en",
    load_weights=False,
)
Preset Parameters Description
embedding_gemma2 744.37M EmbeddingGemma 2: 744M-parameter, 24-layer multimodal embedding model built on Gemma 4. Embeds text, images, video and audio into a 768-dimensional space.

token_embedding property

keras_hub.models.EmbeddingGemma2Backbone.token_embedding

A keras.layers.Embedding instance for embedding token ids.

This layer embeds integer token ids to the hidden dim of the model.


[source]

enable_lora method

EmbeddingGemma2Backbone.enable_lora(rank, target_layer_names=None)

Enable Lora on the backbone.

Calling this method will freeze all weights on the backbone, while enabling Lora on the query & value EinsumDense layers of the attention layers.

Arguments

  • rank: The rank of the LoRA factorization.
  • target_layer_names: A list of strings, the names of the layers to apply LoRA to. If None, this will be populated with the default LoRA layer names as returned by backbone.default_lora_layer_names().