DiffusionGemmaBackbone classkeras_hub.models.DiffusionGemmaBackbone(
vocabulary_size,
image_size,
num_layers,
num_query_heads,
num_key_value_heads,
hidden_dim,
intermediate_dim,
head_dim,
query_head_dim_normalize=True,
attention_logit_soft_cap=None,
final_logit_soft_cap=None,
use_sliding_window_attention=True,
sliding_window_size=512,
sliding_window_pattern=6,
layer_types=None,
global_head_dim=None,
local_rope_scaling_factor=1.0,
global_rope_scaling_factor=1.0,
vision_encoder=None,
layer_norm_epsilon=1e-06,
use_bidirectional_attention=False,
use_vision_bidirectional_attention=False,
dropout=0,
num_global_key_value_heads=None,
global_rope_wavelength=None,
local_rope_wavelength=None,
global_rope_partial_rotary_factor=1.0,
enable_moe_block=False,
num_experts=None,
expert_intermediate_dim=None,
num_experts_per_token=8,
dtype=None,
**kwargs
)
DiffusionGemma core network with hyperparameters.
This backbone implements the DiffusionGemma model architecture. DiffusionGemma extends Gemma4 with discrete block-diffusion generation support: the backbone is called twice per denoising iteration — once as a causal encoder to encode the prompt KV caches, and then repeatedly as a bidirectional decoder over a fixed-length canvas of tokens.
Two diffusion-specific features are always enabled:
encoder_layer_scalar — each decoder block has a second
non-trainable scalar used during the causal encoder pass. The standard
layer_scalar is used for decoder passes.DiffusionGemmaSelfConditioning — refines canvas embeddings at the
start of each denoising step using the logits predicted in the previous
step.In all other respects the architecture matches Gemma4Backbone. Shared
Gemma4 layer classes (norms, vision encoder, etc.) are
imported directly from keras_hub.models.gemma4.
The default constructor gives a fully customised, randomly initialised
DiffusionGemmaBackbone. To load preset weights use from_preset.
Arguments
keras_hub.layers.Gemma4ImageConverter before data
reaches the backbone. The vision_encoder has its own image_size
parameter that controls position embedding sizes.True normalise query pre-attention
using head_dim; otherwise use hidden_dim / num_query_heads.
Unused in DiffusionGemma (always Q-normalised via q_norm).
Kept for API compatibility. Defaults to True.None or float. Tanh soft-cap on attention
logits. Defaults to None.None or float. Tanh soft-cap on output logits.
Defaults to None.True.512.6.None. Explicit specification of the
attention type for every layer sequentially
(e.g. "full_attention", "sliding_attention"). When None,
type sequence is derived from sliding_window_pattern.
Defaults to None.None. Per-head dimension used specifically
for global attention layers. When None, head_dim is used
for all layers. Defaults to None.1.0.1.0.keras_hub.models.Gemma4VisionEncoder or None. When
None the model processes no images.1e-6.True the model uses fully
bidirectional attention for ALL tokens. Defaults to False.True, image tokens
attend to each other bidirectionally during the causal encoder
pass. Defaults to False.0.None. When set, global attention
layers use this many K/V heads instead of num_key_value_heads
and enable the K=V projection. Defaults to None.None. Base RoPE wavelength for
global attention layers. When None, defaults to 1_000_000.0.
Defaults to None.None. Base RoPE wavelength for
local (sliding-window) attention layers. When None, defaults to
10_000.0. Defaults to None.int(factor * head_dim) dimensions are rotated; the remainder are
left unchanged (NoPE). Local layers always use full RoPE
(factor = 1.0). Defaults to 1.0.True, every decoder layer runs a
parallel Mixture-of-Experts path alongside the dense FFW path.
The two outputs are summed before the shared post-FFW norm.
Requires num_experts and expert_intermediate_dim to be set.
Defaults to False.None. Total number of expert MLPs in the MoE
bank. Required when enable_moe_block=True. Defaults to None.None. Intermediate dimension of each
expert MLP. Required when enable_moe_block=True.
Defaults to None.8.keras.mixed_precision.DTypePolicy. Compute dtype.
Defaults to None.Examples
model = keras_hub.models.DiffusionGemmaBackbone.from_preset(
"diffusion_gemma_26b_a4b_it"
)
from_preset methodDiffusionGemmaBackbone.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Backbone from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as a
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''modelscope://user/bert_base_en''./bert_base_en'This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Backbone.from_preset(), or from
a model class like keras_hub.models.GemmaBackbone.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
For any Backbone subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
"gemma_2b_en",
)
# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
"bert_base_en",
load_weights=False,
)
| Preset | Parameters | Description |
|---|---|---|
| diffusion_gemma_26b_a4b_it | 25.82B | DiffusionGemma 26B MoE instruction-tuned model. 25.8B total parameters (3.8B active), 30 layers, 128 experts. |
token_embedding propertykeras_hub.models.DiffusionGemmaBackbone.token_embedding
A keras.layers.Embedding instance for embedding token ids.
This layer embeds integer token ids to the hidden dim of the model.
enable_lora methodDiffusionGemmaBackbone.enable_lora(rank, target_layer_names=None)
Enable Lora on the backbone.
Calling this method will freeze all weights on the backbone,
while enabling Lora on the query & value EinsumDense layers
of the attention layers.
Arguments
None, this will be populated with the
default LoRA layer names as returned by
backbone.default_lora_layer_names().