SmolVLM2Backbone classkeras_hub.models.SmolVLM2Backbone(
vocabulary_size,
image_size,
patch_size,
vision_hidden_dim,
vision_intermediate_dim,
vision_num_layers,
vision_num_heads,
hidden_dim,
intermediate_dim,
num_layers,
num_query_heads,
num_key_value_heads,
scale_factor,
image_token_id,
rope_max_wavelength=10000,
layer_norm_epsilon=1e-05,
vision_layer_norm_epsilon=1e-06,
tie_word_embeddings=False,
dtype=None,
**kwargs
)
SmolVLM2 multimodal backbone.
This backbone implements the SmolVLM2 architecture, combining a SigLIP-based vision encoder, a pixel-shuffle connector, and a Llama-style text decoder. It can process text-only inputs or multimodal (image + text) inputs.
The vision encoder extracts patch features from images, the
connector compresses and projects them to the text decoder's
hidden dimension, and the interleave_embeddings layer replaces
positions indicated by vision_indices in the text token sequence
with the projected visual embeddings.
For text-only calls, a __call__ override injects zero-sized
dummy vision tensors so the functional graph receives all required
keys. This follows the Qwen3.5/Gemma4 multimodal backbone pattern.
Arguments
10000.1e-5.1e-6.False.keras.mixed_precision.DTypePolicy.Examples
# Text-only forward pass (vision inputs injected automatically).
backbone = keras_hub.models.SmolVLM2Backbone(
vocabulary_size=49280,
image_size=384,
patch_size=14,
vision_hidden_dim=1152,
vision_intermediate_dim=4304,
vision_num_layers=27,
vision_num_heads=16,
hidden_dim=2048,
intermediate_dim=8192,
num_layers=24,
num_query_heads=32,
num_key_value_heads=32,
scale_factor=3,
image_token_id=49190,
)
input_data = {
"token_ids": np.ones((1, 12), dtype="int32"),
"padding_mask": np.ones((1, 12), dtype="int32"),
}
output = backbone(input_data)
from_preset methodSmolVLM2Backbone.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Backbone from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as a
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''modelscope://user/bert_base_en''./bert_base_en'This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Backbone.from_preset(), or from
a model class like keras_hub.models.GemmaBackbone.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
For any Backbone subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
"gemma_2b_en",
)
# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
"bert_base_en",
load_weights=False,
)
| Preset | Parameters | Description |
|---|---|---|
| smolvlm2_256m_video_instruct | 256.48M | 256 million parameter SmolVLM2 vision-language model with a 30-layer SmolLM2 text decoder and a 12-layer SigLIP vision encoder (512x512 images). Instruction-tuned for image and video understanding. |
| smolvlm2_500m_video_instruct | 507.48M | 500 million parameter SmolVLM2 vision-language model with a 32-layer SmolLM2 text decoder and a 12-layer SigLIP vision encoder (512x512 images). Instruction-tuned for image and video understanding. |
| smolvlm2_2.2b_instruct | 2.25B | 2.2 billion parameter SmolVLM2 vision-language model with a 24-layer SmolLM2 text decoder and a 27-layer SigLIP vision encoder (384x384 images). Instruction-tuned for image and video understanding. |
token_embedding propertykeras_hub.models.SmolVLM2Backbone.token_embedding
A keras.layers.Embedding instance for embedding token ids.
This layer embeds integer token ids to the hidden dim of the model.
enable_lora methodSmolVLM2Backbone.enable_lora(rank, target_layer_names=None)
Enable Lora on the backbone.
Calling this method will freeze all weights on the backbone,
while enabling Lora on the query & value EinsumDense layers
of the attention layers.
Arguments
None, this will be populated with the
default LoRA layer names as returned by
backbone.default_lora_layer_names().