Mistral3Backbone classkeras_hub.models.Mistral3Backbone(
vocabulary_size,
num_layers,
num_query_heads,
hidden_dim,
intermediate_dim,
num_key_value_heads,
vision_encoder=None,
multimodal_projector=None,
rope_max_wavelength=10000,
rope_scaling_factor=1.0,
rope_type="linear",
beta_fast=32.0,
beta_slow=1.0,
original_max_position_embeddings=4096,
layer_norm_epsilon=1e-06,
sliding_window=None,
head_dim=None,
dropout=0,
image_token_index=10,
tie_word_embeddings=False,
llama_4_scaling_beta=None,
attention_factor=None,
dtype=None,
**kwargs
)
The Mistral3 (Pixtral vision + Mistral text) core architecture.
This network implements a multimodal Transformer-based decoder network, Mistral3, as used by models such as Mistral Small 3.1/3.2. It includes the token embedding lookups, a Pixtral-style vision encoder, and transformer decoder layers.
The default constructor gives a fully customizable, randomly initialized
Mistral3 model with any number of layers, heads, and embedding
dimensions. To load preset architectures and weights, use the
from_preset constructor.
Arguments
keras_hub.models.Mistral3VisionEncoder instance,
or None. When None, the backbone is text-only and does not
declare pixel_values, image_sizes, or placeholder_indices
inputs.Mistral3MultiModalProjector instance, or
None. Required when vision_encoder is not None.10000.1.0."linear" and "yarn". Defaults to "linear". Use
"yarn" for checkpoints with YaRN-scaled rotary embeddings
(e.g. Ministral 3).rope_type="yarn". Defaults to 32.0.rope_type="yarn". Defaults to 1.0.rope_type="yarn". Defaults to 4096.rope_type="yarn". Defaults to None, which derives the scale
as 0.1 * log(rope_scaling_factor) + 1.0 (the standard YaRN
default). Pass an explicit value for checkpoints that compute a
different scale from mscale/mscale_all_dim — e.g. Ministral 3
and Shieldstral set both to 1.0, giving attention_factor=1.0.1e-6.sliding_window
number of tokens are saved in the cache and used to generate the
next token. Defaults to None.None (the default), falls back to hidden_dim // num_query_heads.
Set explicitly when the model's head size is not equal to
hidden_dim // num_query_heads — e.g. Magistral uses
head_dim=128 with hidden_dim=5120 and num_query_heads=32.0.token_ids that
marks image placeholder positions. Defaults to 10.False.attn_temperature_tuning: query *= 1 + llama_4_scaling_beta *
log(1 + floor(pos / original_max_position_embeddings)).
Defaults to None (no scaling).keras.mixed_precision.DTypePolicy. The dtype to use
for model computations and weights. Note that some computations,
such as softmax and layer normalization, will always be done at
float32 precision regardless of dtype.Examples
input_data = {
"token_ids": np.ones(shape=(1, 12), dtype="int32"),
"padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]),
"pixel_values": np.ones(shape=(1, 3, 64, 64), dtype="float32"),
"image_sizes": np.array([[64, 64]], dtype="int32"),
"placeholder_indices": np.zeros(shape=(1, 16), dtype="int32"),
}
# Pretrained Mistral3 decoder.
model = keras_hub.models.Mistral3Backbone.from_preset(
"mistral_small_3.1_24b_instruct_2503_en"
)
model(input_data)
from_preset methodMistral3Backbone.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Backbone from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as a
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''modelscope://user/bert_base_en''./bert_base_en'This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Backbone.from_preset(), or from
a model class like keras_hub.models.GemmaBackbone.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
For any Backbone subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
"gemma_2b_en",
)
# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
"bert_base_en",
load_weights=False,
)
| Preset | Parameters | Description |
|---|---|---|
| ministral_3_3b_base_2512_en | 3.85B | 3 billion parameter, 26-layer, pretrained Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_3b_instruct_2512_en | 3.85B | 3 billion parameter, 26-layer, instruction-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_3b_reasoning_2512_en | 3.85B | 3 billion parameter, 26-layer, reasoning-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| shieldstral_1.0_3b_en | 3.85B | 3 billion parameter, 26-layer Shieldstral model with a Pixtral vision encoder. The model checks text and image content against a safety policy written in natural language. The model returns one safety score for each input. |
| ministral_3_8b_base_2512_en | 8.92B | 8 billion parameter, 34-layer, pretrained Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_8b_instruct_2512_en | 8.92B | 8 billion parameter, 34-layer, instruction-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_8b_reasoning_2512_en | 8.92B | 8 billion parameter, 34-layer, reasoning-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_14b_base_2512_en | 13.95B | 14 billion parameter, 40-layer, pretrained Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_14b_instruct_2512_en | 13.95B | 14 billion parameter, 40-layer, instruction-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| ministral_3_14b_reasoning_2512_en | 13.95B | 14 billion parameter, 40-layer, reasoning-tuned Ministral 3 model with a Pixtral vision encoder for image input. |
| mistral_small_3.1_24b_base_2503_en | 24.01B | 24 billion parameter, 40-layer, pretrained Mistral3 model with a Pixtral vision encoder for image input. |
| mistral_small_3.1_24b_instruct_2503_en | 24.01B | 24 billion parameter, 40-layer, instruction-tuned Mistral3 model with a Pixtral vision encoder for image input. |
| mistral_small_3.2_24b_instruct_2506_en | 24.01B | 24 billion parameter, 40-layer, instruction-tuned Mistral3 model with a Pixtral vision encoder for image input. An updated version of mistral_small_3.1_24b_instruct_2503_en with improved instruction-following and reduced repetition. |
| devstral_small_2_24b_instruct_2512_en | 24.01B | 24 billion parameter, 40-layer, instruction-tuned Devstral Small 2 model for agentic software engineering, with a Pixtral vision encoder for image input. |
| magistral_small_2509_en | 24.01B | 24 billion parameter, 40-layer, reasoning-tuned Magistral Small model with a Pixtral vision encoder for image input. |
token_embedding propertykeras_hub.models.Mistral3Backbone.token_embedding
A keras.layers.Embedding instance for embedding token ids.
This layer embeds integer token ids to the hidden dim of the model.
enable_lora methodMistral3Backbone.enable_lora(rank, target_layer_names=None)
Enable Lora on the backbone.
Calling this method will freeze all weights on the backbone,
while enabling Lora on the query & value EinsumDense layers
of the attention layers.
Arguments
None, this will be populated with the
default LoRA layer names as returned by
backbone.default_lora_layer_names().