BLIP2Backbone model

[source]

BLIP2Backbone class

keras_hub.models.BLIP2Backbone(
    vision_encoder, qformer, language_model, dtype=None, **kwargs
)

BLIP-2 core network.

BLIP-2 is a vision-language model that connects a frozen image encoder and a frozen large language model (LLM) through a lightweight trainable Querying Transformer (Q-Former). The Q-Former distills visual information into a fixed number of query embeddings which are then fed as a soft visual prompt to the language model.

The forward pass follows three stages: 1. A BLIP2VisionEncoder (ViT) maps raw images to patch features. 2. A BLIP2QFormer cross-attends learned query tokens against those patch features and produces a compact set of visual embeddings. 3. A language model (OPT or Flan-T5) receives the query embeddings prepended to its token sequence and generates text.

When vision_encoder is None the backbone operates in text-only mode: the Q-Former is bypassed and the language model receives only token ids.

For a higher-level text-generation interface see keras_hub.models.BLIP2CausalLM.

Arguments

• vision_encoder: A keras_hub.models.BLIP2VisionEncoder instance. Pass None for a text-only backbone.
• qformer: A keras_hub.models.BLIP2QFormer instance. Pass None when vision_encoder is None.
• language_model: The language model instance (e.g. BLIP2CustomOPT or BLIP2FlanT5).
• dtype: string or keras.mixed_precision.DTypePolicy. Dtype used for model computations and weights. Defaults to None (Keras global default).
• **kwargs: Additional keyword arguments forwarded to the base Backbone / keras.Model.

Example

# Text-only (no vision encoder)
backbone = keras_hub.models.BLIP2Backbone(
    vision_encoder=None,
    qformer=None,
    language_model=my_opt,
)
output = backbone({"token_ids": token_ids, "padding_mask": mask})

# Full vision-language backbone
backbone = keras_hub.models.BLIP2Backbone.from_preset("blip2_opt_2.7b")
output = backbone({
    "images": images,           # (B, H, W, 3)
    "token_ids": token_ids,     # (B, seq_len)
    "padding_mask": mask,       # (B, seq_len)
})

[source]

from_preset method

BLIP2Backbone.from_preset(preset, load_weights=True, **kwargs)

Instantiate a keras_hub.models.Backbone from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as a one of:

1. a built-in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a ModelScope handle like 'modelscope://user/bert_base_en'
5. a path to a local preset directory like './bert_base_en'

This constructor can be called in one of two ways. Either from the base class like keras_hub.models.Backbone.from_preset(), or from a model class like keras_hub.models.GemmaBackbone.from_preset(). If calling from the base class, the subclass of the returning object will be inferred from the config in the preset directory.

For any Backbone subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
• load_weights: bool. If True, the weights will be loaded into the model architecture. If False, the weights will be randomly initialized.

Examples

# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
    "gemma_2b_en",
)

# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
    "bert_base_en",
    load_weights=False,
)