BLIP2ImageConverter

[source]

BLIP2ImageConverter class

keras_hub.layers.BLIP2ImageConverter(
    image_size=(224, 224),
    crop_to_aspect_ratio=False,
    interpolation="bicubic",
    antialias=True,
    **kwargs
)

A preprocessing layer for images used by the BLIP-2 model.

This converter resizes, normalizes, and rescales images to the format expected by the BLIP-2 vision encoder (EVA-CLIP). Preprocessing is always performed in float32 regardless of the global Keras dtype policy.

Preprocessing steps: 1. Resizing: Images are stretched to (224, 224) (default) using antialiased bicubic interpolation, matching HuggingFace's BlipImageProcessor (a plain resize, no aspect-ratio crop). 2. Normalization: Pixel values are normalized using EVA-CLIP channel statistics (mean and standard deviation per channel).

Arguments

• image_size: (int, int) tuple or None. The output spatial size of the image, not including the channels axis. Defaults to (224, 224).
• crop_to_aspect_ratio: bool. If True, resize without aspect ratio distortion. Defaults to False to match HuggingFace, which stretches the image to the target size.
• interpolation: string. The interpolation method used during resizing. Defaults to "bicubic".
• antialias: bool. Whether to use an antialiasing filter when downsampling. Defaults to True to match HuggingFace's PIL bicubic resize.
• **kwargs: Standard Keras layer arguments.

Example

converter = keras_hub.layers.BLIP2ImageConverter()
images = np.random.randint(0, 256, (1, 500, 500, 3))
processed = converter(images)  # shape: (1, 224, 224, 3)

References

• Li et al., 2023

[source]

from_preset method

BLIP2ImageConverter.from_preset(preset, **kwargs)

Instantiate a keras_hub.layers.ImageConverter from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

1. a built-in preset identifier like 'pali_gemma_3b_224'
2. a Kaggle Models handle like 'kaggle://user/paligemma/keras/pali_gemma_3b_224'
3. a Hugging Face handle like 'hf://user/pali_gemma_3b_224'
4. a path to a local preset directory like './pali_gemma_3b_224'

You can run cls.presets.keys() to list all built-in presets available on the class.

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
• load_weights: bool. If True, the weights will be loaded into the model architecture. If False, the weights will be randomly initialized.

Examples

batch = np.random.randint(0, 256, size=(2, 512, 512, 3))

# Resize images for `"pali_gemma_3b_224"`.
converter = keras_hub.layers.ImageConverter.from_preset(
    "pali_gemma_3b_224"
)
converter(batch) # # Output shape (2, 224, 224, 3)

# Resize images for `"pali_gemma_3b_448"` without cropping.
converter = keras_hub.layers.ImageConverter.from_preset(
    "pali_gemma_3b_448",
    crop_to_aspect_ratio=False,
)
converter(batch) # # Output shape (2, 448, 448, 3)