BLIP2CausalLMPreprocessor layer

[source]

BLIP2CausalLMPreprocessor class

keras_hub.models.BLIP2CausalLMPreprocessor(
    tokenizer,
    image_converter=None,
    sequence_length=512,
    add_start_token=True,
    add_end_token=True,
    **kwargs
)

Multimodal preprocessor for the BLIP-2 Causal LM.

This preprocessing layer is meant for use with keras_hub.models.BLIP2CausalLM. It can be configured in two ways: text-only and text + vision, based on whether the passed value of image_converter is None. For the former, it takes in batches of strings. For the latter, it takes in batches of images and strings. It returns outputs in a (x, y, sample_weight) format, where the y label is the next token id in the x sequence.

For use with generation, the layer also exposes two methods generate_preprocess() and generate_postprocess(). When this preprocessor is attached to a keras_hub.models.BLIP2CausalLM instance, these methods will be called implicitly in generate(). They can also be called standalone (e.g. to precompute preprocessing inputs for generation in a separate process).

Arguments

• tokenizer: A keras_hub.models.BLIP2OPTTokenizer or keras_hub.models.BLIP2FlanT5Tokenizer instance. The actual tokenizer is resolved from the preset config when using from_preset().
• image_converter: A keras_hub.models.BLIP2ImageConverter instance, or None. If None, the preprocessor operates in text-only mode.
• sequence_length: int. The maximum length of the packed token sequence. Defaults to 512.
• add_start_token: bool. Whether to prepend the tokenizer start token to each input sequence. Defaults to True.
• add_end_token: bool. Whether to append the tokenizer end token to each input sequence. Defaults to True.

Examples

# Load from a preset.
preprocessor = keras_hub.models.BLIP2CausalLMPreprocessor.from_preset(
    "blip2_opt_2_7b"
)

# Text-only input.
preprocessor({"text": ["A photo of a cat", "A photo of a dog"]})

# Image + text input.
preprocessor({
    "images": np.random.randint(0, 256, (2, 500, 500, 3)),
    "text": ["A photo of a cat", "A photo of a dog"],
})

# Prepare tokens for generation.
preprocessor.generate_preprocess({
    "images": np.random.randint(0, 256, (500, 500, 3)),
    "text": "A photo of",
})

# Map generation outputs back to strings.
preprocessor.generate_postprocess({
    "token_ids": np.array([[0, 4, 5, 2, 1, 1]]),
    "padding_mask": np.array([[1, 1, 1, 1, 0, 0]]),
})

References

• Li et al., 2023

[source]

from_preset method

BLIP2CausalLMPreprocessor.from_preset(
    preset, config_file="preprocessor.json", **kwargs
)

Instantiate a keras_hub.models.Preprocessor from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

1. a built-in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Preprocessor subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

As there are usually multiple preprocessing classes for a given model, this method should be called on a specific subclass like keras_hub.models.BertTextClassifierPreprocessor.from_preset().

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.

Examples

# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.CausalLMPreprocessor.from_preset(
    "gemma_2b_en",
)

# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.TextClassifierPreprocessor.from_preset(
    "bert_base_en",
)

image_converter property

keras_hub.models.BLIP2CausalLMPreprocessor.image_converter

The image converter used to preprocess image data.

tokenizer property

keras_hub.models.BLIP2CausalLMPreprocessor.tokenizer

The tokenizer used to tokenize strings.