CLIPPreprocessor

[source]

CLIPPreprocessor class

keras_hub.models.CLIPPreprocessor(
    tokenizer,
    image_converter=None,
    sequence_length=77,
    add_start_token=True,
    add_end_token=True,
    to_lower=True,
    **kwargs
)

CLIP preprocessor.

This preprocessing layer will do 2 things:

This preprocessing layer is meant for use with keras_hub.models.CLIPBackbone. By default, it will take in batches of strings and images, and return token ids and resized images.

Arguments

• tokenizer: A keras_hub.models.CLIPTokenizer instance.
• image_converter: A keras_hub.models.CLIPImageConverter instance.
• sequence_length: The length of the packed inputs.
• add_start_token: If True, the preprocessor will prepend the tokenizer start token to each input sequence.
• add_end_token: If True, the preprocessor will append the tokenizer end token to each input sequence.
• to_lower: bool. Whether to lower the inputs.

Call arguments

• x: A dict with "prompts" and "images" keys, where "prompts" is tf.Tensor or list of python strings and "images" are the image tensors.
• y: Label data. Should always be None since SigLIP doesn't need the label to calculate the loss.
• sample_weight: Label weights.
• sequence_length: Pass to override the configured sequence_length of the layer.

Examples

# Load the preprocessor from a preset.
preprocessor = keras_hub.models.CLIPPreprocessor.from_preset(
    "clip_vit_base_patch16"
)

# Tokenize the sentence and preprocess the image.
preprocessor(
    {
        "prompts": "The quick brown fox jumped.",
        "images": np.ones(shape=(123, 123, 3)),
    }
)

# Tokenize a batch of sentences and preprocess a batch of images.
preprocessor(
    {
        "prompts": ["The quick brown fox jumped.", "The fox slept."],
        "images": np.ones(shape=(2, 123, 123, 3)),
    }
)

[source]

from_preset method

CLIPPreprocessor.from_preset(preset, config_file="preprocessor.json", **kwargs)

Instantiate a keras_hub.models.Preprocessor from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

1. a built-in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Preprocessor subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

As there are usually multiple preprocessing classes for a given model, this method should be called on a specific subclass like keras_hub.models.BertTextClassifierPreprocessor.from_preset().

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.

Examples

# Load a preprocessor for Gemma generation.
preprocessor = keras_hub.models.CausalLMPreprocessor.from_preset(
    "gemma_2b_en",
)

# Load a preprocessor for Bert classification.
preprocessor = keras_hub.models.TextClassifierPreprocessor.from_preset(
    "bert_base_en",
)

tokenizer property

keras_hub.models.CLIPPreprocessor.tokenizer

The tokenizer used to tokenize strings.