FNetMaskedLMPreprocessor layer

[source]

FNetMaskedLMPreprocessor class

keras_nlp.models.FNetMaskedLMPreprocessor(
    tokenizer,
    sequence_length=512,
    truncate="round_robin",
    mask_selection_rate=0.15,
    mask_selection_length=96,
    mask_token_rate=0.8,
    random_token_rate=0.1,
    **kwargs
)

FNet preprocessing for the masked language modeling task.

This preprocessing layer will prepare inputs for a masked language modeling task. It is primarily intended for use with the keras_nlp.models.FNetMaskedLM task model. Preprocessing will occur in multiple steps.

1. Tokenize any number of input segments using the tokenizer.
2. Pack the inputs together with the appropriate "<s>", "</s>" and "<pad>" tokens, i.e., adding a single "<s>" at the start of the entire sequence, "</s></s>" between each segment, and a "</s>" at the end of the entire sequence.
3. Randomly select non-special tokens to mask, controlled by mask_selection_rate.
4. Construct a (x, y, sample_weight) tuple suitable for training with a keras_nlp.models.FNetMaskedLM task model.

Arguments

• tokenizer: A keras_nlp.models.FNetTokenizer instance.
• sequence_length: The length of the packed inputs.
• mask_selection_rate: The probability an input token will be dynamically masked.
• mask_selection_length: The maximum number of masked tokens supported by the layer.
• mask_token_rate: float. mask_token_rate must be between 0 and 1 which indicates how often the mask_token is substituted for tokens selected for masking. Defaults to 0.8.
• random_token_rate: float. random_token_rate must be between 0 and 1 which indicates how often a random token is substituted for tokens selected for masking. Note: mask_token_rate + random_token_rate <= 1, and for (1 - mask_token_rate - random_token_rate), the token will not be changed. Defaults to 0.1.
• truncate: string. The algorithm to truncate a list of batched segments to fit within sequence_length. The value can be either round_robin or waterfall: - "round_robin": Available space is assigned one token at a time in a round-robin fashion to the inputs that still need some, until the limit is reached. - "waterfall": The allocation of the budget is done using a "waterfall" algorithm that allocates quota in a left-to-right manner and fills up the buckets until we run out of budget. It supports an arbitrary number of segments.

Examples

Directly calling the layer on data.

# Load the preprocessor from a preset.
preprocessor = keras_nlp.models.FNetMaskedLMPreprocessor.from_preset(
    "f_net_base_en"
)

# Tokenize and mask a single sentence.
preprocessor("The quick brown fox jumped.")

# Tokenize and mask a batch of single sentences.
preprocessor(["The quick brown fox jumped.", "Call me Ishmael."])

# Tokenize and mask sentence pairs.
# In this case, always convert input to tensors before calling the layer.
first = tf.constant(["The quick brown fox jumped.", "Call me Ishmael."])
second = tf.constant(["The fox tripped.", "Oh look, a whale."])
preprocessor((first, second))

Mapping with tf.data.Dataset.

preprocessor = keras_nlp.models.FNetMaskedLMPreprocessor.from_preset(
    "f_net_base_en"
)

first = tf.constant(["The quick brown fox jumped.", "Call me Ishmael."])
second = tf.constant(["The fox tripped.", "Oh look, a whale."])

# Map single sentences.
ds = tf.data.Dataset.from_tensor_slices(first)
ds = ds.map(preprocessor, num_parallel_calls=tf.data.AUTOTUNE)

# Alternatively, you can create a preprocessor from your own vocabulary.
vocab_data = tf.data.Dataset.from_tensor_slices(
    ["the quick brown fox", "the earth is round"]
)

# Map sentence pairs.
ds = tf.data.Dataset.from_tensor_slices((first, second))
# Watch out for tf.data's default unpacking of tuples here!
# Best to invoke the `preprocessor` directly in this case.
ds = ds.map(
    lambda first, second: preprocessor(x=(first, second)),
    num_parallel_calls=tf.data.AUTOTUNE,
)

[source]

from_preset method

FNetMaskedLMPreprocessor.from_preset(preset, **kwargs)

Instantiate a keras_nlp.models.Preprocessor from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as a one of:

1. a built in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Preprocessor subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

As there are usually multiple preprocessing classes for a given model, this method should be called on a specific subclass like keras_nlp.models.BertPreprocessor.from_preset().

Arguments

• preset: string. A built in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.

Examples

# Load a preprocessor for Gemma generation.
preprocessor = keras_nlp.models.GemmaCausalLMPreprocessor.from_preset(
    "gemma_2b_en",
)

# Load a preprocessor for Bert classification.
preprocessor = keras_nlp.models.BertPreprocessor.from_preset(
    "bert_base_en",
)

tokenizer property

keras_nlp.models.FNetMaskedLMPreprocessor.tokenizer

The tokenizer used to tokenize strings.