AlbertMaskedLMPreprocessor
classkeras_nlp.models.AlbertMaskedLMPreprocessor(
tokenizer,
sequence_length=512,
truncate="round_robin",
mask_selection_rate=0.15,
mask_selection_length=96,
mask_token_rate=0.8,
random_token_rate=0.1,
**kwargs
)
ALBERT preprocessing for the masked language modeling task.
This preprocessing layer will prepare inputs for a masked language modeling
task. It is primarily intended for use with the
keras_nlp.models.AlbertMaskedLM
task model. Preprocessing will occur in
multiple steps.
tokenizer
."<s>"
, "</s>"
and
"<pad>"
tokens, i.e., adding a single "<s>"
at the start of the
entire sequence, "</s></s>"
between each segment,
and a "</s>"
at the end of the entire sequence.mask_selection_rate
.(x, y, sample_weight)
tuple suitable for training with a
keras_nlp.models.AlbertMaskedLM
task model.Arguments
keras_nlp.models.AlbertTokenizer
instance.mask_token_rate
must be
between 0 and 1 which indicates how often the mask_token is
substituted for tokens selected for masking.
Defaults to 0.8
.random_token_rate
must be
between 0 and 1 which indicates how often a random token is
substituted for tokens selected for masking. Default is 0.1.
Note: mask_token_rate + random_token_rate <= 1, and for
(1 - mask_token_rate - random_token_rate), the token will not be
changed. Defaults to 0.1
.sequence_length
. The value can be either
round_robin
or waterfall
:"round_robin"
: Available space is assigned one token at a
time in a round-robin fashion to the inputs that still need
some, until the limit is reached."waterfall"
: The allocation of the budget is done using a
"waterfall" algorithm that allocates quota in a
left-to-right manner and fills up the buckets until we run
out of budget. It supports an arbitrary number of segments.Examples
Directly calling the layer on data.
preprocessor = keras_nlp.models.AlbertMaskedLMPreprocessor.from_preset(
"albert_base_en_uncased"
)
# Tokenize and mask a single sentence.
preprocessor("The quick brown fox jumped.")
# Tokenize and mask a batch of single sentences.
preprocessor(["The quick brown fox jumped.", "Call me Ishmael."])
# Tokenize and mask sentence pairs.
# In this case, always convert input to tensors before calling the layer.
first = tf.constant(["The quick brown fox jumped.", "Call me Ishmael."])
second = tf.constant(["The fox tripped.", "Oh look, a whale."])
preprocessor((first, second))
Mapping with tf.data.Dataset
.
preprocessor = keras_nlp.models.AlbertMaskedLMPreprocessor.from_preset(
"albert_base_en_uncased"
)
first = tf.constant(["The quick brown fox jumped.", "Call me Ishmael."])
second = tf.constant(["The fox tripped.", "Oh look, a whale."])
# Map single sentences.
ds = tf.data.Dataset.from_tensor_slices(first)
ds = ds.map(preprocessor, num_parallel_calls=tf.data.AUTOTUNE)
# Map sentence pairs.
ds = tf.data.Dataset.from_tensor_slices((first, second))
# Watch out for tf.data's default unpacking of tuples here!
# Best to invoke the `preprocessor` directly in this case.
ds = ds.map(
lambda first, second: preprocessor(x=(first, second)),
num_parallel_calls=tf.data.AUTOTUNE,
)
from_preset
methodAlbertMaskedLMPreprocessor.from_preset(preset, **kwargs)
Instantiate a keras_nlp.models.Preprocessor
from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset
can be passed as
one of:
'bert_base_en'
'kaggle://user/bert/keras/bert_base_en'
'hf://user/bert_base_en'
'./bert_base_en'
For any Preprocessor
subclass, you can run cls.presets.keys()
to
list all built-in presets available on the class.
As there are usually multiple preprocessing classes for a given model,
this method should be called on a specific subclass like
keras_nlp.models.BertTextClassifierPreprocessor.from_preset()
.
Arguments
Examples
# Load a preprocessor for Gemma generation.
preprocessor = keras_nlp.models.GemmaCausalLMPreprocessor.from_preset(
"gemma_2b_en",
)
# Load a preprocessor for Bert classification.
preprocessor = keras_nlp.models.BertTextClassifierPreprocessor.from_preset(
"bert_base_en",
)
Preset name | Parameters | Description |
---|---|---|
albert_base_en_uncased | 11.68M | 12-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus. |
albert_large_en_uncased | 17.68M | 24-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus. |
albert_extra_large_en_uncased | 58.72M | 24-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus. |
albert_extra_extra_large_en_uncased | 222.60M | 12-layer ALBERT model where all input is lowercased. Trained on English Wikipedia + BooksCorpus. |
tokenizer
propertykeras_nlp.models.AlbertMaskedLMPreprocessor.tokenizer
The tokenizer used to tokenize strings.