tf.keras.layers.TextVectorization( max_tokens=None, standardize="lower_and_strip_punctuation", split="whitespace", ngrams=None, output_mode="int", output_sequence_length=None, pad_to_max_tokens=False, vocabulary=None, idf_weights=None, sparse=False, ragged=False, encoding="utf-8", **kwargs )
A preprocessing layer which maps text features to integer sequences.
This layer has basic options for managing text in a Keras model. It
transforms a batch of strings (one example = one string) into either a list
of token indices (one example = 1D tensor of integer token indices) or a
dense representation (one example = 1D tensor of float values representing
data about the example's tokens). This layer is meant to handle natural
language inputs. To handle simple string inputs (categorical strings or
pre-tokenized strings) see
The vocabulary for the layer must be either supplied on construction or
adapt(). When this layer is adapted, it will analyze the
dataset, determine the frequency of individual string values, and create a
vocabulary from them. This vocabulary can have unlimited size or be capped,
depending on the configuration options for this layer; if there are more
unique values in the input than the maximum vocabulary size, the most
frequent terms will be used to create the vocabulary.
The processing of each example contains the following steps:
Some notes on passing callables to customize splitting and normalization for this layer:
tf.keras.utils.register_keras_serializablefor more details).
standardize, the data received by the callable will be exactly as passed to this layer. The callable should return a tensor of the same shape as the input.
split, the data received by the callable will have the 1st dimension squeezed out - instead of
[["string to split"], ["another string to split"]], the Callable will see
["string to split", "another string to split"]. The callable should return a Tensor with the first dimension containing the split tokens - in this example, we should see something like
[["string", "to", "split"], ["another", "string", "to", "split"]]. This makes the callable site natively compatible with
For an overview and full list of preprocessing layers, see the preprocessing guide.
pad_to_max_tokens=True. Note that this vocabulary contains 1 OOV token, so the effective number of tokens is
(max_tokens - 1 - (1 if output_mode == "int" else 0)).
None: No standardization. -
"lower_and_strip_punctuation": Text will be lowercased and all punctuation removed. -
"lower": Text will be lowercased. -
"strip_punctuation": All punctuation will be removed. - Callable: Inputs will passed to the callable function, which should standardized and returned.
None: No splitting. -
"whitespace": Split on whitespace. -
"character": Split on each unicode character. - Callable: Standardized inputs will passed to the callable function, which should split and returned.
"tf_idf", configuring the layer as follows: -
"int": Outputs integer indices, one integer index per split string token. When
output_mode == "int", 0 is reserved for masked locations; this reduces the vocab size to
max_tokens - 2instead of
max_tokens - 1. -
"multi_hot": Outputs a single int array per batch, of either vocab_size or max_tokens size, containing 1s in all elements where the token mapped to that index exists at least once in the batch item. -
"multi_hot", but the int array contains a count of the number of times the token at that index appeared in the batch item. -
"multi_hot", but the TF-IDF algorithm is applied to find the value in each token slot. For
"int"output, any shape of input and output is supported. For all other output modes, currently only rank 1 inputs (and rank 2 outputs after splitting) are supported.
output_sequence_lengthvalues, resulting in a tensor of shape
(batch_size, output_sequence_length)regardless of how many tokens resulted from the splitting step. Defaults to None.
"tf_idf"modes. If True, the output will have its feature axis padded to
max_tokenseven if the number of unique tokens in the vocabulary is less than max_tokens, resulting in a tensor of shape
(batch_size, max_tokens)regardless of vocabulary size. Defaults to False.
"tf_idf". A tuple, list, 1D numpy array, or 1D tensor or the same length as the vocabulary, containing the floating point inverse document frequency weights, which will be multiplied by per sample term counts for the final
tf_idfweight. If the
vocabularyargument is set, and
"tf_idf", this argument must be supplied.
"int"output mode. If True, returns a
RaggedTensorinstead of a dense
Tensor, where each sequence may have a different length after string splitting. Defaults to False.
"tf_idf"output modes. If True, returns a
SparseTensorinstead of a dense
Tensor. Defaults to False.
This example instantiates a
TextVectorization layer that lowercases text,
splits on whitespace, strips punctuation, and outputs integer vocab indices.
>>> text_dataset = tf.data.Dataset.from_tensor_slices(["foo", "bar", "baz"]) >>> max_features = 5000 # Maximum vocab size. >>> max_len = 4 # Sequence length to pad the outputs to. >>> >>> # Create the layer. >>> vectorize_layer = tf.keras.layers.TextVectorization( ... max_tokens=max_features, ... output_mode='int', ... output_sequence_length=max_len) >>> >>> # Now that the vocab layer has been created, call `adapt` on the >>> # text-only dataset to create the vocabulary. You don't have to batch, >>> # but for large datasets this means we're not keeping spare copies of >>> # the dataset. >>> vectorize_layer.adapt(text_dataset.batch(64)) >>> >>> # Create the model that uses the vectorize text layer >>> model = tf.keras.models.Sequential() >>> >>> # Start by creating an explicit input layer. It needs to have a shape of >>> # (1,) (because we need to guarantee that there is exactly one string >>> # input per batch), and the dtype needs to be 'string'. >>> model.add(tf.keras.Input(shape=(1,), dtype=tf.string)) >>> >>> # The first layer in our model is the vectorization layer. After this >>> # layer, we have a tensor of shape (batch_size, max_len) containing >>> # vocab indices. >>> model.add(vectorize_layer) >>> >>> # Now, the model can map strings to integers, and you can add an >>> # embedding layer to map these integers to learned embeddings. >>> input_data = [["foo qux bar"], ["qux baz"]] >>> model.predict(input_data) array([[2, 1, 4, 0], [1, 3, 0, 0]])
This example instantiates a
TextVectorization layer by passing a list
of vocabulary terms to the layer's
>>> vocab_data = ["earth", "wind", "and", "fire"] >>> max_len = 4 # Sequence length to pad the outputs to. >>> >>> # Create the layer, passing the vocab directly. You can also pass the >>> # vocabulary arg a path to a file containing one vocabulary word per >>> # line. >>> vectorize_layer = tf.keras.layers.TextVectorization( ... max_tokens=max_features, ... output_mode='int', ... output_sequence_length=max_len, ... vocabulary=vocab_data) >>> >>> # Because we've passed the vocabulary directly, we don't need to adapt >>> # the layer - the vocabulary is already set. The vocabulary contains the >>> # padding token ('') and OOV token ('[UNK]') as well as the passed >>> # tokens. >>> vectorize_layer.get_vocabulary() ['', '[UNK]', 'earth', 'wind', 'and', 'fire']