► KerasHub: Pretrained Models / API documentation / Tokenizers / BytePairTokenizer

BytePairTokenizer

`BytePairTokenizer` class

keras_hub.tokenizers.BytePairTokenizer(
    vocabulary=None,
    merges=None,
    sequence_length=None,
    add_prefix_space=False,
    unsplittable_tokens=None,
    dtype="int32",
    **kwargs
)

Bype-pair encoding tokenizer layer.

This BPE tokenizer provides the same functionality as the official GPT-2 tokenizer. Given the same vocabulary which maps tokens to ids, and merges which describes BPE merge rules, it should provide the same output as OpenAI implementation (https://github.com/openai/gpt-2/blob/master/src/encoder.py). Different from OpenAI, this implementation is graph-compatible, so you can use it within a tf.data pipeline.

If input is a batch of strings (rank > 0): By default, the layer will output a tf.RaggedTensor where the last dimension of the output is ragged. If sequence_length is set, the layer will output a dense tf.Tensor where all inputs have been padded or truncated to sequence_length. If input is a scalar string (rank == 0): By default, the layer will output a dense tf.Tensor with static shape [None]. If sequence_length is set, the output will be a dense tf.Tensor of shape [sequence_length].

Arguments

vocabulary: string or dict, maps token to integer ids. If it is a string, it should be the file path to a json file.
merges: string or list, contains the merge rule. If it is a string, it should be the file path to merge rules. The merge rule file should have one merge rule per line.
sequence_length: int. If set, the output will be padded or truncated to the sequence_length. Defaults to None.
add_prefix_space: bool. Whether to add an initial space to the input. This tokenizer is whitespace aware, and will tokenize a word with a leading space differently. Adding a prefix space to the first word will cause it to be tokenized equivalently to all subsequent words in the sequence. Defaults to False.
unsplittable_tokens: list. A list of strings that will never be split during the word-level splitting applied before the byte-pair encoding. This can be used to ensure special tokens map to unique indices in the vocabulary, even if these special tokens contain splittable characters such as punctuation. Special tokens must still be included in vocabulary. Defaults to None.

Examples

Tokenize

>>> vocab = {"butter": 1, "fly": 2}
>>> merge = ["b u", "t t", "e r", "bu tt", "butt er", "f l", "fl y"]
>>> tokenizer = keras_hub.tokenizers.BytePairTokenizer(vocab, merge)
>>> outputs = tokenizer("butterfly")
>>> np.array(outputs)
array([1, 2], dtype=int32)
>>> seq1, seq2 = tokenizer(["butterfly", "butter"])
>>> np.array(seq1)
array([1, 2])
>>> np.array(seq2)
array([1])
>>> tokenizer = keras_hub.tokenizers.BytePairTokenizer(
...     vocab, merge, sequence_length=2)
>>> seq1, seq2 = tokenizer(["butterfly", "butter"])
>>> np.array(seq1)
array([1, 2], dtype=int32)
>>> np.array(seq2)
array([1, 0], dtype=int32)

Detokenize

>>> vocab = {"butter": 1, "fly": 2}
>>> merge = ["b u", "t t", "e r", "bu tt", "butt er", "f l", "fl y"]
>>> tokenizer = keras_hub.tokenizers.BytePairTokenizer(vocab, merge)
>>> tokenizer.detokenize([[1, 2]])
['butterfly']