MixtralTokenizer

[source]

MixtralTokenizer class

keras_hub.tokenizers.MixtralTokenizer(proto, **kwargs)

A SentencePiece tokenizer layer.

This layer provides an implementation of SentencePiece tokenization as described in the SentencePiece paper and the SentencePiece package. The tokenization will run entirely within the Tensorflow graph, and can be saved inside a keras.Model.

By default, the layer will output a tf.RaggedTensor where the last dimension of the output is ragged after whitespace splitting and sub-word tokenizing. If sequence_length is set, the layer will output a dense tf.Tensor where all inputs have been padded or truncated to sequence_length. The output dtype can be controlled via the dtype argument, which should be either an integer or string type.

Arguments

• proto: Either a string path to a SentencePiece proto file, or a bytes object with a serialized SentencePiece proto. See the SentencePiece repository for more details on the format.
• sequence_length: If set, the output will be converted to a dense tensor and padded/trimmed so all outputs are of sequence_length.
• add_bos: Add beginning of sentence token to the result.
• add_eos: Add end of sentence token to the result. Token is always truncated if output is longer than specified sequence_length.

References

• Kudo and Richardson, 2018

Examples

From bytes.

def train_sentence_piece_bytes(ds, size):
    bytes_io = io.BytesIO()
    sentencepiece.SentencePieceTrainer.train(
        sentence_iterator=ds.as_numpy_iterator(),
        model_writer=bytes_io,
        vocab_size=size,
    )
    return bytes_io.getvalue()

# Train a sentencepiece proto.
ds = tf.data.Dataset.from_tensor_slices(["the quick brown fox."])
proto = train_sentence_piece_bytes(ds, 20)
# Tokenize inputs.
tokenizer = keras_hub.tokenizers.SentencePieceTokenizer(proto=proto)
ds = ds.map(tokenizer)

From a file.

def train_sentence_piece_file(ds, path, size):
    with open(path, "wb") as model_file:
        sentencepiece.SentencePieceTrainer.train(
            sentence_iterator=ds.as_numpy_iterator(),
            model_writer=model_file,
            vocab_size=size,
        )

# Train a sentencepiece proto.
ds = tf.data.Dataset.from_tensor_slices(["the quick brown fox."])
proto = train_sentence_piece_file(ds, "model.spm", 20)
# Tokenize inputs.
tokenizer = keras_hub.tokenizers.SentencePieceTokenizer(proto="model.spm")
ds = ds.map(tokenizer)

[source]

from_preset method

MixtralTokenizer.from_preset(preset, config_file="tokenizer.json", **kwargs)

Instantiate a keras_hub.models.Tokenizer from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

1. a built-in preset identifier like 'bert_base_en'
2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
3. a Hugging Face handle like 'hf://user/bert_base_en'
4. a path to a local preset directory like './bert_base_en'

For any Tokenizer subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

This constructor can be called in one of two ways. Either from the base class like keras_hub.models.Tokenizer.from_preset(), or from a model class like keras_hub.models.GemmaTokenizer.from_preset(). If calling from the base class, the subclass of the returning object will be inferred from the config in the preset directory.

Arguments

• preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
• load_weights: bool. If True, the weights will be loaded into the model architecture. If False, the weights will be randomly initialized.

Examples

# Load a preset tokenizer.
tokenizer = keras_hub.tokenizer.Tokenizer.from_preset("bert_base_en")

# Tokenize some input.
tokenizer("The quick brown fox tripped.")

# Detokenize some input.
tokenizer.detokenize([5, 6, 7, 8, 9])