Keras 3 API documentation / KerasNLP / Models / DebertaV3 / DebertaV3Tokenizer

DebertaV3Tokenizer

[source]

DebertaV3Tokenizer class

keras_nlp.models.DebertaV3Tokenizer(proto, **kwargs)

DeBERTa tokenizer layer based on SentencePiece.

This tokenizer class will tokenize raw strings into integer sequences and is based on keras_nlp.tokenizers.SentencePieceTokenizer. Unlike the underlying tokenizer, it will check for all special tokens needed by DeBERTa models and provides a from_preset() method to automatically download a matching vocabulary for a DeBERTa preset.

This tokenizer does not provide truncation or padding of inputs. It can be combined with a keras_nlp.models.DebertaV3Preprocessor layer for input packing.

If input is a batch of strings (rank > 0), the layer will output a tf.RaggedTensor where the last dimension of the output is ragged.

If input is a scalar string (rank == 0), the layer will output a dense tf.Tensor with static shape [None].

Note: The mask token ("[MASK]") is handled differently in this tokenizer. If the token is not present in the provided SentencePiece vocabulary, the token will be appended to the vocabulary. For example, if the vocabulary size is 100, the mask token will be assigned the ID 100.

Arguments

  • proto: Either a string path to a SentencePiece proto file, or a bytes object with a serialized SentencePiece proto. See the SentencePiece repository for more details on the format.

Examples

# Unbatched input.
tokenizer = keras_nlp.models.DebertaV3Tokenizer.from_preset(
    "deberta_v3_base_en",
)
tokenizer("The quick brown fox jumped.")

# Batched inputs.
tokenizer(["the quick brown fox", "the earth is round"])

# Detokenization.
tokenizer.detokenize(tokenizer("The quick brown fox jumped."))

# Custom vocabulary.
bytes_io = io.BytesIO()
ds = tf.data.Dataset.from_tensor_slices(["The quick brown fox jumped."])
sentencepiece.SentencePieceTrainer.train(
    sentence_iterator=ds.as_numpy_iterator(),
    model_writer=bytes_io,
    vocab_size=9,
    model_type="WORD",
    pad_id=0,
    bos_id=1,
    eos_id=2,
    unk_id=3,
    pad_piece="[PAD]",
    bos_piece="[CLS]",
    eos_piece="[SEP]",
    unk_piece="[UNK]",
)
tokenizer = keras_nlp.models.DebertaV3Tokenizer(
    proto=bytes_io.getvalue(),
)
tokenizer("The quick brown fox jumped.")

[source]

from_preset method

DebertaV3Tokenizer.from_preset()

Instantiate DebertaV3Tokenizer tokenizer from preset vocabulary.

Arguments

  • preset: string. Must be one of "deberta_v3_extra_small_en", "deberta_v3_small_en", "deberta_v3_base_en", "deberta_v3_large_en", "deberta_v3_base_multi".

Examples

# Load a preset tokenizer.
tokenizer = DebertaV3Tokenizer.from_preset("deberta_v3_extra_small_en")

# Tokenize some input.
tokenizer("The quick brown fox tripped.")

# Detokenize some input.
tokenizer.detokenize([5, 6, 7, 8, 9])
Preset name Parameters Description
deberta_v3_extra_small_en 70.68M 12-layer DeBERTaV3 model where case is maintained. Trained on English Wikipedia, BookCorpus and OpenWebText.
deberta_v3_small_en 141.30M 6-layer DeBERTaV3 model where case is maintained. Trained on English Wikipedia, BookCorpus and OpenWebText.
deberta_v3_base_en 183.83M 12-layer DeBERTaV3 model where case is maintained. Trained on English Wikipedia, BookCorpus and OpenWebText.
deberta_v3_large_en 434.01M 24-layer DeBERTaV3 model where case is maintained. Trained on English Wikipedia, BookCorpus and OpenWebText.
deberta_v3_base_multi 278.22M 12-layer DeBERTaV3 model where case is maintained. Trained on the 2.5TB multilingual CC100 dataset.