StartEndPacker
classkeras_nlp.layers.StartEndPacker(
sequence_length,
start_value=None,
end_value=None,
pad_value=None,
return_padding_mask=False,
name=None,
**kwargs
)
Adds start and end tokens to a sequence and pads to a fixed length.
This layer is useful when tokenizing inputs for tasks like translation,
where each sequence should include a start and end marker. It should
be called after tokenization. The layer will first trim inputs to fit, then
add start/end tokens, and finally pad, if necessary, to sequence_length
.
Input data should be passed as tensors, tf.RaggedTensor
s, or lists. For
batched input, inputs should be a list of lists or a rank two tensor. For
unbatched inputs, each element should be a list or a rank one tensor.
Arguments
None
, no start value will be
added.None
, no end value
will be added.None
,
0 or "" will be added depending on the dtype of the input tensor.pad_value
.Call arguments
tf.Tensor
, tf.RaggedTensor
, or list of python strings.sequence_length
of
the layer.False
to not append a start value for this
input.False
to not append an end value for this
input.Examples
Unbatched input (int).
>>> inputs = [5, 6, 7]
>>> start_end_packer = keras_nlp.layers.StartEndPacker(
... sequence_length=7, start_value=1, end_value=2,
... )
>>> outputs = start_end_packer(inputs)
>>> np.array(outputs)
array([1, 5, 6, 7, 2, 0, 0], dtype=int32)
Batched input (int).
>>> inputs = [[5, 6, 7], [8, 9, 10, 11, 12, 13, 14]]
>>> start_end_packer = keras_nlp.layers.StartEndPacker(
... sequence_length=6, start_value=1, end_value=2,
... )
>>> outputs = start_end_packer(inputs)
>>> np.array(outputs)
array([[ 1, 5, 6, 7, 2, 0],
[ 1, 8, 9, 10, 11, 2]], dtype=int32)
Unbatched input (str).
>>> inputs = tf.constant(["this", "is", "fun"])
>>> start_end_packer = keras_nlp.layers.StartEndPacker(
... sequence_length=6, start_value="<s>", end_value="</s>",
... pad_value="<pad>"
... )
>>> outputs = start_end_packer(inputs)
>>> np.array(outputs).astype("U")
array(['<s>', 'this', 'is', 'fun', '</s>', '<pad>'], dtype='<U5')
Batched input (str).
>>> inputs = tf.ragged.constant([["this", "is", "fun"], ["awesome"]])
>>> start_end_packer = keras_nlp.layers.StartEndPacker(
... sequence_length=6, start_value="<s>", end_value="</s>",
... pad_value="<pad>"
... )
>>> outputs = start_end_packer(inputs)
>>> np.array(outputs).astype("U")
array([['<s>', 'this', 'is', 'fun', '</s>', '<pad>'],
['<s>', 'awesome', '</s>', '<pad>', '<pad>', '<pad>']], dtype='<U7')
Multiple start tokens.
>>> inputs = tf.ragged.constant([["this", "is", "fun"], ["awesome"]])
>>> start_end_packer = keras_nlp.layers.StartEndPacker(
... sequence_length=6, start_value=["</s>", "<s>"], end_value="</s>",
... pad_value="<pad>"
... )
>>> outputs = start_end_packer(inputs)
>>> np.array(outputs).astype("U")
array([['</s>', '<s>', 'this', 'is', 'fun', '</s>'],
['</s>', '<s>', 'awesome', '</s>', '<pad>', '<pad>']], dtype='<U7')