FNetEncoder layer

[source]

FNetEncoder class

keras_nlp.layers.FNetEncoder(
    intermediate_dim,
    dropout=0,
    activation="relu",
    layer_norm_epsilon=1e-05,
    kernel_initializer="glorot_uniform",
    bias_initializer="zeros",
    name=None,
    **kwargs
)

FNet encoder.

This class follows the architecture of FNet encoder layer in the FNet paper. Users can instantiate multiple instances of this class to stack up the encoder.

Note on masking: In the official FNet code, padding tokens are added to the the input. However, the padding masks are deleted, i.e., mixing of all tokens is done. This is because certain frequencies will be zeroed out if we apply padding masks in every encoder layer. Hence, we don't take padding mask as input in the call() function.

Arguments

• intermediate_dim: int. The hidden size of feedforward network.
• dropout: float, defaults to 0. The dropout value, applied in the feedforward network.
• activation: string or keras.activations, defaults to "relu". The activation function of feedforward network.
• layer_norm_epsilon: float, defaults to 1e-5. The epsilon value in layer normalization components.
• kernel_initializer: "string" or keras.initializers initializer, defaults to "glorot_uniform". The kernel initializer for the dense layers.
• bias_initializer: "string" or keras.initializers initializer, defaults to "zeros". The bias initializer for the dense layers.
• name: string, defaults to None. The name of the layer.
• **kwargs: other keyword arguments.

Examples

# Create a single FNet encoder layer.
encoder = keras_nlp.layers.FNetEncoder(
    intermediate_dim=64)

# Create a simple model containing the encoder.
input = keras.Input(shape=[10, 64])
output = encoder(input)
model = keras.Model(inputs=input, outputs=output)

# Call encoder on the inputs.
input_data = tf.random.uniform(shape=[1, 10, 64])
output = model(input_data)

References

• Lee-Thorp et al., 2021