MPNetBackbone classkeras_hub.models.MPNetBackbone(
vocabulary_size,
num_layers,
num_heads,
hidden_dim,
intermediate_dim,
dropout=0.1,
max_sequence_length=512,
layer_norm_epsilon=1e-05,
dtype=None,
**kwargs
)
An MPNet encoder network.
This network implements the bi-directional Transformer encoder described in "MPNet: Masked and Permuted Pre-training for Language Understanding". It includes the embedding lookups and transformer layers, but does not include the masked and permuted language modeling head used during pretraining.
MPNet combines learned absolute position embeddings with a bucketed relative position bias, which is shared by all transformer layers.
The default constructor gives a fully customizable, randomly initialized
MPNet encoder with any number of layers, heads, and embedding dimensions.
To load preset architectures and weights, use the from_preset()
constructor.
Disclaimer: Pre-trained models are provided on an "as is" basis, without warranties or conditions of any kind. The underlying model is provided by a third party and subject to a separate license, available here.
Arguments
0.1.512.1e-5.keras.mixed_precision.DTypePolicy. The dtype to use
for model computations and weights. Note that some computations,
such as softmax and layer normalization, will always be done at
float32 precision regardless of dtype.Examples
input_data = {
"token_ids": np.ones(shape=(1, 12), dtype="int32"),
"padding_mask": np.array(
[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]], dtype="int32"
),
}
# Pretrained MPNet encoder.
model = keras_hub.models.MPNetBackbone.from_preset("all_mpnet_base_v2")
model(input_data)
# Randomly initialized MPNet encoder with a custom config.
model = keras_hub.models.MPNetBackbone(
vocabulary_size=30527,
num_layers=4,
num_heads=4,
hidden_dim=256,
intermediate_dim=512,
max_sequence_length=128,
)
model(input_data)
from_preset methodMPNetBackbone.from_preset(preset, load_weights=True, **kwargs)
Instantiate a keras_hub.models.Backbone from a model preset.
A preset is a directory of configs, weights and other file assets used
to save and load a pre-trained model. The preset can be passed as a
one of:
'bert_base_en''kaggle://user/bert/keras/bert_base_en''hf://user/bert_base_en''modelscope://user/bert_base_en''./bert_base_en'This constructor can be called in one of two ways. Either from the base
class like keras_hub.models.Backbone.from_preset(), or from
a model class like keras_hub.models.GemmaBackbone.from_preset().
If calling from the base class, the subclass of the returning object
will be inferred from the config in the preset directory.
For any Backbone subclass, you can run cls.presets.keys() to list
all built-in presets available on the class.
Arguments
True, the weights will be loaded into the
model architecture. If False, the weights will be randomly
initialized.Examples
# Load a Gemma backbone with pre-trained weights.
model = keras_hub.models.Backbone.from_preset(
"gemma_2b_en",
)
# Load a Bert backbone with a pre-trained config and random weights.
model = keras_hub.models.Backbone.from_preset(
"bert_base_en",
load_weights=False,
)
| Preset | Parameters | Description |
|---|---|---|
| all_mpnet_base_v2 | 108.89M | 12-layer MPNet sentence embedding model. Maps sentences and paragraphs to 768-dimensional normalized vectors. Trained on 1B+ sentence pairs for semantic search, clustering, and similarity. |
| nli_mpnet_base_v2 | 108.89M | 12-layer MPNet sentence embedding model trained on natural language inference (NLI) data. Maps sentences to 768-dimensional dense vectors that are not normalized. Inputs are truncated to 75 tokens. |
| paraphrase_mpnet_base_v2 | 108.89M | 12-layer MPNet sentence embedding model trained on paraphrase data. Maps sentences and paragraphs to 768-dimensional dense vectors that are not normalized, for paraphrase mining, clustering, and semantic search. |
token_embedding propertykeras_hub.models.MPNetBackbone.token_embedding
A keras.layers.Embedding instance for embedding token ids.
This layer embeds integer token ids to the hidden dim of the model.