ModernBertTextClassifier model

[source]

ModernBertTextClassifier class

keras_hub.models.ModernBertTextClassifier(
    backbone,
    num_classes,
    preprocessor=None,
    activation=None,
    hidden_dim=None,
    dropout=0.0,
    **kwargs
)

An end-to-end ModernBERT model for classification tasks.

This model attaches a classification head to a keras_hub.models.ModernBertBackbone instance.

The representation of the first token is used as the sequence-level representation for classification.

Arguments

  • backbone: A keras_hub.models.ModernBertBackbone instance.
  • num_classes: int. Number of classes to predict.
  • preprocessor: A keras_hub.models.ModernBertTextClassifierPreprocessor or None. If None, inputs should already be preprocessed.
  • activation: Optional str or callable. The activation function applied to the output. Set activation="softmax" to return probabilities.
  • hidden_dim: int. The size of the intermediate pooler layer. Defaults to the backbone hidden dimension.
  • dropout: float. Dropout probability applied to the pooled representation and classifier output.

Examples

import keras_hub
import numpy as np

# Fine-tune from a preset.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modern_bert_base_en",
    num_classes=2,
)
classifier.fit(
    x=["The quick brown fox jumped.", "Call me Ishmael."],
    y=np.array([1, 0]),
    batch_size=2,
)

# Predict on new text.
classifier.predict(["What an amazing movie!"])

[source]

from_preset method

ModernBertTextClassifier.from_preset(preset, load_weights=True, **kwargs)

Instantiate a keras_hub.models.Task from a model preset.

A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:

  1. a built-in preset identifier like 'bert_base_en'
  2. a Kaggle Models handle like 'kaggle://user/bert/keras/bert_base_en'
  3. a Hugging Face handle like 'hf://user/bert_base_en'
  4. a path to a local preset directory like './bert_base_en'

For any Task subclass, you can run cls.presets.keys() to list all built-in presets available on the class.

This constructor can be called in one of two ways. Either from a task specific base class like keras_hub.models.CausalLM.from_preset(), or from a model class like keras_hub.models.BertTextClassifier.from_preset(). If calling from the a base class, the subclass of the returning object will be inferred from the config in the preset directory.

Arguments

  • preset: string. A built-in preset identifier, a Kaggle Models handle, a Hugging Face handle, or a path to a local directory.
  • load_weights: bool. If True, saved weights will be loaded into the model architecture. If False, all weights will be randomly initialized.

Examples

# Load a Gemma generative task.
causal_lm = keras_hub.models.CausalLM.from_preset(
    "gemma_2b_en",
)

# Load a Bert classification task.
model = keras_hub.models.TextClassifier.from_preset(
    "bert_base_en",
    num_classes=2,
)
Preset Parameters Description
modern_bert_base_en 149.01M 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers.
modern_bert_embed_base 149.01M 22-layer ModernBERT Base sentence embedding model from Nomic AI (nomic-ai/modernbert-embed-base). Produces 768-dimensional, L2-normalized, mean-pooled embeddings for inputs of up to 8192 tokens. Prefix queries with 'search_query: ' and documents with 'search_document: '.
gte_modern_bert_base 149.01M 22-layer ModernBERT Base sentence embedding model from Alibaba-NLP (Alibaba-NLP/gte-modernbert-base). Produces 768-dimensional [CLS]-pooled embeddings that are not normalized, so compare them with cosine similarity. The model supports inputs of up to 8192 tokens.
modern_bert_large_en 394.78M 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feedforward layers.

backbone property

keras_hub.models.ModernBertTextClassifier.backbone

A keras_hub.models.Backbone model with the core architecture.


preprocessor property

keras_hub.models.ModernBertTextClassifier.preprocessor

A keras_hub.models.Preprocessor layer used to preprocess input.