MeanReciprocalRank metric

[source]

MeanReciprocalRank class

keras_rs.metrics.MeanReciprocalRank(
    k: Optional[int] = None,
    shuffle_ties: bool = True,
    seed: Union[int, keras.src.random.seed_generator.SeedGenerator, NoneType] = None,
    **kwargs: Any
)

Computes Mean Reciprocal Rank (MRR).

This metric evaluates ranking quality. It focuses on the rank position of the single highest-scoring relevant item. The metric processes true relevance labels in y_true (binary indicators (0 or 1) of relevance) against predicted scores in y_pred. The scores in y_pred are used to determine the rank order of items, by sorting in descending order. Scores range from 0 to 1, with 1 indicating the first relevant item was always ranked first.

For each list of predicted scores s in y_pred and the corresponding list of true labels y in y_true, the per-query MRR score is calculated as follows:

MRR(y, s) = max_{i} y_{i} / rank(s_{i})

The final MRR score reported is typically the weighted average of these per-query scores across all queries/lists in the dataset.

Note: sample_weight is handled differently for ranking metrics. For batched inputs, sample_weight can be scalar, 1D, 2D. The scalar case and 1D case (list-wise weights) are straightforward. The 2D case (item- wise weights) is different, in the sense that the sample weights are aggregated to get 1D weights. For more details, refer to keras_rs.src.metrics.ranking_metrics_utils.get_list_weights.

Arguments

• k: int. The number of top-ranked items to consider (the 'k' in 'top-k'). Must be a positive integer.
• shuffle_ties: bool. Whether to randomly shuffle scores before sorting. This is done to break ties. Defaults to True.
• seed: int. Random seed used for shuffling.
• name: Optional name for the loss instance.
• dtype: The dtype of the metric's computations. Defaults to None, which means using keras.backend.floatx(). keras.backend.floatx() is a "float32" unless set to different value (via keras.backend.set_floatx()). If a keras.DTypePolicy is provided, then the compute_dtype will be utilized.

Example

>>> batch_size = 2
>>> list_size = 5
>>> labels = np.random.randint(0, 2, size=(batch_size, list_size))
>>> scores = np.random.random(size=(batch_size, list_size))
>>> metric = keras_rs.metrics.MeanReciprocalRank()(
...     y_true=labels, y_pred=scores
... )

Mask certain elements (can be used for uneven inputs):

>>> batch_size = 2
>>> list_size = 5
>>> labels = np.random.randint(0, 2, size=(batch_size, list_size))
>>> scores = np.random.random(size=(batch_size, list_size))
>>> mask = np.random.randint(0, 2, size=(batch_size, list_size), dtype=bool)
>>> metric = keras_rs.metrics.MeanReciprocalRank()(
...     y_true={"labels": labels, "mask": mask}, y_pred=scores
... )