Using KerasCV COCO Metrics

Author: lukewood
Date created: 2022/04/13
Last modified: 2022/04/13
Description: Use KerasCV COCO metrics to evaluate object detection models.

View in Colab • GitHub source

Overview

With KerasCV's COCO metrics implementation, you can easily evaluate your object detection model's performance all from within the TensorFlow graph. This guide shows you how to use KerasCV's COCO metrics and integrate it into your own model evaluation pipeline. Historically, users have evaluated COCO metrics as a post training step. KerasCV offers an in graph implementation of COCO metrics, enabling users to evaluate COCO metrics during training!

Let's get started using KerasCV's COCO metrics.

Input format

All KerasCV components that process bounding boxes, including COCO metrics, require a bounding_box_format parameter. This parameter is used to tell the components what format your bounding boxes are in. While this guide uses the xyxy format, a full list of supported formats is available in the bounding_box API documentation.

The metrics expect y_true and be a float Tensor with the shape [batch, num_images, num_boxes, 5], with the ordering of last set of axes determined by the provided format. The same is true of y_pred, except that an additional confidence axis must be provided.

Due to the fact that each image may have a different number of bounding boxes, the num_boxes dimension may actually have a mismatching shape between images. KerasCV works around this by allowing you to either pass a RaggedTensor as an input to the KerasCV COCO metrics, or padding unused bounding boxes with -1.

Utility functions to manipulate bounding boxes, transform between formats, and pad bounding box Tensors with -1s are available from the keras_cv.bounding_box package.

Independent metric use

The usage first pattern for KerasCV COCO metrics is to manually call update_state() and result() methods. This pattern is recommended for users who want finer grained control of their metric evaluation, or want to use a different format for y_pred in their model.

Let's run through a quick code example.

1.) First, we must construct our metric:

import keras_cv

# import all modules we will need in this example
import tensorflow as tf
from tensorflow import keras

# only consider boxes with areas less than a 32x32 square.
metric = keras_cv.metrics.COCORecall(
    bounding_box_format="xyxy", class_ids=[1, 2, 3], area_range=(0, 32**2)
)

2.) Create Some Bounding Boxes:

y_true = tf.ragged.stack(
    [
        # image 1
        tf.constant([[0, 0, 10, 10, 1], [11, 12, 30, 30, 2]], tf.float32),
        # image 2
        tf.constant([[0, 0, 10, 10, 1]], tf.float32),
    ]
)
y_pred = tf.ragged.stack(
    [
        # predictions for image 1
        tf.constant([[5, 5, 10, 10, 1, 0.9]], tf.float32),
        # predictions for image 2
        tf.constant([[0, 0, 10, 10, 1, 1.0], [5, 5, 10, 10, 1, 0.9]], tf.float32),
    ]
)

3.) Update metric state:

metric.update_state(y_true, y_pred)

4.) Evaluate the result:

metric.result()

<tf.Tensor: shape=(), dtype=float32, numpy=0.25>

Evaluating COCORecall for your object detection model is as simple as that!

Metric use in a model

You can also leverage COCORecall in your model's training loop. Let's walk through this process.

1.) Construct your the metric and a dummy model

i = keras.layers.Input((None, 6))
model = keras.Model(i, i)

2.) Create some fake bounding boxes:

y_true = tf.constant([[[0, 0, 10, 10, 1], [5, 5, 10, 10, 1]]], tf.float32)
y_pred = tf.constant([[[0, 0, 10, 10, 1, 1.0], [5, 5, 10, 10, 1, 0.9]]], tf.float32)

3.) Create the metric and compile the model

recall = keras_cv.metrics.COCORecall(
    bounding_box_format="xyxy",
    max_detections=100,
    class_ids=[1],
    area_range=(0, 64**2),
    name="coco_recall",
)
model.compile(metrics=[recall])

4.) Use model.evaluate() to evaluate the metric

model.evaluate(y_pred, y_true, return_dict=True)

1/1 [==============================] - 1s 1s/step - loss: 0.0000e+00 - coco_recall: 1.0000

{'loss': 0.0, 'coco_recall': 1.0}

Looks great! That's all it takes to use KerasCV's COCO metrics to evaluate object detection models.

Supported constructor parameters

KerasCV COCO Metrics are sufficiently parameterized to support all of the permutations evaluated in the original COCO challenge, all metrics evaluated in the accompanying pycocotools library, and more!

Check out the full documentation for COCORecall and COCOMeanAveragePrecision.

Conclusion & next steps

KerasCV makes it easier than ever before to evaluate a Keras object detection model. Historically, users had to perform post training evaluation. With KerasCV, you can perform train time evaluation to see how these metrics evolve over time!

As an additional exercise for readers, you can:

• Configure iou_thresholds, max_detections, and area_range to reproduce the suite of metrics evaluted in pycocotools
• Integrate COCO metrics into a RetinaNet using the keras.io RetinaNet example