AUC
classtf.keras.metrics.AUC(
num_thresholds=200,
curve="ROC",
summation_method="interpolation",
name=None,
dtype=None,
thresholds=None,
multi_label=False,
label_weights=None,
)
Computes the approximate AUC (Area under the curve) via a Riemann sum.
This metric creates four local variables, true_positives
, true_negatives
,
false_positives
and false_negatives
that are used to compute the AUC.
To discretize the AUC curve, a linearly spaced set of thresholds is used to
compute pairs of recall and precision values. The area under the ROC-curve is
therefore computed using the height of the recall values by the false positive
rate, while the area under the PR-curve is the computed using the height of
the precision values by the recall.
This value is ultimately returned as auc
, an idempotent operation that
computes the area under a discretized curve of precision versus recall values
(computed using the aforementioned variables). The num_thresholds
variable
controls the degree of discretization with larger numbers of thresholds more
closely approximating the true AUC. The quality of the approximation may vary
dramatically depending on num_thresholds
. The thresholds
parameter can be
used to manually specify thresholds which split the predictions more evenly.
For best results, predictions
should be distributed approximately uniformly
in the range [0, 1] and not peaked around 0 or 1. The quality of the AUC
approximation may be poor if this is not the case. Setting summation_method
to 'minoring' or 'majoring' can help quantify the error in the approximation
by providing lower or upper bound estimate of the AUC.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
ROC
.
For PR-AUC, interpolates (true/false) positives but not the ratio that
is precision (see Davis & Goadrich 2006 for details);
'minoring' applies left summation
for increasing intervals and right summation for decreasing intervals;
'majoring' does the opposite.num_thresholds
parameter is ignored. Values should be in [0, 1]. Endpoint thresholds
equal to {-epsilon, 1+epsilon} for a small positive epsilon value will
be automatically included with these to correctly handle predictions
equal to exactly 0 or 1.multi_label
is True,
the weights are applied to the individual label AUCs when they are
averaged to produce the multi-label AUC. When it's False, they are used
to weight the individual label predictions in computing the confusion
matrix on the flattened data. Note that this is unlike class_weights in
that class_weights weights the example depending on the value of its
label, whereas label_weights depends only on the index of that label
before flattening; therefore label_weights
should not be used for
multi-class data.Standalone usage:
>>> m = tf.keras.metrics.AUC(num_thresholds=3)
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9])
>>> # threshold values are [0 - 1e-7, 0.5, 1 + 1e-7]
>>> # tp = [2, 1, 0], fp = [2, 0, 0], fn = [0, 1, 2], tn = [0, 2, 2]
>>> # recall = [1, 0.5, 0], fp_rate = [1, 0, 0]
>>> # auc = ((((1+0.5)/2)*(1-0))+ (((0.5+0)/2)*(0-0))) = 0.75
>>> m.result().numpy()
0.75
>>> m.reset_states()
>>> m.update_state([0, 0, 1, 1], [0, 0.5, 0.3, 0.9],
... sample_weight=[1, 0, 0, 1])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd', loss='mse', metrics=[tf.keras.metrics.AUC()])
Precision
classtf.keras.metrics.Precision(
thresholds=None, top_k=None, class_id=None, name=None, dtype=None
)
Computes the precision of the predictions with respect to the labels.
The metric creates two local variables, true_positives
and false_positives
that are used to compute the precision. This value is ultimately returned as
precision
, an idempotent operation that simply divides true_positives
by the sum of true_positives
and false_positives
.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If top_k
is set, we'll calculate precision as how often on average a class
among the top-k classes with the highest predicted values of a batch entry is
correct and can be found in the label for that entry.
If class_id
is specified, we calculate precision by considering only the
entries in the batch for which class_id
is above the threshold and/or in the
top-k highest predictions, and computing the fraction of them for which
class_id
is indeed a correct label.
Arguments
true
, below is false
). One metric value is generated
for each threshold value. If neither thresholds nor top_k are set, the
default is to calculate precision with thresholds=0.5
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Standalone usage:
>>> m = tf.keras.metrics.Precision()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result().numpy()
0.6666667
>>> m.reset_states()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
>>> # With top_k=2, it will calculate precision over y_true[:2] and y_pred[:2]
>>> m = tf.keras.metrics.Precision(top_k=2)
>>> m.update_state([0, 0, 1, 1], [1, 1, 1, 1])
>>> m.result().numpy()
0.0
>>> # With top_k=4, it will calculate precision over y_true[:4] and y_pred[:4]
>>> m = tf.keras.metrics.Precision(top_k=4)
>>> m.update_state([0, 0, 1, 1], [1, 1, 1, 1])
>>> m.result().numpy()
0.5
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.Precision()])
Recall
classtf.keras.metrics.Recall(
thresholds=None, top_k=None, class_id=None, name=None, dtype=None
)
Computes the recall of the predictions with respect to the labels.
This metric creates two local variables, true_positives
and
false_negatives
, that are used to compute the recall. This value is
ultimately returned as recall
, an idempotent operation that simply divides
true_positives
by the sum of true_positives
and false_negatives
.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
If top_k
is set, recall will be computed as how often on average a class
among the labels of a batch entry is in the top-k predictions.
If class_id
is specified, we calculate recall by considering only the
entries in the batch for which class_id
is in the label, and computing the
fraction of them for which class_id
is above the threshold and/or in the
top-k predictions.
Arguments
true
, below is false
). One metric value is generated
for each threshold value. If neither thresholds nor top_k are set, the
default is to calculate recall with thresholds=0.5
.[0, num_classes)
, where
num_classes
is the last dimension of predictions.Standalone usage:
>>> m = tf.keras.metrics.Recall()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result().numpy()
0.6666667
>>> m.reset_states()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.Recall()])
TruePositives
classtf.keras.metrics.TruePositives(thresholds=None, name=None, dtype=None)
Calculates the number of true positives.
If sample_weight
is given, calculates the sum of the weights of
true positives. This metric creates one local variable, true_positives
that is used to keep track of the number of true positives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
true
, below is false
). One metric
value is generated for each threshold value.Standalone usage:
>>> m = tf.keras.metrics.TruePositives()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1])
>>> m.result().numpy()
2.0
>>> m.reset_states()
>>> m.update_state([0, 1, 1, 1], [1, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.TruePositives()])
TrueNegatives
classtf.keras.metrics.TrueNegatives(thresholds=None, name=None, dtype=None)
Calculates the number of true negatives.
If sample_weight
is given, calculates the sum of the weights of
true negatives. This metric creates one local variable, accumulator
that is used to keep track of the number of true negatives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
true
, below is false
). One metric
value is generated for each threshold value.Standalone usage:
>>> m = tf.keras.metrics.TrueNegatives()
>>> m.update_state([0, 1, 0, 0], [1, 1, 0, 0])
>>> m.result().numpy()
2.0
>>> m.reset_states()
>>> m.update_state([0, 1, 0, 0], [1, 1, 0, 0], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.TrueNegatives()])
FalsePositives
classtf.keras.metrics.FalsePositives(thresholds=None, name=None, dtype=None)
Calculates the number of false positives.
If sample_weight
is given, calculates the sum of the weights of
false positives. This metric creates one local variable, accumulator
that is used to keep track of the number of false positives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
true
, below is false
). One metric
value is generated for each threshold value.Standalone usage:
>>> m = tf.keras.metrics.FalsePositives()
>>> m.update_state([0, 1, 0, 0], [0, 0, 1, 1])
>>> m.result().numpy()
2.0
>>> m.reset_states()
>>> m.update_state([0, 1, 0, 0], [0, 0, 1, 1], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.FalsePositives()])
FalseNegatives
classtf.keras.metrics.FalseNegatives(thresholds=None, name=None, dtype=None)
Calculates the number of false negatives.
If sample_weight
is given, calculates the sum of the weights of
false negatives. This metric creates one local variable, accumulator
that is used to keep track of the number of false negatives.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
true
, below is false
). One metric
value is generated for each threshold value.Standalone usage:
>>> m = tf.keras.metrics.FalseNegatives()
>>> m.update_state([0, 1, 1, 1], [0, 1, 0, 0])
>>> m.result().numpy()
2.0
>>> m.reset_states()
>>> m.update_state([0, 1, 1, 1], [0, 1, 0, 0], sample_weight=[0, 0, 1, 0])
>>> m.result().numpy()
1.0
Usage with compile()
API:
model.compile(optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.FalseNegatives()])
PrecisionAtRecall
classtf.keras.metrics.PrecisionAtRecall(
recall, num_thresholds=200, name=None, dtype=None
)
Computes best precision where recall is >= specified value.
This metric creates four local variables, true_positives
, true_negatives
,
false_positives
and false_negatives
that are used to compute the
precision at the given recall. The threshold for the given recall
value is computed and used to evaluate the corresponding precision.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
Arguments
[0, 1]
.Standalone usage:
>>> m = tf.keras.metrics.PrecisionAtRecall(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result().numpy()
0.5
>>> m.reset_states()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[2, 2, 2, 1, 1])
>>> m.result().numpy()
0.33333333
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.PrecisionAtRecall(recall=0.8)])
SensitivityAtSpecificity
classtf.keras.metrics.SensitivityAtSpecificity(
specificity, num_thresholds=200, name=None, dtype=None
)
Computes best sensitivity where specificity is >= specified value.
the sensitivity at a given specificity.
Sensitivity
measures the proportion of actual positives that are correctly
identified as such (tp / (tp + fn)).
Specificity
measures the proportion of actual negatives that are correctly
identified as such (tn / (tn + fp)).
This metric creates four local variables, true_positives
, true_negatives
,
false_positives
and false_negatives
that are used to compute the
sensitivity at the given specificity. The threshold for the given specificity
value is computed and used to evaluate the corresponding sensitivity.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
For additional information about specificity and sensitivity, see the following.
Arguments
[0, 1]
.Standalone usage:
>>> m = tf.keras.metrics.SensitivityAtSpecificity(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result().numpy()
0.5
>>> m.reset_states()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[1, 1, 2, 2, 1])
>>> m.result().numpy()
0.333333
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.SensitivityAtSpecificity()])
SpecificityAtSensitivity
classtf.keras.metrics.SpecificityAtSensitivity(
sensitivity, num_thresholds=200, name=None, dtype=None
)
Computes best specificity where sensitivity is >= specified value.
Sensitivity
measures the proportion of actual positives that are correctly
identified as such (tp / (tp + fn)).
Specificity
measures the proportion of actual negatives that are correctly
identified as such (tn / (tn + fp)).
This metric creates four local variables, true_positives
, true_negatives
,
false_positives
and false_negatives
that are used to compute the
specificity at the given sensitivity. The threshold for the given sensitivity
value is computed and used to evaluate the corresponding specificity.
If sample_weight
is None
, weights default to 1.
Use sample_weight
of 0 to mask values.
For additional information about specificity and sensitivity, see the following.
Arguments
[0, 1]
.Standalone usage:
>>> m = tf.keras.metrics.SpecificityAtSensitivity(0.5)
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8])
>>> m.result().numpy()
0.66666667
>>> m.reset_states()
>>> m.update_state([0, 0, 0, 1, 1], [0, 0.3, 0.8, 0.3, 0.8],
... sample_weight=[1, 1, 2, 2, 2])
>>> m.result().numpy()
0.5
Usage with compile()
API:
model.compile(
optimizer='sgd',
loss='mse',
metrics=[tf.keras.metrics.SpecificityAtSensitivity()])