tumourkit.utils.calibration_error.calibration_error

tumourkit.utils.calibration_error.calibration_error(y_true, y_prob, sample_weight=None, norm='l2', n_bins=10, strategy='uniform', pos_label=None, reduce_bias=True)

Compute calibration error of a binary classifier. Across all items in a set of N predictions, the calibration error measures the aggregated difference between (1) the average predicted probabilities assigned to the positive class, and (2) the frequencies of the positive class in the actual outcome. The calibration error is only appropriate for binary categorical outcomes. Which label is considered to be the positive label is controlled via the parameter pos_label, which defaults to 1. Parameters ———- y_true : array-like of shape (n_samples,)

True targets of a binary classification task.

y_probarray-like of (n_samples,)

Probabilities of the positive class.

sample_weightarray-like of shape (n_samples,), default=None

Sample weights.

norm{‘l1’, ‘l2’, ‘max’}, default=’l2’

Norm method. The l1-norm is the Expected Calibration Error (ECE), and the max-norm corresponds to Maximum Calibration Error (MCE).

n_binsint, default=10

The number of bins to compute error on.

strategy{‘uniform’, ‘quantile’}, default=’uniform’

Strategy used to define the widths of the bins. uniform

All bins have identical widths.

quantile

All bins have the same number of points.

pos_labelint or str, default=None

Label of the positive class. If None, the maximum label is used as positive class.

reduce_biasbool, default=True

Add debiasing term as in Verified Uncertainty Calibration, A. Kumar. Only effective for the l2-norm.

Returns

scorefloat

calibration error

Examples

>>> import numpy as np
>>> from sklearn.metrics import calibration_error
>>> y_true = np.array([0, 0, 0, 1] + [0, 1, 1, 1])
>>> y_pred = np.array([0.25, 0.25, 0.25, 0.25] + [0.75, 0.75, 0.75, 0.75])
>>> calibration_error(y_true, y_pred, n_bins=5)
0.0
>>> calibration_error(
...     y_true, y_pred, n_bins=5, norm="max"
... )
0.0
>>> y_true = np.array([0, 0, 0, 0] + [1, 1, 1, 1])
>>> calibration_error(y_true, y_pred, n_bins=5)
0.25
>>> calibration_error(
...     y_true, y_pred, n_bins=5, norm="max"
... )
0.25