tumourkit.utils.calibration_error
Calibration error implementation yet to be merged to sklearn.
Github Thread: https://github.com/scikit-learn/scikit-learn/pull/11096
Functions
Compute calibration error of a binary classifier. Across all items in a set of N predictions, the calibration error measures the aggregated difference between (1) the average predicted probabilities assigned to the positive class, and (2) the frequencies of the positive class in the actual outcome. The calibration error is only appropriate for binary categorical outcomes. Which label is considered to be the positive label is controlled via the parameter pos_label, which defaults to 1. Parameters ---------- y_true : array-like of shape (n_samples,) True targets of a binary classification task. y_prob : array-like of (n_samples,) Probabilities of the positive class. sample_weight : array-like of shape (n_samples,), default=None Sample weights. norm : {'l1', 'l2', 'max'}, default='l2' Norm method. The l1-norm is the Expected Calibration Error (ECE), and the max-norm corresponds to Maximum Calibration Error (MCE). n_bins : int, default=10 The number of bins to compute error on. strategy : {'uniform', 'quantile'}, default='uniform' Strategy used to define the widths of the bins. uniform All bins have identical widths. quantile All bins have the same number of points. pos_label : int or str, default=None Label of the positive class. If None, the maximum label is used as positive class. reduce_bias : bool, default=True Add debiasing term as in Verified Uncertainty Calibration, A. Kumar. Only effective for the l2-norm. Returns ------- score : float calibration error Examples -------- >>> import numpy as np >>> from sklearn.metrics import calibration_error >>> y_true = np.array([0, 0, 0, 1] + [0, 1, 1, 1]) >>> y_pred = np.array([0.25, 0.25, 0.25, 0.25] + [0.75, 0.75, 0.75, 0.75]) >>> calibration_error(y_true, y_pred, n_bins=5) 0.0 >>> calibration_error( ... y_true, y_pred, n_bins=5, norm="max" ... ) 0.0 >>> y_true = np.array([0, 0, 0, 0] + [1, 1, 1, 1]) >>> calibration_error(y_true, y_pred, n_bins=5) 0.25 >>> calibration_error( ... y_true, y_pred, n_bins=5, norm="max" ... ) 0.25. |