Similarity scores calibration
The similarity.calibration module offers tools to improve the interpretability and utility of similarity scores by calibrating them. Calibration allows similarity scores to be interpreted as probabilities, making them suitable for confidence assessments and thresholding. This also enables the effective ensemble of multiple scores by mapping them onto a common probabilistic scale.
- Calibration Methods:
LogisticCalibration: Uses logistic regression to map similarity scores to probabilities, providing a smooth and parametric approach to calibration.IsotonicCalibration: A non-parametric approach that fits isotonic regression. Conceptually similar to score binning.
- Visualization:
- The
reliability_diagramfunction allows for visual comparison of calibrated and uncalibrated scores. This tool is helpful for assessing calibration quality, as it visualizes how well the predicted probabilities align with observed outcomes.
- The
similarity.calibration
LogisticCalibration()
Performs logistic regression calibration.
fit(scores, hits)
Fit the logistic regression model to calibrate raw scores.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores |
array
|
Raw uncalibrated scores. |
required |
hits |
array
|
Ground truth binary labels. |
required |
predict(scores)
Predict calibrated scores using a fitted calibration model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores |
array
|
Raw uncalibrated scores. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
prediction |
array
|
Calibrated scores. |
IsotonicCalibration(interpolate=True, strict=True)
Performs isotonic regression calibration for ranking.
Compared to standard isotonic regression, this implementation uses spline interpolation to ensure that the calibration curve is strictly increasing, which is necessary for ranking.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interpolate |
bool
|
If True, use spline interpolation for calibration. |
True
|
strict |
bool
|
If True, apply strict adjustment to predictions. |
True
|
fit(scores, hits)
Fit the isotonic regression model to calibrate the scores.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores |
array
|
Raw uncalibrated scores. |
required |
hits |
array
|
Ground truth binary labels. |
required |
predict(scores)
Predict calibrated scores using a fitted calibration model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores |
array
|
Raw uncalibrated scores. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
calibrated_scores |
array
|
Calibrated scores. |
reliability_diagram(scores, hits, ax=None, skip_plot=False, num_bins=10, title='Reliability Diagram')
Calculates ECE (Expected calibration error) and plots reliability diagram for a given set of scores and hits.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores |
array
|
Raw uncalibrated scores. |
required |
hits |
array
|
Ground truth binary labels. |
required |
ax |
Axes
|
Axes to plot the diagram on. If None, a new figure is created. |
None
|
skip_plot |
bool
|
If True, only return ECE value. |
False
|
num_bins |
int
|
Number of bins to divide the scores into. |
10
|
title |
str
|
Title of the plot. |
'Reliability Diagram'
|
Returns:
| Name | Type | Description |
|---|---|---|
ece |
float
|
Expected Calibration Error. |
Examples
Example of isotonic regression calibration.
from wildlife_tools.similarity.calibration import IsotonicCalibration
calibration = IsotonicCalibration()
calibration.fit([0, 0.5, 1], [0, 1, 1])
calibration.predict([0, 0.25, 0.8])