Start with evidence integrity
A calibration curve cannot repair a selected or late record
The lab reuses the same strict seven-field contract as the complete-record Accuracy Audit. Duplicate, malformed and post-kick-off rows are excluded before any frequency is counted.
- match_id
- Unique stable identifier; duplicates are excluded
- published_at
- ISO 8601 time with Z or a numeric UTC offset
- kickoff_at
- Must be strictly later than publication
- p_home / p_draw / p_away
- One 0–1 or 0–100 scale whose three values sum correctly
- outcome
- H, D, A, home, draw or away
- SHA-256
- Binds the audit to the same bytes; it is not proof of publication time
Download the synthetic calibration example CSV. It demonstrates the input and chart contract, not model accuracy or live performance.
Different questions, separate curves
Top-label confidence is not classwise calibration
Top-label calibration asks whether the most likely pick wins as often as its confidence implies. Conditioned views split that question by the predicted class. One-vs-rest views instead ask whether each home, draw or away probability matches that outcome's frequency across every accepted match.
- 01
Top label
Maximum probability versus whether the selected result occurred.
- 02
Predicted H / D / A
The same confidence test, restricted to each selected class.
- 03
One versus rest
Every probability for one class versus whether that class occurred.
- 04
Never average blindly
A good overall curve can hide a weak draw or away subpopulation.
Binning is an assumption
If the conclusion changes with the bins, publish that sensitivity
Equal-width bins hold probability ranges constant but can leave sparse tails. Equal-mass bins aim for similar support and never split identical probabilities across two groups. The lab runs both strategies at 5, 10, 15 and 20 requested bins, retaining effective-bin counts and the smallest supporting sample.
ECE weights absolute bin gaps; signed calibration error preserves direction; RMSCE emphasizes larger gaps; MCE shows the largest occupied-bin gap. All four are sample and bin dependent. None is a universal certificate of a good model.
Make support visible
Every plotted frequency keeps n and its Wilson interval
The diagonal marks perfect sample calibration. A point shows one occupied bin's mean forecast probability and observed frequency. Its vertical Wilson 95% interval exposes finite-bin uncertainty, while the table preserves the exact counts behind the graphic.
Wilson intervals still assume independent Bernoulli observations. Repeated teams, seasons and temporal clusters can make them too optimistic. Use the sample-size lab for planning, and the walk-forward protocol for time-ordered evaluation.
Interpretation boundary
Calibration diagnostics do not replace proper scoring rules
ECE-like summaries can be improved by changing bins and do not reward sharp, truthful forecasts in the same way as a proper score. Compare probability error with the Brier decomposition lab, and inspect every exclusion before comparing models.
- No same-sample recalibration.This page diagnoses supplied probabilities; it does not fit isotonic or temperature mappings to the evaluated outcomes.
- No publication proof.The local hash identifies bytes but cannot establish when the CSV first existed.
- No binary verdict.Small samples, class imbalance and bin sensitivity prevent a universal pass/fail threshold.
- No betting conclusion.Calibration alone does not establish bookmaker value, profit or future performance.
Primary literature
Calibration metrics need their limitations attached
These sources ground the diagnostic choices. They do not certify this implementation or any user-supplied history.
- Guo et al. (2017), calibration and ECE.Proceedings paper
- Nixon et al. (2019), binning and class-choice limitations.Research paper
- Gupta & Ramdas (2022), top-label calibration.OpenReview paper
- Wilson (1927), binomial interval.Journal article DOI
- Gneiting & Raftery (2007), proper scoring rules.Journal article DOI
Cite this tool as: Football Proof AI. “Football Prediction Calibration & Reliability Diagram Lab.” Version football-prediction-calibration/1.0.0, 13 July 2026, https://footballproofai.com/tools/football-prediction-calibration-calculator. Editorial technical note; not externally peer reviewed.