1,900 EPL matches · Five seasons · Aggregate only
Premier League 1X2 Probability Calibration Benchmark
Direct answer: under the declared 10-bin fixed-width analysis, top-label ECE is 2.03% across 1,899 non-tied matches. Home, draw and away ECE are 1.92%, 0.91%, and 1.12%. These are closing-market results—not AI model performance or a same-time comparison with a forecast locked 24 hours before kick-off.
Published by Football Proof AI · Published · football-1x2-probability-calibration-benchmark/1.0.0Canonical publication record
Abstract
Aggregate top-label and classwise reliability bins, Wilson 95% intervals and ECE sensitivity for de-vigged closing average 1X2 market probabilities across 1,900 completed Premier League matches from 2021/22 through 2025/26.
- Author and publisher
- Football Proof AI
- Technical report
football-1x2-probability-calibration-benchmark/1.0.0- Published
- Last modified
- Release status
- Current release
- Review status
- Editorial technical note; not externally peer reviewed
- Version history
football-1x2-probability-calibration-benchmark/1.0.0: Initial public release.
- Immutable artifacts
- football-1x2-probability-calibration-benchmark-v1.csv
sha256:c4267695092083c539af874c1988e858199514122762b6efeaeca6d6c77615af - 1.0.0.json
sha256:04482e2eaaba871bee5330b78f4ccfe735e30bbd7ac7991c67df4ed3d3d9481c - 1.0.0-manifest.json
sha256:1a16037252ff5ccd2cecb405894a4605627770ea77030096f2bc5b4d17d392cc
- football-1x2-probability-calibration-benchmark-v1.csv
The question calibration actually answers
When the market said 60%, did the outcome happen about 60%?
Each point below represents a disclosed probability bin. Its horizontal position is the bin's mean forecast probability; its vertical position is the observed event frequency. Perfect empirical agreement sits on the diagonal. Distance from that line is descriptive calibration error, not automatically a tradable edge.
Top-label confidence
6 non-empty bins · n=1,899
| Bin | Probability range | n | Mean probability | Observed | Absolute gap | Wilson 95% CI |
|---|---|---|---|---|---|---|
| 3 | [30.0%, 40.0%) | 280 | 37.84% | 46.07% | 8.23% | 40.33%–51.92% |
| 4 | [40.0%, 50.0%) | 581 | 44.70% | 44.75% | 0.05% | 40.75%–48.81% |
| 5 | [50.0%, 60.0%) | 467 | 55.12% | 56.75% | 1.62% | 52.21%–61.17% |
| 6 | [60.0%, 70.0%) | 301 | 64.80% | 64.45% | 0.35% | 58.89%–69.65% |
| 7 | [70.0%, 80.0%) | 192 | 74.57% | 76.04% | 1.48% | 69.53%–81.53% |
| 8 | [80.0%, 90.0%) | 78 | 83.69% | 88.46% | 4.78% | 79.50%–93.81% |
Four views · One declared protocol
Overall confidence can hide a draw-specific error.
Top-label calibration asks whether the market's most likely outcome wins as often as its confidence implies. Classwise calibration treats home, draw and away as separate binary events, so one strong class cannot mask another weak class.
ECE · n=1,899 · 6 non-empty bins
ECE · n=1,900 · 9 non-empty bins
ECE · n=1,900 · 4 non-empty bins
ECE · n=1,900 · 9 non-empty bins
Classwise reliability tables
Inspect home, draw and away without collapsing the evidence.
Every classwise row uses the same 1,900-match cohort. The observed target is one when that outcome occurs and zero otherwise. Expand each table to inspect counts, probability spans, empirical rates and uncertainty together.
Home · one vs restECE 1.92% · MCE 7.92% · RMSCE 2.55%
| Bin | Probability range | n | Mean probability | Observed | Absolute gap | Wilson 95% CI |
|---|---|---|---|---|---|---|
| 0 | [0.0%, 10.0%) | 38 | 7.92% | 0.00% | 7.92% | 0.00%–9.18% |
| 1 | [10.0%, 20.0%) | 190 | 15.81% | 12.63% | 3.18% | 8.64%–18.11% |
| 2 | [20.0%, 30.0%) | 281 | 25.43% | 25.27% | 0.17% | 20.54%–30.66% |
| 3 | [30.0%, 40.0%) | 352 | 35.09% | 36.65% | 1.56% | 31.78%–41.80% |
| 4 | [40.0%, 50.0%) | 327 | 44.81% | 44.95% | 0.14% | 39.65%–50.37% |
| 5 | [50.0%, 60.0%) | 293 | 55.09% | 58.70% | 3.61% | 52.99%–64.19% |
| 6 | [60.0%, 70.0%) | 208 | 64.86% | 62.50% | 2.36% | 55.75%–68.80% |
| 7 | [70.0%, 80.0%) | 144 | 74.85% | 73.61% | 1.24% | 65.87%–80.13% |
| 8 | [80.0%, 90.0%) | 67 | 84.06% | 89.55% | 5.49% | 79.97%–94.85% |
Draw · one vs restECE 0.91% · MCE 8.59% · RMSCE 2.02%
| Bin | Probability range | n | Mean probability | Observed | Absolute gap | Wilson 95% CI |
|---|---|---|---|---|---|---|
| 0 | [0.0%, 10.0%) | 23 | 8.59% | 0.00% | 8.59% | 0.00%–14.31% |
| 1 | [10.0%, 20.0%) | 353 | 16.23% | 17.56% | 1.33% | 13.95%–21.88% |
| 2 | [20.0%, 30.0%) | 1,444 | 25.61% | 25.90% | 0.29% | 23.71%–28.22% |
| 3 | [30.0%, 40.0%) | 80 | 30.68% | 22.50% | 8.18% | 14.73%–32.79% |
Away · one vs restECE 1.12% · MCE 9.61% · RMSCE 2.00%
| Bin | Probability range | n | Mean probability | Observed | Absolute gap | Wilson 95% CI |
|---|---|---|---|---|---|---|
| 0 | [0.0%, 10.0%) | 162 | 7.08% | 6.79% | 0.29% | 3.83%–11.75% |
| 1 | [10.0%, 20.0%) | 396 | 15.48% | 14.90% | 0.58% | 11.73%–18.74% |
| 2 | [20.0%, 30.0%) | 407 | 25.01% | 23.59% | 1.42% | 19.72%–27.95% |
| 3 | [30.0%, 40.0%) | 355 | 34.68% | 34.37% | 0.31% | 29.62%–39.45% |
| 4 | [40.0%, 50.0%) | 254 | 44.56% | 44.49% | 0.07% | 38.50%–50.64% |
| 5 | [50.0%, 60.0%) | 174 | 55.17% | 53.45% | 1.72% | 46.04%–60.71% |
| 6 | [60.0%, 70.0%) | 93 | 64.64% | 68.82% | 4.17% | 58.81%–77.33% |
| 7 | [70.0%, 80.0%) | 48 | 73.72% | 83.33% | 9.61% | 70.42%–91.30% |
| 8 | [80.0%, 90.0%) | 11 | 81.42% | 81.82% | 0.40% | 52.30%–94.86% |
ECE is a summary, not a verdict
Publish the formula, the bins and the uncertainty.
Expected calibration error weights each bin's absolute forecast-frequency gap by its share of the sample. The same forecasts can receive a different ECE under different boundaries, so the release keeps the full bin table beside every summary.
- p̄ᵦ
- Mean stated probability in bin b
- oᵦ
- Observed event frequency in bin b
- nᵦ
- Non-empty bin sample count
- N
- Total observations in that view
Wilson 95% intervals describe binomial sampling uncertainty in each observed frequency. They do not guarantee independence across matches, prove the underlying probability lies inside a realised interval or turn a small ECE into a significance test.
Binning sensitivity
Fixed-width and equal-mass views must tell the same broad story.
Fixed-width bins preserve interpretable probability ranges but can be sparse at the edges. Equal-mass bins balance counts but move the boundaries; ties must stay together, which can reduce the effective bin count. Reporting both prevents one favourable partition from standing in for a robust conclusion.
| Strategy | Requested bins | Effective bins | Top-label ECE | Home ECE | Draw ECE | Away ECE |
|---|---|---|---|---|---|---|
| Fixed-width | 5 | Top 5 · H 5 · D 5 · A 5 | 1.92% | 1.87% | 0.27% | 1.12% |
| Fixed-width | 10 | Top 10 · H 10 · D 10 · A 10 | 2.03% | 1.92% | 0.91% | 1.12% |
| Fixed-width | 15 | Top 15 · H 15 · D 15 · A 15 | 2.71% | 3.52% | 0.92% | 1.67% |
| Fixed-width | 20 | Top 20 · H 20 · D 20 · A 20 | 2.70% | 3.64% | 1.09% | 1.81% |
| Equal-mass | 5 | Top 5 · H 5 · D 5 · A 5 | 2.14% | 1.36% | 1.83% | 0.89% |
| Equal-mass | 10 | Top 10 · H 10 · D 10 · A 10 | 2.91% | 1.71% | 2.39% | 1.89% |
| Equal-mass | 15 | Top 15 · H 14 · D 15 · A 15 | 3.12% | 3.22% | 2.52% | 2.41% |
| Equal-mass | 20 | Top 20 · H 20 · D 20 · A 20 | 4.83% | 3.89% | 3.76% | 3.09% |
Critical timing boundary
Closing prices know more than a forecast locked 24 hours earlier.
Football-Data labels AvgCH, AvgCD and AvgCAas average closing odds. The public rows do not provide one universal capture timestamp for every constituent bookmaker price. Closing prices may incorporate late injuries, confirmed line-ups, weather and market movement unavailable at the site's minimum 24-hour publication boundary.
Reproducible method
From quoted odds to an auditable reliability row.
- 01
Freeze the cohort
Use the same 1,900 completed EPL matches and declared source snapshots as the empirical scoring benchmark.
- 02
Remove the overround
For each match, compute inverse closing odds and divide each value by the three-way reciprocal sum.
- 03
Create four views
Audit unique top-label confidence plus one-vs-rest home, draw and away probabilities; disclose tie handling.
- 04
Bin twice
Apply fixed-width and tie-preserving equal-mass rules without choosing boundaries after seeing the result.
- 05
Keep denominators
Publish every non-empty bin's range, n, mean probability, observed count, frequency and Wilson interval.
- 06
Hash the release
Bind the page only to versioned aggregate JSON, CSV and a manifest that records exact source and artifact identities.
Downloads and rights boundary
Aggregates are downloadable; source rows are not republished.
Football-Data.co.uk supplied the public match and odds files under the column conventions in its source notes, not this analysis. The release publishes derived aggregate bins only. It does not assert a redistribution licence over the raw source rows and does not imply source endorsement or independent review.
- JSON SHA-256
04482e2eaaba871bee5330b78f4ccfe735e30bbd7ac7991c67df4ed3d3d9481c- CSV SHA-256
c4267695092083c539af874c1988e858199514122762b6efeaeca6d6c77615af- Manifest SHA-256
1a16037252ff5ccd2cecb405894a4605627770ea77030096f2bc5b4d17d392cc
Original aggregate analysis is offered under CC BY 4.0only to the extent Football Proof AI holds rights in it. Exact source URLs and their snapshot hashes belong in the versioned manifest. The public rights notice keeps source rights separate from the aggregate release. GitHub's immutable release attests the release tag and attached asset digests; the tag identifies the commit. That is supply-chain provenance, not independent validation.
Calibration benchmark FAQ
Five questions to ask before quoting one ECE number.
What does probability calibration mean in football?
Calibration compares stated probabilities with observed frequencies across a defined sample. If comparable forecasts average 60%, the event should occur about 60% of the time over a sufficiently large sample. It is a property of a collection of forecasts, not one match.
Does a low ECE prove that a football forecast is accurate?
No. Expected calibration error depends on the binning rule and measures reliability, not resolution, ranking skill, profitability or causal model quality. Read it with the full reliability table, Brier score, log loss, sample size and a time-safe baseline.
Why publish separate home, draw and away calibration curves?
One aggregate can hide class-specific failure. A market can look acceptable overall while systematically under- or over-stating draw probability. The one-vs-rest views expose that asymmetry.
Can closing market probabilities be compared fairly with a model published 24 hours before kick-off?
Not as a same-time contest. Closing prices can incorporate late team news, line-up information and market movement unavailable to a forecast locked at least 24 hours before kick-off. This dataset is a descriptive closing-market reference, not a fair launch gate for that earlier model.
Does this calibration dataset show betting profit?
No. The release evaluates probability reliability after removing the quoted overround. It does not model available stake size, price access, limits, commission, slippage or a betting strategy, and it is not a profit claim.