Synthetic fixtures · Non-empirical · Versioned
An open conformance benchmark for football prediction scores
Direct answer: this dataset tests whether an implementation reproduces 54 declared 1X2 scoring cases. It is not real match data, a model leaderboard or evidence that any football prediction system is accurate.
Published by Football Proof AI · Published · football-1x2-scoring-benchmark/1.0.0 · Editorial technical note; not externally peer reviewedCanonical publication record
Abstract
An open, deterministic conformance dataset of 54 synthetic football 1X2 cases with expected top-pick, summed and class-averaged Brier, exact and explicitly clipped log-loss, and normalized H-D-A Ranked Probability Score outputs.
- Author and publisher
- Football Proof AI
- Technical report
football-1x2-scoring-benchmark/1.0.0- Published
- Last modified
- Release status
- Current release
- Review status
- Editorial technical note; not externally peer reviewed
- Version history
football-1x2-scoring-benchmark/1.0.0: Initial public release.
- Immutable artifacts
- football-1x2-scoring-benchmark-v1.csv
sha256:a2d7dc9aec8faae9b62562b5f50de68f3e6bb787c2619539c057e6beeb4d7128 - 1.0.0.json
sha256:2481c8c2d42e88333fb2b92494a023692830e8e560dad146f132d45d5dc3fc74
- football-1x2-scoring-benchmark-v1.csv
- References
54 fixed cases · Two immutable formats
Download the same benchmark bytes every time
Eighteen probability vectors are crossed with home, draw and away outcomes. Both files declare the formulas, tie rule, numerical boundary and release version; response headers expose their SHA-256 byte identities.
License: CC BY 4.0. Cite the exact version and preserve the artifact SHA-256 when results must be reproducible.
Truth boundary
Conformance is narrower than model validation
Matching the expected cells supports only the claim that code reproduced these cases under these conventions. It cannot prove historical publication, archive completeness, calibration, predictive skill, bookmaker advantage or profit.
- 01
Synthetic
No row names a real fixture, team, league, provider, price or trained model.
- 02
Non-empirical
The cases reveal mathematical behaviour; they estimate no population performance.
- 03
No metric winner
A paired example can show disagreement without proving one scoring rule is statistically superior.
- 04
No external review
The version and hash identify this publication; they do not create independent replication or peer review.
Representative rows
Where the scoring rules agree—and where they answer different questions
The table shows eight of 54 cases. Full-precision expected outputs and all declared pairs are in both downloads.
| Case | H / D / A | Result | Top hit | Brier avg | Log loss | RPS |
|---|---|---|---|---|---|---|
uniform-h | 0.333333 / 0.333333 / 0.333333 | H | tie—excluded | 0.222222 | 1.098612 | 0.277778 |
uniform-d | 0.333333 / 0.333333 / 0.333333 | D | tie—excluded | 0.222222 | 1.098612 | 0.111111 |
strong-home-h | 0.8 / 0.15 / 0.05 | H | yes | 0.021667 | 0.223144 | 0.021250 |
ordinal-home-near-h | 0.6 / 0.35 / 0.05 | H | yes | 0.095000 | 0.510826 | 0.081250 |
ordinal-home-far-h | 0.6 / 0.05 / 0.35 | H | yes | 0.095000 | 0.510826 | 0.141250 |
certain-home-d | 1 / 0 / 0 | D | no | 0.666667 | +∞ | 0.500000 |
certain-home-a | 1 / 0 / 0 | A | no | 0.666667 | +∞ | 1.000000 |
tie-home-draw-d | 0.45 / 0.45 / 0.1 | D | tie—excluded | 0.171667 | 0.798508 | 0.106250 |
Locked conventions
The benchmark never silently changes scale or handles zero for you
Σ(pᵢ − yᵢ)²
The class-averaged companion is exactly the summed value divided by three.
−ln(p observed)
Zero probability is positive infinity, represented by a blank numeric cell plus an explicit status.
ε = 1e-15
The finite numerical variant is a separate column, never a replacement for the exact score.
H → D → A · divide by 2
The declared order is an analytical choice, not a claim that RPS is football's uniquely correct metric.
For definitions and sample-level interpretation, read the football prediction accuracy metrics guide. Recalculate individual cases in the Brier scale checker, log-loss calculator and RPS calculator.
Expected invariants
A useful implementation should pass relationships, not only isolated cells
- 01
Complete matrix
The 18 declared probability vectors crossed with H, D and A create exactly 54 rows.
- 02
Invariant 2
Home and away mirror cases retain equal Brier, exact log loss and normalized RPS after reversing both labels and outcomes.
- 03
Invariant 3
The ordinal near and far pairs can hold Brier and exact log loss equal while normalized H-D-A RPS changes.
- 04
Invariant 4
A one-hot forecast that misses has summed Brier 2 and class-averaged Brier 2/3; normalized RPS is 0.5 for an adjacent miss and 1 for an opposite miss.
- 05
Invariant 5
The uniform forecast has outcome-invariant Brier and exact log loss; normalized RPS is 5/18 for H or A and 1/9 for D under the declared order.
- 06
Invariant 6
A tied maximum leaves unique top pick and hit blank while every proper score remains defined.
Machine-readable contract
Column dictionary
benchmark_version- Semantic release of this immutable benchmark.
case_id- Stable test-case identifier within the release.
family- Synthetic scenario family; never a match or league label.
pair_id- Optional identifier for a declared comparison or symmetry pair.
p_home- Input home-win probability on the 0–1 scale.
p_draw- Input draw probability on the 0–1 scale.
p_away- Input away-win probability on the 0–1 scale.
outcome- Synthetic observed result: H, D or A.
unique_top_pick- Unique maximum-probability label, blank for a tie.
top_pick_hit- Boolean hit for a unique top pick, blank for a tie.
brier_sum- Sum of the three squared probability errors.
brier_class_average- Summed multiclass Brier divided by three.
rps_normalized_hda- Two-boundary H-D-A RPS divided by two.
log_loss_nats_exact- Negative natural log of the observed-outcome probability; blank at zero.
log_loss_exact_status- Finite or positive_infinity for the exact log score.
log_loss_nats_clipped_1e15- Separate log loss after clipping to [1e-15, 1−1e-15].
Primary sources and football context
The formulas are established; this fixture set is our editorial release
The cited papers define or analyse the scoring concepts and football forecasting context. They did not review, endorse or independently validate this benchmark dataset.
- Brier (1950), probability forecast verification.Original paper
- Gneiting & Raftery (2007), proper scoring rules.JASA paper
- Epstein (1969), Ranked Probability Score.Original paper
- Constantinou & Fenton (2012), scoring probabilistic football forecasts.Journal paper
- Dubitzky et al. (2018), open soccer database and prediction challenge.Machine Learning paper
- Berrar et al. (2024), 2023 Soccer Prediction Challenge framework.Machine Learning paper