Reproducible evaluation guide
How to measure football prediction accuracy
Direct answer: no single number establishes that a football model is good. Report top-pick hit rate for decisions, Brier score and log loss for the full 1X2 distribution, calibration for probability honesty, and sample uncertainty plus pre-declared baselines for context.
Published by Football Proof AI · Published · Updated · football-accuracy-metrics/1.0.0 · Editorial technical note; not externally peer reviewedOne forecast, fully specified
Recalculate every example from 54 / 26 / 20
Assume a model published 54% home, 26% draw and 20% away before kick-off. The probabilities sum to one. Home is the top pick, but the other outcomes still carry 46% of the probability mass.
0.54 + 0.26 + 0.20 = 1.00. The top pick is home; certainty is not implied.
A single result can be scored, but one match cannot validate a model. Use the interactive probability and Brier calculator to reproduce the arithmetic, and the glossary to keep definitions fixed.
Metric 1 · Classification
Top-pick hit rate answers one narrow question
For each forecast, select the largest of home, draw and away using a tie rule fixed in advance. Hit rate is the proportion of those selections that match the settled result.
In the worked example, home is the top pick. A home result counts as one hit; a draw or away result counts as one miss. The metric discards whether home was 54% or 94%, so it cannot evaluate the honesty of the full probability distribution.
Metric 2 · Quadratic probability error
Class-averaged multiclass Brier score uses all three probabilities
Encode the observed result as a one-hot vector: the outcome that occurred is 1 and the other two are 0. Square each probability error, add the three errors, and divide by three. Lower is better.
If home wins: (0.46² + 0.26² + 0.20²) ÷ 3 = 0.1064
| Actual result | One-hot target | Class-averaged Brier | Log loss |
|---|---|---|---|
| Home | 1 / 0 / 0 | 0.1064 | −ln(0.54) = 0.6162 |
| Draw | 0 / 1 / 0 | 0.2931 | −ln(0.26) = 1.3471 |
| Away | 0 / 0 / 1 | 0.3331 | −ln(0.20) = 1.6094 |
On this class-averaged scale, a perfect forecast scores 0, an equal 1/3–1/3–1/3 forecast scores 0.2222 for any result, and the maximum error is 0.6667. Some publications omit the division by three, producing a score three times larger. State the convention before comparing numbers. The score originated in Brier's 1950 probability-forecast verification paper; modern scoring-rule theory explains why quadratic and logarithmic scores reward honest probability reports.
Metric 3 · Reliability
Calibration asks whether stated probabilities match frequencies
Across a sufficiently large, pre-defined group of comparable forecasts labelled near 54%, the selected outcome should occur near 54% of the time. Calibration is a property of a collection, not of one match.
Plot predicted against observed
Group forecasts into disclosed bins, plot each bin's mean stated probability against its observed frequency, and always display the count. Inspect top-pick and classwise views because one aggregate can hide draw-specific errors.
ECE = Σ (nᵦ ÷ N) × |observedᵦ − predictedᵦ|
If 100 forecasts in one disclosed bin average 54% confidence and 57 are correct, that bin contributes |0.57 − 0.54| = 0.03. With several bins, weight each absolute gap by its share of the sample.
ECE depends on the bins
Changing boundaries or the number of bins can change ECE. Publish the binning rule, counts and diagram; do not use one small ECE value as standalone proof that one model is better.
Metric 4 · Logarithmic probability error
Log loss strongly penalises confidence in the wrong outcome
For each match, take the negative natural logarithm of the probability assigned to the result that occurred, then average across the complete evaluation sample. Lower is better.
The worked example scores 0.6162 if home wins, 1.3471 if the match is drawn and 1.6094 if away wins. Unlike the Brier scale, log loss is unbounded: assigning a probability close to zero to the event that occurs incurs a very large penalty. Implementations commonly clip machine-zero probabilities for numerical stability and should disclose that rule.
Sample uncertainty
A percentage without its denominator is incomplete
Suppose a model records 54 top-pick hits in 100 settled matches. Its observed hit rate is 54%, but sampling variation means 54% should not be presented as exact underlying skill.
Point estimate 54.0% · n = 100 · two-sided z = 1.95996
The Wilson interval is generally more informative than the simple normal approximation for a binomial proportion. It quantifies sampling uncertainty under its assumptions; it is not the probability that the model's true hit rate lies inside this one realised interval, and it does not prove an advantage over a baseline. League mix, model changes and time dependence can create additional uncertainty that this interval does not capture.
Comparison protocol
A score becomes meaningful only against time-safe baselines
Choose baselines before viewing the test results. Compute every baseline on the identical fixtures, settlement rules and metric convention used for the model.
- 01
Always-most-common
For hit rate, always select the outcome that was most common in training data. “Random equals 33%” is not an honest baseline when home, draw and away are imbalanced.
- 02
Uniform probability
Use 33.33% for each outcome as a transparent probability reference. Its class-averaged three-class Brier score is always 0.2222, but it ignores real outcome frequencies.
- 03
Historical frequency
Estimate league or home-draw-away base rates only from data available before each test match. This tests whether model complexity adds value beyond prevalence.
- 04
Simple model
Compare with a pre-declared Elo or multinomial logistic model evaluated through the same walk-forward folds, not a weaker benchmark given different information.
- 05
Market reference
If timestamped odds and a disclosed margin-removal method are legally available, compare against the market snapshot that genuinely existed at the model's publication time.
- 06
Previous model
A replacement should beat or materially improve the operational trade-offs of the version it replaces on a locked out-of-time window, with uncertainty reported.
Minimum evidence bundle
What an accuracy report should disclose
These fields make two reports comparable and make silent exclusions harder. Missing fields should reduce confidence in the claim, not invite optimistic assumptions.
- Population
- Leagues, seasons, date range, inclusion rules, voids and postponed matches
- Sample
- Eligible forecasts, settled forecasts, hits and the count inside each calibration bin
- Model identity
- Version, training cutoff, feature availability and probability-calibration method
- Protocol
- Walk-forward folds, no future-data rule, frozen metrics and baseline definitions
- Metrics
- Hit rate, class-averaged Brier, log loss, calibration diagram/ECE and intervals
- Breakdowns
- League, season, outcome class and confidence band, without hiding the aggregate
- Record integrity
- Pre-match timestamps, permanent rows, corrections and every eligible miss
- Limitations
- Small samples, feed gaps, distribution shifts and any benchmark unavailable for comparison
See how Football Proof AI separates the public accuracy ledger, model cards and walk-forward methodology. For a broader integrity review, use the AI football prediction audit guide.
Primary references
Research and official technical documentation
These sources support the metric definitions and statistical context. They do not validate any particular football model or imply that a published score will persist in future matches.
- Brier (1950), probability forecast verification.Monthly Weather Review paper and DOI
- Gneiting & Raftery (2007), strictly proper scoring rules.Journal of the American Statistical Association DOI
- Naeini, Cooper & Hauskrecht (2015), calibrated probabilities and ECE.Proceedings of AAAI paper and DOI
- Wilson (1927), interval estimation for proportions.Journal of the American Statistical Association DOI
- scikit-learn, official log-loss API definition.Official technical documentation