Football Proof AIProbabilities with a public recordEvidence library

Validation research

Train on the past. Test on the future. Never reverse the arrow.

Direct answer: a football model should be evaluated through a sequence of historical publication moments. At each moment, the model may use only information that existed then, while the next untouched block supplies the score. Randomly shuffling matches answers a different—and usually easier—question.

Published by Football Proof AI · Published · Updated · walk-forward-validation/1.0.0 · Editorial technical note; not externally peer reviewed

The wrong counterfactual

Why random shuffle is not a match-day simulation

Football observations are ordered and connected. Team strength, injuries, managers, promoted clubs and league conditions move through time. A random split can train on a club's later form and then score an earlier fixture. Even when each feature row was computed correctly, the fitted model has learned from a future football environment that would not have existed at publication.

The non-negotiable ordering rulemax(train time) < min(test time)

Every scored probability must travel from past evidence to a later result.

Rolling-origin evaluation was developed to produce multiple genuine out-of-sample periods rather than one convenient split. The design choices—origin, window, refitting and test horizon—must be stated so the estimate can be reproduced. See Tashman's original review of out-of-sample forecast testing and the official scikit-learn time-series split documentation.

Rolling-origin folds

Replay several deployment windows, in order

A single holdout can be unusually easy or hard. Walk-forward validation repeats the train-then-test operation as the origin advances, producing probabilities from several seasons and match contexts without letting a later fold repair an earlier one.

Past → Future

Fold 1: train through cutoff A, score window A. Fold 2: advance the cutoff, refit under the same protocol, score window B. Continue until every planned window has been evaluated. The folds may use an expanding history or a pre-declared rolling window; that choice is part of the experiment, not a post-result adjustment.

Fold contract

Freeze what each historical forecaster was allowed to know

The timestamp boundary applies to the entire pipeline, not only the fixture table. Training labels, league tables, Elo updates, rolling form and model-selection decisions all need an explicit availability time.

  1. 01

    Training cutoff

    Include only matches whose outcomes were known before the fold cutoff. Store the cutoff with the model artifact and reject rows whose availability cannot be proved.

  2. 02

    Publication gap

    Generate the test probability at the same lead-time rule used in production. A feature available at kick-off is invalid for a forecast promised 24 hours earlier.

  3. 03

    Untouched score block

    Do not inspect test outcomes while selecting features, hyperparameters, calibration methods or thresholds. Score only after the fold recipe is fixed.

Point-in-time features

An as-of join is a boundary, not a convenience

For a prediction timestamp t, each feature must resolve to the latest eligible record whose availability time is at or before t. Joining by team and season alone is unsafe: a corrected table, rescheduled fixture or later rating can silently flow backwards.

Join key
Entity identity plus an ordered availability timestamp
Direction
Backward only: latest record available at or before prediction time
Tolerance
Declared per source so stale values are missing, not silently carried forever
Audit field
Persist the source row ID and observed-at time used for every feature snapshot

The operational behaviour maps directly to the official pandas merge_asof contract: ordered keys, an explicit direction and optional tolerance. The tool does not prove correctness by itself; the availability timestamp and grouping rules still need domain tests.

Calibration fitting

The calibrator needs its own honest evidence

Calibration is a fitted model. Training an isotonic or sigmoid mapping on the same outcomes used to report final performance leaks the answer into the probabilities. Instead, generate out-of-sample training probabilities inside the historical region, fit the calibrator there, freeze it, and then transform the later test fold exactly once.

Fit → Freeze → Score

Choose the calibration method before the outer fold is opened. Isotonic regression is flexible but can overfit when calibration samples are scarce; a simpler mapping or no post-hoc calibration may be more honest. Calibration should be judged with reliability tables and proper scores, not by whether it improves a selected headline.

Read the original ICML comparison of probability calibration methods

Frozen comparisons

Baselines cannot move after the model score is known

Baseline formulas, eligible matches and missing-data rules should be versioned before evaluation. Otherwise a favourable comparator can be selected after the fact. Each baseline must use the same folds and the same published population as the candidate model.

Sanity floor

Uniform 1X2

Assign one third to home, draw and away. It tests whether a candidate adds information beyond complete uncertainty under the same multiclass scoring definition.

Historical prior

Training-only frequency

Estimate outcome frequencies from the fold's training period only. Reusing the full-dataset class balance would import the future into the reference.

Operational comparator

Market consensus, if valid

Include bookmaker probabilities only when timestamp, margin removal and coverage rules are fixed for every fold. Missing odds cannot be dropped only when they make the model look weaker.

Decision record

One baseline manifest

Store the formula version, source cutoff, coverage and score next to the candidate model so future comparisons use the same contract.

Coverage and sample thresholds

A score without enough deployment-shaped evidence stays provisional

Thresholds do not make a model good; they stop a tiny or narrow test from being presented as broad proof. Define them before the run and report failed or excluded matches in the denominator.

Settled test matches
At least 500 across the complete walk-forward evaluation
Independent time windows
At least three folds, with no future-to-past training path
Calendar coverage
At least 365 days between the first and last scored window
Population coverage
Publish eligible, predicted, missing and settled counts by league and fold

These are Football Proof AI's current minimum evidence floors, not universal scientific constants. League, season and probability-bucket slices may still be too small for strong conclusions even when the overall total passes. Sparse slices should be labelled “insufficient evidence,” not merged or hidden until they look persuasive.

Deployment gate

Passing a backtest permits a trial; it does not prove the future

A candidate version may enter production only when its evidence artifact, feature schema, calibrator, training cutoff and fold metrics all pass the same immutable gate. Live predictions then begin a separate prospective record. Backtest rows and production rows must never be blended to preserve a launch narrative.

>48% + <0.22

The current admission gate requires top-pick hit rate above 48% and class-averaged multiclass Brier below 0.22, together with the coverage floors above. Both conditions are necessary; neither is a guarantee of profit, superiority to bookmakers or future accuracy. The Brier formulation traces to Glenn Brier's original probability-forecast verification paper.

Leakage red flags

Fast ways an impressive backtest can become invalid

  1. Random K-fold over multiple seasons.Later team and league states can influence predictions for earlier fixtures.
  2. Rolling statistics without a one-match lag.The target match result can enter its own form, goals or Elo feature.
  3. A database join with no availability timestamp.The stored row may be historically dated but corrected or published only later.
  4. Calibration fitted on the reported test fold.The final probabilities have already seen the outcomes used to grade them.
  5. Feature or threshold changes after viewing fold scores.The test set has become part of model selection and is no longer untouched.
  6. Coverage reported only for successful predictions.Feed failures and difficult matches disappear from the denominator.
  7. A baseline reconstructed with full-period information.The comparator itself knows the future, making the comparison uninterpretable.
  8. One aggregate score with no fold or league breakdown.Instability and concentrated failure can hide behind a global mean.

Primary references

Original research and official implementation contracts

These sources support the evaluation concepts above. They do not validate any unpublished Football Proof AI model result.

  1. Tashman (2000), International Journal of Forecasting.Out-of-sample tests of forecasting accuracy: an analysis and review.
  2. Niculescu-Mizil and Caruana (2005), ICML.Predicting Good Probabilities with Supervised Learning.
  3. Brier (1950), Monthly Weather Review.Verification of Forecasts Expressed in Terms of Probability.
  4. scikit-learn documentation.TimeSeriesSplit: time-ordered train/test indices.
  5. pandas documentation.merge_asof: ordered nearest-key joins with direction and tolerance.