First decision
Use the comparison your match history actually supports
Statistical significance is not a property of two percentages alone. The calculation changes when predictions share the same matches because their wins and misses are correlated.
- 01
One model vs reference
Use one accepted prediction record and a reference rate declared before examining that record.
- 02
Independent samples
Use when A and B were evaluated on separate fixtures or genuinely independent match samples.
- 03
Paired same-match test
Use four correctness cells when both models forecast every one of the same settled fixtures.
Two paired models can have identical marginal hit rates but very different uncertainty depending on how often they disagree. Keep the row-level match pairing; totals alone cannot reconstruct it. A feasible discordant rate q must satisfy |pA − pB| ≤ q ≤ min(pA + pB, 2 − pA − pB); the calculator rejects combinations that would imply a negative four-cell count.
Reference choice
A football baseline is not automatically 33.3%
- Uniform
- A one-third reference describes random equal 1X2 picks, not the real outcome mix.
- Empirical
- A historical most-common-outcome strategy can be much stronger, but must be estimated only from earlier matches.
- Operational
- A bookmaker-implied top pick or frozen prior model can be the relevant hurdle if specified before evaluation.
The calculator therefore keeps the reference editable. Document its source, time cutoff, league scope and decision rule next to every published result.
Formula contract
What each result estimates
Confidence intervals, planning equations and most tests below use stated approximations; the small-discordance McNemar p-value is exact conditional. Every result is for transparent reporting, not a substitute for a pre-declared protocol.
- Single rate
- Wilson score interval at 95%; one-sample normal score test against the chosen reference
- Independent difference
- Newcombe-Wilson score interval for A minus B; pooled two-sample z statistic under equal null rates
- Paired difference
- Normal interval from per-match correctness differences only when at least 25 pairs are discordant; otherwise the interval is withheld and the p-value uses the exact conditional McNemar test
- Required sample
- Two-sided normal planning approximation at the selected alpha, power and meaningful effect
- Minimum detectable effect
- Smallest effect whose approximate required sample is no larger than the current sample
- Paired Brier plan
- Paired-mean normal approximation using the supplied SD of per-match A-minus-B Brier differences
Paired probability quality
Brier sample size needs the variation of paired differences
Evaluate both models on each same match, calculate one Brier score for each, subtract B from A and estimate the standard deviation of those differences from a separate pilot or earlier time window. Use the same summed or class-averaged convention in the pilot and the planned study.
Marginal Brier means or separate standard deviations are not enough because they omit the within-match covariance. A pilot SD chosen after inspecting the final evaluation can make the sample plan look more precise than it was. The calculator also enforces the bounded-difference feasibility limit SD ≤ √(M² − δ²), where M is the convention's largest absolute per-match difference and δ is the planned mean-difference magnitude. The inverse bound, |δ| ≤ √(M² − SD²), also prevents the current-sample MDE from being reported when no bounded per-match difference distribution could attain it at the supplied SD.
Limits before claims
Approximate power is not a performance guarantee
Required sample size is a design aid under stated assumptions. It does not guarantee significance, future accuracy or betting value in the realized sample.
- Small discordant counts need an exact paired test.The normal McNemar approximation can be inaccurate when few matches differ.
- Match observations may be dependent.Repeated teams, leagues, seasons and time periods can create clustering that these formulas do not model.
- Tuning and selection consume evidence.Choosing the model, threshold, baseline or metric on the same matches invalidates a clean confirmatory interpretation.
- Multiple comparisons inflate false positives.Testing many leagues, horizons or model versions needs a declared multiplicity policy.
- Power assumptions can drift.Reference rates, paired discordance and Brier-difference variance can change after deployment.
- Publication timing still matters.A statistically strong result is not a forecast record unless its probabilities were publicly fixed before kick-off.
Primary references
Intervals, paired tests and proper scoring rules
This is an editorial technical note, not externally peer reviewed. The cited papers define the statistical building blocks; they do not certify any entered evidence or model claim.
- Wilson (1927), score interval for one proportion.Original paper and DOI
- Newcombe (1998), interval for the difference between independent proportions.Original paper and DOI
- McNemar (1947), paired binary response comparison.Original paper and DOI
- Brier (1950), verification of probability forecasts.Original paper and DOI
- Gneiting & Raftery (2007), proper scoring rules.Journal paper and DOI
- Cohen (1992), statistical power planning.Method paper and DOI
Cite this page as: Football Proof AI (2026), “Football Prediction Sample Size & Significance Calculator,” version football-sample-size/1.0.0. Access the canonical page at https://footballproofai.com/tools/football-prediction-sample-size-calculator.