The Audibles are a series of short-form research articles about thorny issues in college football analytics
Accounting for Error in Preseason Predictions
If you are a college football fan, you have probably dropped by the ESPN-FPI (Football Power Index) page to peruse power rankings and championship probabilities. It presents a lot of interesting information: power rankings and ratings, division and championship probabilities, and College Football Playoff (CFP) chances. It’s a useful stop for comparing the relative strength (as measured by the power ratings) of all the FBS teams, at least based on the FPI model. And the FPI model is perfectly reasonable for rating team strength, having performed generally well since its inception.
However, there is one aspect that is emblematic of a common problem in CFB prediction modeling, especially given the rapid proliferation of people using software and statistical methods to make data-based forecasts:
ESPN-FPI does not account for the error in their preseason power ratings when computing season-based probabilities.
Likewise, most other prognosticators widely sharing their predictions fail to include the error in their own power rating predictions when projecting the chances that a given team wins its division/conference, or when computing the team’s projected win distribution.
Naturally, you might wonder…does it really matter?
And the answer is a resounding YES.
For example, if you visited the FPI page on 8/1/23 (the date this article was posted) and looked closely at the first row of probabilities – corresponding to ESPN’s preseason #1 Ohio State – you’d see something stunning:
The Buckeyes have a 71% chance of winning the Big Ten!
That’s nearly 4X the combined chances of top East competitors Michigan (who won the Big Ten and made the playoff in 2022) and Penn State. All three will begin the season ranked in the top ten, and all three are widely believed to be legitimate CFP contenders. If that consensus view is true, then Michigan and Penn State should be within striking distance of Ohio State in the conference championship odds.
When in doubt, consider what the Vegas sportsbooks have to say about the matter, since they are not in the business of making systematic, exploitable errors that would cost them a great deal of money.
Here are recently posted odds, along with the implied probability, for each team winning the Big Ten in 2023:
- Ohio State +175 / 36%
- Michigan +185 / 35%
- Penn State +575 / 15%
These odds and probabilities demonstrate much more balance and common sense. Ohio State and Michigan are about equally likely to win the Big Ten, according to these odds, with Penn State behind but squarely in the picture.
At mcillecesports.com, our simulations more closely resemble the sportsbook view in terms of balance, but favor the Buckeyes over the Wolverines – similar to FPI in terms of pure power rating:
- Ohio State +150 / 40%
- Michigan +330 / 23%
- Penn State +610 / 14%
The key distinction between how we run simulations here and how ESPN-FPI runs theirs is that we incorporate random prediction error into every simulation, based on observed error distributions in the past. Specifically, the error distribution is Normal with a mean of zero and a known standard deviation, or sigma, consistent to a certain precision level year to year.
Thus, to include error in a season simulation, each team’s power rating is NOT treated as a fixed, true number; instead, it is randomly drawn from a Normal distribution centered at the predicted power rating (because the error is zero, on average, meaning the prediction is unbiased) with sigma drawn from the error distribution above.
That results in a simulation that truly accounts for errors in preseason predictions once run enough times to fully capture the variability in the system. Here, that means 100,000 full simulations with error included; ESPN runs 10,000 without error included.
Essentially, systems that exclude error from their preseason predictions treat them as if they’re postseason ratings rather than preseason ratings. Unfortunately, when doing the former, any probabilities computed from that system are invalid because they are biased, indicating more certainty than actually exists in college football projections.
There are two simple ways to identify this preseason fallacy in CFB probabilities:
- Compare futures (division, conference, CFP, etc) to Vegas sportsbooks. If, across the board, a prediction system’s probabilities indicate more certainty (higher probabilities) of the top teams winning their conferences or advancing to the CFP, that is a red flag.
- If available, look at the projected win distribution; i.e., the probabilities associated with going 0-12, 1-11, 2-10, …, 11-1, 12-0 in the regular season. If the distribution suggests that a team has almost zero probability of falling outside a narrow win range (such as +/- 2 from the mean prediction), that is too narrow. True win distributions are much more dispersed than that. Returning to the Buckeye example, Ohio State’s most likely record (according to our projections) is 11-1, but there is about a 15% chance they lose 4+ games.
If utilizing preseason CFB information from “zero-error” systems like ESPN-FPI, keep in mind that while the power ratings are useful for comparisons and point spreads, probabilities associated with them should be avoided – especially in any wagering context.