A 12-Team College Football Playoff Would Be Big Enough … 90 Percent Of The Time

Almost since the start of the College Football Playoff in 2014, people (including us, obviously) have complained that more teams should be included. We may get our wish before too long. Earlier this year, the playoff management committee presented a plan to expand the playoff — the first ranking for which will be released today — to 12 teams, and that committee is meeting this week to discuss it. It would be a four-round playoff, with four teams earning a bye into the second round.

With this proposed expansion, it’s only natural to wonder what the “right” number of teams might be. During the transition from the Bowl Championship Series to the playoff seven years ago, there was consensus that the in-place two-team championship was too small. Meanwhile, a 128-team playoff, which would include all but two Football Bowl Subdivision programs, is assuredly too large. The ideal number is somewhere in between. Is it four? Is it 12?

Earlier this year, I wrote that I believed the playoff committee was answering the wrong question. Instead of identifying the four supposedly best teams, I argued, the committee should try to pick the four teams with the greatest chance of being the single best team. There is a subtle but significant difference between these two processes: Instead of choosing a team that it knew was not the best (but was likely, say, fourth best), the committee should instead prioritize teams whose rankings were less certain — e.g., a team that is probably 10th-best but that had some small, non-zero chance of being the best.

In the 2020 college football season, this thinking wound up favoring dark-horse teams like Oklahoma (which cruised through its bowl game), as well as Georgia and undefeated Cincinnati (which played a very competitive bowl game against each other), all at the expense of what was fourth-ranked Notre Dame (which got crushed by top-ranked Alabama in the playoff). 

So my suggestion was to put aside the notion of ranking as we know it, and to instead embrace the uncertainty that is college football by quantifying each team according to how likely it is to be the best in the country. And that’s how we can explore and compare different-sized playoffs. With more teams in a playoff, the probability that the best team is included goes up. Of course, there are diminishing returns, since each additional team we include has a lower and lower probability of being the best. Therefore, we might opt for a cutoff, such as 90 percent. That is, how many teams should be included in a playoff so that there’s a 90 percent chance that the best team has been included?

Last time around, I used ESPN’s Football Power Index in my analysis; this time, I used FiveThirtyEight’s Elo ratings, which operate along a similar scale. So if a team has an Elo of 30, that value represents our current best guess about the quality of the team. We could better represent this as a probability distribution, normal in shape, and with a mean at 30. To find the standard deviation, or spread, of this distribution, my colleague Neil Paine looked historically at teams’ Elo ratings midway through each season of the playoff era and compared these to how those teams played from their sixth game onward. Neil found this difference to be a normal distribution, centered at zero, and with a standard deviation of 9.364. And that is the very uncertainty in Elo I used for this analysis.

For all the completed seasons in the playoff era (2014 through 2020), I started with the Elo ratings of every team at the moment the playoff committee issued its final, pre-bowl rankings of the season. I then ran 100,000 simulations for each season. Within each simulation, I looked at every team and randomly assigned it an Elo rating from within that team’s probability distribution. For each team, simulated ratings close to FiveThirtyEight’s computed rating were more likely, while ratings much higher or lower were less likely. Then, for each simulation, the best team was the one with the highest simulated Elo rating. Finally, I tallied the number of simulations in which each team came out the best. This represented the probability that each team was indeed the best.

For example, if we were to look at the 2021 season as of this week, Georgia has the highest Elo rating and would be the best team in a plurality of simulations. But it would not be the best in every simulation, so we couldn’t say with 100 percent certainty that Georgia is the best team. If another team, like Cincinnati or Alabama, happened to get lucky with a higher-than-expected rating (or higher than Georgia’s, at least) in any of those simulations, then it would be the best team.

Different seasons from 2014 through 2020 produced very different results. In 2016, Alabama’s undefeated team capped off its pre-playoff season by crushing Florida in the SEC championship game by a score of 54-16. While it would ultimately lose to Clemson in the national championship, that Crimson Tide squad had a 49 percent chance of being the best team — the highest of any team (according to this analysis) going into the playoff in this era. Meanwhile, there was far less certainty in 2017, when top-ranked Clemson had just a 20 percent chance of being the best team. (It would get bounced by Alabama in the playoff semifinals.) 

The season in which the highest total probability of one of the four playoff teams being the best team came in 2019, when undefeated LSU, undefeated Ohio State, undefeated Clemson and one-loss Oklahoma combined for a whopping 78 percent. But this demonstrates the limits of a four-team playoff. Even in a year with three dominant teams, the top four teams could not sufficiently separate themselves to capture 90 percent of the overall probability of including the best team.

So then, how many teams does it take in the field to be 90 percent sure that we have the best team? The chart below shows the combined probability of being the best team among the top schools for each of the years 2014 through 2020 according to the number of teams — four through 25 — hypothetically let into the playoff. Despite the differences among the seasons, they exhibit the same overall trend: A few schools account for most of the probability, but there is a long tail with diminishing returns.

On average, to hit that 90 percent chance of including the best team, a playoff would have needed at least 11 schools. Unfortunately, 11 isn’t a very convenient number for a playoff tournament, so 12 sounds more reasonable. Meanwhile, the top four teams only had a 68 percent chance of including the best team, on average.

What if we wanted to be extra sure to include the best team, increasing our cutoff from 90 percent to 95 percent? On average, you would have needed 16 schools. Now that’s a tournament-ready number!

However, if you read the fine print of the committee’s proposal to expand the playoff, they did not suggest picking the top 12 teams by FiveThirtyEight’s Elo. Instead, the 12 teams would include the six highest-ranked conference champions, along with the six highest-ranked remaining teams. How would this selection of 12 teams have performed?

We can explore this by taking a closer look at the 2019 season:

In 2019, there was a 94 percent chance that the best team was among the top 12 teams by Elo. Using the committee’s proposed criteria, that probability would have only dropped a little, to 91 percent. That far surpasses the aforementioned 78 percent accounted for by the top four teams, and that’s because the probabilities are more comparable among the second tier of teams. Whether you include Memphis or Auburn makes relatively little difference, since only a few tenths of a percentage point are at stake.

Rather than picking multiples of two, four or eight that it finds appealing, the playoff committee has a chance to set a format with a high likelihood that the playoff includes the best team.³⁵ In a typical year, the proposed 12-team format would get us to a 90 percent likelihood that the best team is included. 

Check out our latest college football predictions.

Watch: https://abcnews.go.com/fivethirtyeight/video/makes-patrick-mahomes-great-fivethirtyeight-75237605