The College Football Playoff Committee Is Answering The Wrong Question

Before losing to Alabama in the College Football Playoff semifinals, Notre Dame was widely considered one of the four best teams in the country. However, you’d have been hard-pressed to find anyone who thought Notre Dame could have been the best, given its blowout loss to Clemson in the ACC championship. So if Notre Dame clearly wasn’t the best, what was the point of including the Irish in the playoff at all?

Under its own guidelines, the playoff selection committee picks what it deems to be the four best teams, which seems pretty straightforward. Outside of expanding the playoff, there’s another way it could make its selection — a way that is arguably more fair and that could increase the diversity among selected teams.¹ Instead of picking the teams it deems most likely to be among the top four, which might include a team that has lost to a slightly stronger team and is therefore unlikely to end up as the best, the committee could instead pick the four teams that still have the best chance to be the best. While this sounds like a trivial change, it might have affected this year’s playoffs — and could very well bring new programs into the playoff.

To see how this is possible, we need a way to assess the probability that a team is indeed the nation’s best. Here I introduce one such way — a way that makes several simplifying assumptions, but it’s a start.

First, let’s make use of ESPN’s Football Power Index, or FPI, which takes into account both strength of schedule and margin of victory, and which has been shown to be predictive of game scores. To predict how two teams will fare against each other at a neutral site, the team with the higher FPI is expected to win, and the most probable margin of victory is the difference between their FPIs.

Now, for any given team, let’s assume we have no prior information about its FPI — nada, zip. Its probability distribution is uniform, equally likely to be -100, 0 or 100. Next, let’s look at how this team performed one game at a time. For example, if this team beats a team with an FPI of 10 by a margin of 20 points, it becomes more likely that this team’s true FPI was close to 30 and less likely that it was much lower than 30 or much greater than 30. If we play this out over the course of a season, we get a probability distribution for this team’s FPI.² Once we have the FPI probability distributions for the different teams, we can calculate the probability of each being the best team.

Before we see how this plays out, it’s also worth noting that teams playing fewer games (like Ohio State, which went 6-0 on the season and played about half as many games as most other title contenders) had a slight advantage. Playing fewer games meant they had a wider FPI probability distribution, which increased their chances of being the best team overall.

With these FPI probability distributions calculated, it’s time for the big question: Prior to the playoffs, what was the probability that each school was the best? And would this have changed the playoff picture?

To figure this out, I ran 1 million simulations with the top 10 teams from the final College Football Playoff rankings (before the playoff semifinals) and recorded the share of simulations in which each team ranked first overall.

According to this model, there was about an 85 percent chance that Alabama was the best team in the country before the playoff, which sounds about right. The next tier down included Clemson and Ohio State, each of which claimed between a 5 and 10 percent chance of being the nation’s best team.

The third tier of schools was where this model significantly diverged from the CFP rankings. Sure enough, Notre Dame was not fourth most likely to be the best team — that distinction fell to unbeaten Cincinnati (No. 8 in the rankings), followed closely by Big 12 champion Oklahoma (No. 6). Meanwhile, Notre Dame failed to distinguish itself from Texas A&M and Florida, which similarly had solid seasons but were not in strong contention for being considered the best team. Ninth-ranked Georgia and 10th-ranked Iowa State were both more likely to be the best than Notre Dame.

So according to these results, how well did the playoff committee do in picking its four teams? By selecting Alabama, Clemson, Ohio State and Notre Dame, there was a 97.8 percent chance that one of them was the best team in the country. That may sound pretty good, but the committee could have done better. If it had chosen Cincinnati over Notre Dame, its chances of having selected the best team would have been 98.6 percent.

Let’s return to the present. Now that the two semifinal matches (and all of the other nonplayoff bowl games) have been played, how are these 10 teams faring? With the championship game just days away, here are the updated probabilities that each team is the best in the nation:

In all likelihood, Alabama is still the best team in the country (duh), while Ohio State’s chances of being the best now stand at close to 12 percent. Clemson and Oklahoma are next, at a bit more than 3 percent, followed by a more distant Cincinnati, and then everyone else.

While the very best teams will always rise to the top — whether it’s a committee picking them or an algorithm — every year brings some new controversy over who deserves the final spot or two in the playoffs. And every year, it seems that the committee plays it safe, picking a legacy team that had a strong prior season instead of an undefeated up-and-comer who may be deserving of a shot at the playoffs.

Cincinnati may have narrowly lost its bowl game this year. But had the Bearcats won, we could have had another UCF situation on our hands, as in 2017 when the NCAA was forced to recognize two national champions. With the slightest shift in its decision-making process — picking the four teams most likely to be the best, rather than the four teams it perceives to be the best — the committee could make the playoffs more diverse while also increasing the chances that the title winner is the true premier team in college football. And isn’t that the point?