Championships Aren’t Won On Paper. But What If They Were?


You probably missed this as baseball’s postseason was coming to an end last week — but congratulations are in order to the Houston Astros. Why? Because Houston finished the 2019 MLB season with the No. 1 Elo rating in Major League Baseball, of course.

The Astros were No. 1 in 2019 … on paper

Top 10 MLB teams in 2019 according to FiveThirtyEight’s Elo rating*

Reg. Season
Team Wins Losses Playoffs Elo Rating
1 Astros 107 55 Lost WS 1595
2 Dodgers 106 56 Lost LDS 1590
3 Nationals 93 69 Won WS 1589
4 Yankees 103 59 Lost LCS 1585
5 Athletics 97 65 Lost WC 1570
6 Rays 96 66 Lost LDS 1554
7 Cardinals 91 71 Lost LCS 1543
8 Braves 97 65 Lost LDS 1541
9 Indians 93 69 None 1538
10 Twins 101 61 Lost LDS 1537

*Using the version of Elo found in our “Complete History of MLB” interactive, which does not adjust for the quality of pitchers.

I mean, sure, the Washington Nationals just beat the Astros to win the World Series in seven games. But still, the Astros were your official Elo champs for the 2019 season. (Somehow I doubt the Astros will throw a parade or put up a banner for the honor.)

Because Elo takes a long view of the entire season, being the best in it is a pretty good proxy for being the best team in the league “on paper.” And it’s actually not uncommon for Elo’s Paper Champion and the team that wins the World Series not to be one and the same. Including Houston this year, it’s happened 28 times — or in a whopping 52 percent of seasons — since the late 1960s, when MLB expanded to a division-based playoff format. Simply put, baseball is a sport in which the best team doesn’t always win. (Or even if it does, maybe we don’t always know who the best team is anyway.)

Take a tour through MLB’s Hall of (Paper) Champions

Actual World Series champions and end-of-season MLB Elo champions* (for years where they were not the same team), 1966-2019

Paper Champ Actual Champ Paper Champ Actual Champ
Year Team Elo Team Elo Year Team Elo Team Elo
2019 HOU 1595 WAS 1589 1995 CLE 1596 ATL 1580
2017 CLE 1596 HOU 1572 1993 ATL 1588 TOR 1565
2015 TOR 1562 KC 1561 1992 MIL 1558 TOR 1555
2014 BAL 1559 SF 1542 1990 OAK 1567 CIN 1544
2012 TB 1566 SF 1561 1988 NYM 1569 LAD 1555
2011 NYY 1575 STL 1555 1987 TOR 1562 MIN 1521
2010 PHI 1570 SF 1563 1985 NYY 1571 KC 1544
2008 BOS 1567 PHI 1564 1982 MIL 1555 STL 1552
2006 NYY 1551 STL 1531 1980 BAL 1577 PHI 1545
2003 NYY 1567 FLA 1547 1974 LAD 1569 OAK 1559
2001 OAK 1596 ARI 1567 1973 BAL 1569 OAK 1556
2000 SF 1559 NYY 1542 1972 PIT 1560 OAK 1557
1997 ATL 1572 FLA 1538 1971 BAL 1591 PIT 1568
1996 CLE 1568 NYY 1547 1969 BAL 1576 NYM 1567

*Using the version of Elo found in our “Complete History of MLB” interactive, which does not adjust for the quality of pitchers.

Ratings include results from all regular-season and postseason games.

Other sports have their own Paper Champs. (Although none happen anywhere near as frequently as in MLB.) I went back to the start of the Super Bowl era in 19661 and looked at the other sports for which we keep Elo — the NFL, NBA, college football, and men’s and women’s college basketball. Using the versions of our Elo that contain no adjustments for trades or players being in and out of the lineup,2 I found each case where the champion at the end of the season3 was not the team that finished atop the Elo leaderboard. Across all of these sports, these Paper Champs come up more frequently than you might think:

Since 1966, all but seven seasons4 (2013, 2009, 2005, 1998, 1989, 1979 and 1967) contained at least one Paper Champion across these five sports. Some years featured a lot more: In 2011, for instance, there were four Paper Champs — the Yankees in MLB, the Patriots in the NFL, and Ohio State (men’s) and UConn (women’s) in college basketball. (The only champs that actually led in Elo were the Dallas Mavericks in the NBA and Alabama in college football.)

In general, there is about a 50-50 chance that a given baseball season would produce a Paper Champ and somewhere between a 20 and 30 percent probability each of the other sports will as well.

How each sport’s Paper Championship rate compares

Frequency of Elo “Paper Champions” (and rate of the real champion being decided head-to-head) by sport, 1966-2019

Sport
MLB NFL NBA CFB MBB WBB*
No. of Paper Champs 28 11 15 14 14 4
Share of seasons 51.9% 20.4% 27.8% 25.9% 25.9% 22.2%
% of Paper Champs lost H2H** 46.4% 90.9% 60.0% 14.3% 78.6% 25.0%

* Women’s NCAA basketball data only goes back to 2001-02.

** This is the share of seasons with a Paper Champ that saw the actual champ beat them head-to-head in the postseason (or, in college football, the entire season).

Source: Sports-Reference.com

What does all of this mean? Well, it could be that Elo is broken. Even though it is calibrated to be the best predictor for a team’s next game — given its recent form, long-term expectations, wins and losses, scoring margin, opponent quality and game locations — maybe there are certain aspects of each sport’s postseason that aren’t captured by the algorithm. (This is the “Billy Beane’s Shit Doesn’t Work In The Playoffs” theory.)

A fundamental challenge of forecasting is the balancing act between considering a large amount of information — some of which may be of less relevance than others — and a more specific one that is more relevant, but also more prone to factors such as random variance in a small sample. Our Elo models attempt to straddle this divide, but it’s impossible to find the perfect mix of information that works in every single case.

Then again, maybe the real issue is that playoff systems are too small of a sample to determine the best team. Perhaps the best we can do is be content believing the champion was simply one of the top teams in a given season, nothing more.

Still another way of reconciling Paper Champs to postseason reality, though, is to consider that most of these actual champions vanquished their on-paper rivals head-to-head along the way. When there was a Paper Champ in baseball, for instance, 46 percent of the time that team lost directly to the eventual champion in a postseason game or series. (See: Nationals over Astros.) In the NBA, that number was 60 percent; in men’s college basketball, 79 percent; and in the NFL, a whopping 91 percent. Although Elo still wasn’t convinced that the matter was settled afterward, the Paper Champ at least had a chance to make its case on the field or court.

And in most of the cases where things weren’t settled head-to-head, you only have to zoom out a little to find a path of head-to-head superiority between the actual champ and the paper one. Like in the 2017 NCAA Women’s Tournament, when UConn finished as Paper Champ … but lost in the national semis to Mississippi State, which then lost to South Carolina in the title game. Almost every disagreement between Elo and the official championship can be settled either directly head-to-head or in this manner — with the exceptions of a few pre-wild-card MLB seasons (in which the Paper Champ didn’t even make the postseason at all) and a number of older college football campaigns that underscored just how broken the sport’s pre-playoff system truly was.

In 2007, famously one of the weirdest college football seasons ever, USC was Elo’s choice, while LSU prevailed in the BCS. To find a head-to-head path that put LSU over USC, even if you open up the possibilities to include the regular season,5 you needed to follow a trail of four games: LSU beat Ohio State, which beat Washington, which beat Stanford, which beat … USC.

But at least the BCS existed by then. Before it came along, the 1970s and ’80s often required even more ludicrous daisy-chaining of head-to-head results to reconcile the championship. In 1976, the path from actual champ Pitt to Paper Champ USC required a string of five games. And in 1983, the chain went like this: Actual champ Miami beat Notre Dame, which beat Boston College, which beat Clemson, which tied Georgia, which beat Texas, which beat Auburn, Elo’s Paper Champion. No wonder college football fans clamored for a proper playoff (even if the one they have now could also probably stand to be expanded).

But even with the perfect playoff system, you can never really avoid Paper Champs. Random variance and matchups — plus a million other factors — will always cause teams to play better or worse than they look on paper. And would we really want it any other way? If we look at who would have benefited most over the past half-century if Elo perfectly aligned with actual championships, the rich would mostly have gotten richer:

Who has beaten their on-paper odds most (and least) often?

MLB, NFL, NBA, college basketball (women’s* and men’s) and college football teams with the biggest positive — and negative — differentials between their actual championships, 1966-2019

Biggest gainers No. of championships
Team Sport Actual Elo Diff
Notre Dame Fighting Irish CFB 4 0 +4
Los Angeles Lakers NBA 11 7 +4
New York Giants NFL 4 1 +3
St. Louis Cardinals MLB 4 1 +3
Miami/Florida Marlins MLB 2 0 +2
Houston Rockets NBA 2 0 +2
Kansas City Royals MLB 2 0 +2
LSU Tigers CFB 2 0 +2
Clemson Tigers CFB 3 1 +2
Miami Heat NBA 3 1 +2
San Francisco Giants MLB 3 1 +2
Connecticut Huskies MBB 4 2 +2
Ohio State Buckeyes CFB 4 2 +2
Boston Celtics NBA 9 7 +2
Biggest losers No. of championships
Team Sport Actual Elo Diff
Baltimore Orioles MLB 3 8 -5
Cleveland Indians MLB 0 3 -3
Oklahoma Sooners CFB 4 7 -3
San Antonio Spurs NBA 5 8 -3
Connecticut Huskies WBB 9 12 -3
Milwaukee Brewers MLB 0 2 -2
Philadelphia 76ers NBA 2 4 -2
Kentucky Wildcats MBB 4 6 -2
North Carolina Tar Heels MBB 5 7 -2
New England Patriots NFL 6 8 -2
New York Yankees MLB 7 9 -2
Alabama Crimson Tide CFB 9 11 -2

*Women’s NCAA basketball data only goes back to 2001-02.

Source: Sports-Reference.com

Although I feel bad for the Orioles and Indians, who would have won multiple extra championships if Elo had been perfectly predictive, other teams that have won plenty of real titles — Alabama football, UConn women’s basketball, the Patriots, the Yankees, the Spurs, etc. — “should” have even more under Elo.

Oftentimes, it’s the unpredictability of sports that make them great — just ask the Nationals. So we’re sorry, Astros. Although the first-ever Elo Championship pennant would be something to behold.