When we first published an interactive checking the accuracy of our various prognostications, back in April 2019, the sports world was relatively normal. The idea of playing in empty stadiums was still novel, game postponements over anything other than weather were rare, and masks were strictly for goalies and catchers (and Rip Hamilton).
But a lot has changed since then — and not in a way that’s especially conducive to accurate forecasts. Although the championship results from sports’ pandemic season(s) have been surprisingly predictable, that’s a pretty rough gauge of how much more chaotic things have been since early 2020.
So with our interactive getting a fresh update this week, let’s dive deeper into how well our sports predictions survived the upheaval that defined the past 15 months.
NFL
The last big change for our NFL prediction model came in 2019, when we added an adjustment to account for the starting quarterbacks in each game. That upgrade served the model well in its debut season, increasing its predictive accuracy as compared with our old system. On a game-to-game basis, we can see that the model is generally calibrated well — even though it is somewhat overconfident in moderate-to-large favorites and a little underconfident in slight favorites.
Some of those trends stayed the same in 2020, as the model was still down on slight favorites: When we said a team had a 55 percent chance to win, the team ended up winning 63 percent of the time. But we were too high on the next tier of favorites — teams with a 60 percent chance actually lost 55 percent of their games — and underconfident on teams in the 70-to-80 percent range. (Then again, all of those differences were well within the bounds of a 95 percent confidence interval around each bin’s predictions.)
Overall, our NFL predictions had a surprisingly good year in 2020, despite the pandemic backdrop. The Brier score for our pregame win probabilities last regular season (0.208) was the most accurate mark in any year since 2015, and teams favored by the model won 68.6 percent of their games — the best record for favorites in a season since 2015 as well.
MLB
Baseball is the sport where we traditionally go out on a limb the least in our predictions, since each game is so unpredictable. (For instance, the top-ranked Dodgers would have only a 72 percent chance of beating the bottom-ranked Pirates at a neutral field, and that’s about as extreme as mismatches can get in MLB.) So our calibration plots tend to be bunched a lot more tightly around 50-50 matchups in baseball than in sports such as football and basketball.
In 2020, our model handled the toss-up games well, though it was somewhat inconsistent with moderate favorites: Teams with a 60 percent chance of winning actually won 63 percent of games, while teams with a 65 percent chance won only 60 percent of the time. But with bigger mismatches, it was on the money — teams with a 69 percent chance to win won exactly 69 percent of the time.
In terms of overall accuracy, 2020 wasn’t our baseball model’s worst year — though it wasn’t its best, either. The model’s 2020 Brier score (0.243) was tied with 2017 for the worst showing since 2016, and a 57.1 percent winning percentage for favorites — while better than in 2016 or 2017 — was more than 2 percentage points worse than it had been in 2018 and 2019 (59.4 percent).
NBA
Our primary NBA model has probably undergone the most radical evolution over the years since we started predicting games, moving from a pure Elo-based forecast to a hybrid version that also used player ratings, and finally to a system based totally on player ratings. (And even within that last form, we’ve used a few different metrics for the player ratings.) So when we look at the model’s performance over many years, it’s synthesizing all the pros and cons of the various different ways we’ve made predictions over time.
Still, there are some overall themes that were true in 2020, deeply strange as that season ended up being. The NBA models tend to be overconfident in favorites, consistently forecasting a higher win probability for teams above 50 percent odds than the rate they actually win at. The most extreme example in the overall sample is for teams with 75 percent odds, which only won at a 66 percent clip. You can see this in the 2020 sample as well, with teams in the 75 percent bin winning only 66 percent of games and teams in the 80 percent bucket winning at only a 69 percent rate:
All of those differences are within their bins’ respective confidence intervals, but it does seem to be a trend — and perhaps indicative of how hoops favorites operate nowadays, no matter how much we try to account for load management and the like.
Either way, in terms of accuracy, 2020 was a season to forget for our NBA model. The regular season was second only to 2017 in terms of the worst Brier scores since 2016, and things got worse in the playoffs. (In response, we made a big change to how we handled our RAPTOR metric before the 2021 season.) With a winning percentage of just 64.9 percent for favorites, 2020 edged out 2017 (65 percent) as our model’s worst year for predicting winners since 2016.
Club soccer
Soccer is somewhat different from the sports listed above, since draws are a very regular occurrence in game results. Still, we can compare win-probability forecasts with actual outcomes using the same format as our other calibration plots:
In general, our soccer forecasts line up really well along the calibration line. They do tend to be slightly underconfident in heavy underdogs and, at the other end of the spectrum, slightly overconfident in massive favorites. That trend played out a bit in 2020 — teams with a predicted 80 percent chance of winning won at only a 72 percent clip. But those types of lopsided matchups are rare; the vast majority of soccer match-win probabilities fall into a range between 15 and 50 percent, where the model is best calibrated.
Accuracy-wise, our soccer model didn’t do quite as well as usual in 2020. Its Brier score (0.161)3 was the worst it’s been since 2017, and its winning percentage for favorites (61.6 percent) was its lowest mark in that span. The absence of home-field advantage — the effect of which we had to guess at, with empty stadiums for much of last season — may have had a particular effect on club soccer predictions, as home teams in our sample went from winning 57.8 percent of matches (again, counting ties as half-wins) in 2017, 2018 and 2019 to just 54.6 percent in 2020.
March Madness
There were no men’s or women’s NCAA basketball tournaments in 2020. But we did have them in 2021, and I thought it would be interesting to see how those tournaments — as well as the NCAA models in general over the years — fared.
For a sport built on its “madness,” the college basketball models are generally well calibrated. Both the women’s and men’s versions are slightly overconfident in teams in the 25 to 30 percent range of win probabilities, as well as for those in the 80 to 85 percent range. All of those differences are within the confidence bounds for each bin, though:
To a certain degree, 2021 was a tale of two very different tournaments. While the men’s side settled down a bit after a host of early upsets, it still ended up with its worst Brier score (0.212) since we started predicting March Madness in 2014. After Baylor capped off the tournament with a surprising knockout of favored Gonzaga in the championship game, favorites finished the 2021 men’s tourney with just a 62.7 percent win rate — easily the lowest for any year in our sample. But the women’s tourney was a bit more predictable. Its Brier score (0.133) wasn’t as good as it had been in 2015 (0.107) or 2019 (0.118), but it was much better than in 2018 (0.162), ending up around the overall average for tournaments since 2015. Likewise, favorites won 82.5 percent of their games in 2021, which wasn’t quite a six-year high4 but was better than the historical average for the women’s tournament.
All in all, the difficulties and disorder of 2020 (and beyond) clearly showed up in our sports forecasts over the past year-plus. Although some leagues were fazed less than others, with the NFL somehow coming out as predictable as ever — because of course it did — many of the sports we track had some of their most chaotic seasons in recent memory. Chalk it up to changes in home advantage, less reliable data on team performance, or simply players and coaches having to worry about real-world problems in addition to on-field ones. Whatever the reason, most sports got a lot harder to predict with the pandemic raging all around them.