Warren Buffett has a great investment track record. So perhaps it’s no accident that he declined to offer $1 billion for correctly predicting the outcome of all World Cup knockout stage games, as he did for the NCAA men’s basketball tournament games.
The odds of winning Buffett’s NCAA challenge were about one in 7.4 billion, assuming you chose the favorite in each game as selected by FiveThirtyEight’s NCAA model. But the odds of correctly filling out a 16-team knockout tournament such as the World Cup are much shorter.
In fact, though there have been some thrilling matches in the knockout stage so far — six of 12 have gone to extra time and only two were decided by more than one goal — the favorite has advanced every time (at least as according to the FiveThirtyEight’s World Cup predictions).
Here is the breakdown so far. The following table lists the win probability for the FiveThirtyEight favorite as of the day of the match, along with the cumulative probability of the model having called all knockout stages correctly up to that point in time.
For instance, the probability of correctly identifying the winners in each of the first four knockout matches — Brazil over Chile, Colombia over Uruguay, the Netherlands over Mexico and Costa Rica over Greece — was about 23 percent, or one chance in 4.3. And the chance of going 12 for 12, as the FiveThirtyEight favorites have done so far, is just one in 75.
It’s an upset, in other words, when all the favorites prevail. On average, we’d have expected three or four upsets through this point in the knockout round.
Of course, there are four matches left — counting the World Cup’s third-place playoff between the two semifinal losers. According to the FiveThirtyEight forecasts, Brazil is favored over Germany on Tuesday (even after accounting for Neymar’s injury) and Argentina is slightly favored over the Netherlands on Wednesday. To complete a perfect knockout bracket, Germany would then need to beat the Netherlands in the consolation game while Brazil prevailed over Argentina in the final.
All of the remaining matches look pretty close, so the FiveThirtyEight forecasts are likely to fail at some stage. If the model gets the matches right, however, it will have made good on a 1-in-553 chance of calling all 16 knockout stage winners correctly.
Incidentally, this isn’t the huge success for the FiveThirtyEight model that it might seem. The FiveThirtyEight forecasts are probabilistic. Teams listed as 75 percent favorites are supposed to win about 75 percent of the time over the long run — not much less than that but also not any more often. There are supposed to be some upsets. If 75 percent favorites are winning 100 percent of the time over the long run instead, that means the forecasts are miscalibrated and overestimating the chances for the underdogs.
In this case, the success of the favorites does seem to be mostly a matter of luck. Three games have gone to a penalty shootout so far — pre-match favorites might have a slight edge in those but not much of one. Mexico, meanwhile, was a few minutes away from defeating the Netherlands, and the U.S. was a few inches away from beating Belgium.
The best way to test probabilistic forecasts is to check their calibration and to compare them against alternative probabilistic estimates. For example, if your model says that the U.S. has a 40 percent chance of beating Belgium and the consensus betting line gives the U.S. just a 25 percent chance instead, you should bet on the Americans — even though you expect Belgium to win most of the time. So far, the FiveThirtyEight forecasts have done well against consensus betting lines when used in this fashion — although that could reflect good luck, too.