Five Thirty Eight World Cup Projections: How Did They Do?

We have gotten through the Group Stages, mostly intact. We can now take a step back and assess what has happened. That goes for soccer prognostication as well. Nate Silver’s site FiveThirtyEight.com has been posting daily capsules that contain game projections, with win percentages assigned to each team, as well as the chances of a draw.

Since the start of the tournament, I have been tracking these predictions against the results, and also comparing them to odds listed by a wagering site. Regardless of the results, I was going to publish them to see just how the estimates measured up.

How have Silver’s projections done thus far?

Pretty darn good.

Let’s start with this. The 48 teams listed as the “favorite” at FiveThirtyEight.com were expected, if we count draws as half-wins, to win 31.65 matches (65.9%). Those same 48 teams actually won 32.5 (67.7%), meaning the final results for group play were within one win of the overall projections for the favorites.

The 16 matches with the highest winning percentage projected for the favorite ended up even outperforming the projection, and those heavy favorites went 14-0-2 with a 38 to 12 goals advantage.

It’s not just, though, that the projections were pretty close to actual results in aggregate. They’ve also outperformed the betting market. To test this, I calculated the game projections based on the betting odds (and removing the vig to get them to equal 100% for win-lose-draw). I then compared the FiveThirtyEight.com odds to see which of the three outcomes had the largest positive discrepancy. For example, everyone had Costa Rica as an underdog versus Italy. FiveThirtyEight.com gave Costa Rica a 30% chance, though, compared to 15% based on the betting odds. Thus, the “pot odds” would dictate a play on Costa Rica (similar to how one might bet a live underdog at the money line in the NFL even if the team was still viewed as more likely than not to lose).

Using a simple “bet $100 on every play for FiveThirtyEight.com” model, you would have made $2,716.93 through the group stage. Seth Burn did a more complicated tracking, wherein he used the Kelly criterion along with the FiveThirtyEight.com projections, putting the optimal amount on the betting line that offers the highest expected value. A fictional $100,000 bankroll would be at $647,555 after the group stage.

Here were the 10 games where the odds of winning differed the most between the betting odds projections and the FiveThirtyEight.com projections. The teams went a combined 7-1-2 (and as American fans know, were within a 95th minute goal of being 8-1-1).

Why did FiveThirtyEight.com perform so well in the group stage?

SPI, the ratings that are the underlying basis for the projections, is based on a combination of national team results and an estimate of each roster’s underlying “talent” using club-level performance.

I spoke with Neil Paine of FiveThirtyEight.com about some of the projections and underlying rationale. Paine stated that the player-rating component, which had France at #1, showed a correlation of +0.5 with team points accumulated in the group stage. “With all of the noise in the differing group strengths and even in the game outcomes themselves, I think that’s a reasonably strong relationship between the talent ratings and what we’ve seen play out on the pitch,” Paine said. (For comparison, the correlation between point differential in the 2012 NFL season, and team wins the following year was +0.26).

The simplest answer for why the projections have been fairly good is that the site projections were more bullish on the South American and CONCACAF teams than the betting market. Of the 10 teams from the Western Hemisphere, 8 of them are alive in the knockout rounds.

I asked whether the projections had any sort of home continent or region adjustment, beyond that for Brazil as the host nation.

According to Paine:

"“We experimented with those in the lead-up to the tournament, but we couldn’t find evidence for a home-continent advantage per se after controlling from home-country and east-west travel distance. Of course, the horizontal travel-distance factor is currently included in our projections, and probably explains most of what is perceived as any home-continent advantage. So Brazil is getting a boost from being the host and low travel distances (although the effect of the latter isn’t very big)."

"In our research, we also found that the horizontal travel-distance variable mattered a lot more in the past than it does now. It’s very significant if you look at the entire history of the World Cup, but less so in recent Cups, which is why we included it but made it a slight effect. You can probably attribute that change to improvements in air travel over the years."

I also noted that in 16 matches between a team from the Western Hemisphere and a team from Europe, FiveThirtyEight.com was more favorable to the Western Hemisphere team’s chances 11 times, while only favoring the European team twice (3 others were draws).

"Some of that may be due to the horizontal travel distance effect showing up at the margins, although, again, it’s not a large adjustment. (I would have been inclined to think we’d rate the UEFA teams strongly, since the talent component uses data only from the Big Five european leagues — England, Spain, Germany, Italy and France — and the Champions League and the UEFA Cup/Europa League.)"

"It’s also worth noting that the South American Cup qualifiers were above .500 against those from Europe in friendlies over the 2010-2014 span, so South American is probably equal with Europe talent-wise. And, of course, I’ll repeat a fact Nate [Silver] stated before the tournament began: no European team has ever won a World Cup played in the Americas."

Overall, through the group stages, the FiveThirtyEight projections have been reasonable in the aggregate. According to Paine, they have been well received, with most criticism coming from fans of countries who were projected poorly, as well as “hardcore gambling types who took exception to Brazil being a much heavier favorite in our model than in the betting markets.”

So far, Brazil hasn’t necessarily looked as impressive as expected, though they have advanced with a 2-0-1 record (counterpoint: the last four host nations to win the World Cup were 9-2-1 in the group stage, so this wouldn’t be abnormal). Whether they win or not will not prove anything, as even the FiveThirtyEight.com model had them as more likely than not to not win the Cup.

After 48 matches, though, the model appears to be doing reasonably well, even if individual results can be questioned.