In “No Team Can Beat the Draft,” my colleague Neil Paine argued that the player-evaluation market in the NFL appears to be pretty efficient. No team or general manager has shown an ability to consistently pick players who earn better stats in their first five years than we would expect given where they were taken in the draft.
Neil’s argument doesn’t imply that no teams are better at general draft strategy than others. Some add value by trading their picks for more valuable picks. Some get value by picking in the “sweet spot” — the area where the return on investment for a pick is the highest (typically late in Round 1 or early in Round 2) — more often than others. Some address their team needs better than others. And so on.
Of course, there’s also a lot of uncertainty in how players are valued. For Neil’s analysis, that’s not particularly important, because any arbitrary metric will do: If someone drafts better players, they will generally have more fruitful careers as measured by most metrics. But one weakness that almost all of those metrics share is not really understanding how much each player contributes to the overall quality of a team. This is a completely open question in football, with a wide range of estimates.
So rather than looking at how players perform, I thought I’d take a quick look at how teams making certain types of picks perform overall over the next five years, relative to how we would expect teams of their strength to perform over the same period.
For example, here’s how teams perform relative to their SRS (Simple Rating System, or margin of victory adjusted for strength of schedule) depending on where in the first round they drafted (using data from 1970 onward):
Teams drafting in the top five slightly exceed expectations, while teams in the early and mid-round underperform; teams who draft late in the round do better than we’d think (again, this is in addition to the fact that teams who draft late are generally stronger).
This is completely consistent (and, in fact, supports) the Massey-Thaler analysis, which suggests that early first-round picks are somewhat valuable, mid-round picks are not, and then the value of picks rises steadily until peaking in the second round.
If we use this result to adjust for when a team makes its picks, we can start to look for other patterns, such as how teams have done after drafting players of certain positions in various spots.
For the following table, I first calculated each team’s expected performance (in SRS) over the next five years based on its record, its SRS in the previous year and where it was making their draft pick. Then I compared that expectation to its actual performance, broken down by its draft slot and the position of the player it drafted. Here are the results for the eight positions most commonly drafted in the first round:
Granted, these results are noisy. A lot of the difference is likely just variance, but slightly more results seem statistically significant than we would expect. A couple of the strongest results also reflect plausible theories:
First, teams who take quarterbacks in the first five picks tend to perform better than expected (unsurprisingly), but teams who take QBs later in the round typically perform worse. This is the result I was hoping to find, based on the theory that marquee quarterbacks are easy to identify, but sorting out the rest is difficult. In other words: If the top QBs are off the board, it’s probably not worth spending a first-round pick (and more money) speculatively.
Second, tackles possibly aren’t as valuable as people think. Guards didn’t make the list because a lot fewer of them were taken in the first round, though for those that were, the teams who took them did better than average. This could be explained by tackles being poorly-valued by the market relative to guards.