From NFL front offices to fan message boards, the amount of time spent arguing over which players teams should draft is mind-boggling. Ahead of the 2016 draft — which begins tonight — the prospect-focused site WalterFootball.com, for example, has compiled 315 mock drafts from across the internet. And apparently it’s been a slow year; in 2013, it collected 618.
Such obsessive study would be unnecessary if the right answers were obvious. They rarely are. So some observers have called the draft a “crapshoot,” but things are more complicated than that, too. There’s plenty of data to suggest that the draft acts like an efficient market and that when a player is picked speaks volumes about what kind of career he will have. We studied it ourselves, looking for evidence that teams know what they’re doing. But every step of the way, we also found reasons to believe that many of the measures used to quantify a draft pick’s success contain flaws — some relating to draft position itself — that may be unavoidable for now.
For our draft research, we used information from Pro-Football-Reference.com for players who were drafted in 1990 or later, made the NFL and retired before 2013. First, we looked at the most basic measure of NFL success — the average length of a player’s career — based on the round in which he was drafted.
No surprises here: The higher the draft pick, the longer a player will stick around in the NFL. First-rounders last a year longer than second-rounders, and the same goes for second-rounders compared with third-rounders. The gaps between rounds narrow slightly in the latter half of the draft, but a seventh-round pick like Mr. Irrelevant, the last pick of the NFL draft, can expect a career just under half as long as the average first-rounder.
This is evidence that teams are getting better talent in earlier rounds. And these different career lengths can’t be explained away by different positions being drafted in different rounds: Aside from quarterbacks being more common in the first round and special teamers being more common later in the draft, the positional breakdowns are remarkably similar across rounds.
But there are several issues with using career length to measure draft success. One is that it’s undeniable that early-round draft picks are given more opportunities than most players. In addition to whatever real (or perceived) talent advantage allowed them to be picked highly, first-round picks are lavished with extra coaching and playing time not afforded to lower-status players, likely helping their careers last longer. Although NFL teams have a very vested interest in winning, it can be difficult to know when to give up on a highly touted prospect, particularly if the decision makers who selected him are still in place. This, in turn, makes it hard to tell how much of the longevity advantage of higher picks is earned and how much is a consequence of merely being picked so highly.
Career length isn’t the only measure whose flaws can hamstring player evaluation. As another proxy for NFL success, we can use appearances on the All-Pro team, which honors the best players at each position in a given season.
Just as with career length, players taken earlier in the draft are much more likely to be honored as All-Pros. And the distinction between the first round and the remainder of the draft is impressive: About a quarter of all first-rounders who make the NFL find their way on to an All-Pro team, compared with roughly 10 percent for second-round picks. Even within a round, earlier picks tend to do better; for every additional pick in the first 100 selections, a player sees his chances of making an All-Pro team decline by about half a percentage point. This finding is similar to Chase Stuart’s research on the correlation between draft position and Pro-Football-Reference’s Approximate Value metric.
When you look at these results by position, however, some players appear to benefit more than others from the reputation that comes with a high draft pick. Linebackers and offensive linemen, for example, are among the positions most likely to see first-rounders named to an All-Pro team, and they’re also among the most difficult positions to judge statistically. It’s possible that teams find it easier to assess the potential of players at those positions, making it more likely that they’ll draft players who go on to become stars. But it’s also possible that players with good pedigrees and name recognition are being given preferential treatment when awards are granted, in the absence of meaningful performance data (like what exists for offensive skill players) to challenge our perceptions.
Perhaps tellingly, the one position that doesn’t show a strong tendency for first-round picks to turn into All-Pros is quarterback. That might mean good quarterbacks are especially hard to identify, but it also may not be a coincidence that we have better, more granular performance data for quarterbacks than any other position, even compared with other offensive “skill players.” With more data, the media members who decide All-Pro status are better at differentiating between good and bad performers without having to resort to prior information such as draft position. And it follows that they may not be doing that for positions with less data.
With all this uncertainty, teams are increasingly finding roles for players who enter the league outside the orderly stratification of draft day. Although undrafted players rarely persist on NFL rosters for very long — their average career length has declined more severely than drafted players in recent years — the league is churning through more players than ever before, and the overall pool of undrafted players has never been larger. According to our research, the number of undrafted players on NFL rosters increased from 497 players in 2005 to 746 in 2014 — a 50 percent increase.
Criticisms of NFL draft decision-making usually focus on outliers such as Tom Brady infamously being selected with a sixth-round pick, but studies consistently show that teams do a solid job of sorting talent by pick and by round. At the same time, however, that skill varies substantially depending on a player’s position — and therefore the amount of data we have to judge individual performance. In the absence of better data, this means we still don’t really know how much of a crapshoot the draft is, no matter how much we study it.
CORRECTION (April 28, 2:40 p.m.): An earlier version of the chart in this article gave an incorrect description of the data. The chart shows players drafted since 1990, not just those drafted since then who retired between 1990 and 2013.
Disclosure: Author Zach Binney works as an analyst for an NFL team.
