So your favorite team just got a good grade in this year’s NFL draft. Congrats! Surely this is the first step on the path to future Super Bowls, right?
Well, umm … not exactly.
The NFL draft is a hard thing to figure out even if you are among the best talent evaluators in the world. There isn’t a proven formula for consistently picking diamonds in the rough — and the same goes for us media folks who evaluate the evaluators. While it’s hard to resist the urge to make instant judgments on which teams did or didn’t have a good draft, those assessments mean very little. Truth is, there is almost no correlation between a team’s draft grades and its future performance — or even the performances of its players specifically from that draft, when judged in retrospect.
Watch: https://abcnews.go.com/fivethirtyeight/video/makes-patrick-mahomes-great-fivethirtyeight-75237605
To examine this, I turned to an incredible archive of draft grades collected by Football Outsiders going back to the mid-2000s. After every draft, they compile each team’s letter-grade assessments from a variety of outlets, including ESPN’s Mel Kiper — the godfather of draftniks. They also converted those grades into a composite “grade point average”-style score, which can serve as a consensus for how the media thought a given team’s draft went in any given year.
related: Do Alabama Alums Deliver In The Pros? Read more. »
Using that data going back to 2005,29 I normalized the GPA scores so that the average draft received the same grade each year — to prevent any instances of grade inflation — and looked at the correlation between a team’s consensus draft performance and its Simple Rating System (SRS) score30 in future seasons. And in general, there was very little relationship (a correlation of 0.058) between a team’s consensus draft grade and its SRS five years down the line:
It’s not that draft grades tell us nothing at all about how good a team will be in the future. For instance, after controlling for their SRS in the previous season, we would expect a stellar, A-plus drafting team to be about 2.7 points of SRS better than a terrible, D-minus team after five seasons. But at the same time, draft grades explain only 0.3 percent of the variation in team performance over that time frame, which means they miss the vast majority of what actually determines whether teams succeed or fail.
Maybe that’s not unexpected, though. In the NFL, predicting team performance is really difficult — to the point that even guessing at it for next season is largely a useless exercise. (Not that it stops us from trying!) Since the draft is only one component of roster building, and probably the one with the biggest emphasis on hazy, long-term bets, it’s not necessarily surprising that snap judgments on draft day aren’t actually that accurate when it comes to predicting a team’s future arc.
So perhaps looking at overall team performance is the wrong way to gauge the effectiveness of draft grades. Limiting our scope some instead, we might expect draft grades to at least carry some relationship to how well a team’s prospects from that specific class end up performing. Yet they tend to fall short in that department as well.
To measure each team’s draft-class production, I collected Pro-Football-Reference.com’s Approximate Value (AV) for players in their first five seasons after being drafted.31 I also compared that with how much value we’d expect each player to produce based on where he was drafted (according to Chase Stuart’s AV-based draft value chart), to tell us whether teams ended up getting good value for their picks (which is theoretically what draft grades are ultimately trying to evaluate). Comparing draft GPAs with the total AV each class ended up producing, draft grades appear to perform better — with a 0.325 correlation between the two categories (or 10.6 percent explanatory power). But almost all of that is simply driven by teams with higher expected-value picks tending to get better grades from the media.32 After adjusting for the inherent value of each pick, a team’s draft GPA has only a 0.144 correlation with how productive that draft class ended up being:
Again, we would expect a higher-graded draft to end up getting slightly more value from its picks than a lower-rated one. But overall, draft grades explain only 2.1 percent of the variation in team drafting performance — i.e., the primary thing they are trying to measure.
Sometimes, a well-regarded draft does end up doing great things. Dallas’s 2005 class received the league’s highest consensus mark, and it became the fourth-most valuable draft group of the past decade and a half thanks to the performances of DeMarcus Ware, Jay Ratliff and Marion Barber III. But just as often, a poorly graded draft will also rise to the top — the best draft in the entire sample by AV (Seattle’s in 2012) received a consensus C-minus grade — while a draft with high marks will flop horribly, like the 2005 Vikings, who were given a consensus A-minus but ended up being the most underperforming class in our sample.
It’s something to keep in mind during this period of rapid-fire post-draft judgments. In reality, it takes years to know how productive a draft class will end up being, and most experts’ guesses before then are just that — guesses.
Watch: https://abcnews.go.com/fivethirtyeight/video/chaotic-fictional-football-coach-fivethirtyeight-74531599
Watch: https://abcnews.go.com/fivethirtyeight/video/baseball-fan-negro-league-stars-fivethirtyeight-76389391