As he prepares for his third season in the NFL, Andrew Luck of the Indianapolis Colts is already regarded as one of the league’s premier quarterbacks. ESPN’s Mike Sando polled 26 NFL front-office and scouting types this month, asking them to sort every current starting quarterback into one of five tiers; Luck was in the top group, ranking fifth overall.
In terms of justifications, the Colts’ record under Luck speaks for itself. Since installing Luck as its starting QB before his rookie season of 2012, the team has won nearly 69 percent of its games (posting a pair of 11-5 campaigns). Luck also looks like a prototypical quarterback. At 6 feet 3 inches tall and 235 pounds, Luck’s size jibes far more with the preconceived notion of what an elite QB should look like than, say, Seattle’s Russell Wilson (who stands 5 feet 11 inches). Further, Sando’s scouts are probably the same people who regarded Luck as “the top quarterback prospect to come along in the past 30 years” when he was at Stanford. Research in psychology has shown that it’s incredibly difficult to alter people’s first impressions, so it might have taken a Ryan Leaf-esque flop for scouts to change their opinions about Luck’s potential superstardom.
In terms of individual statistics, however, Luck has hardly been a top-five quarterback. Adjusted Net Yards per Attempt (ANY/A) — which quantifies a quarterback’s passing efficiency while taking into account his touchdown passes, interceptions and sacks — ranks Luck 18th in the NFL over the past two seasons. He finished 16th in 2013. By that measure, Luck hasn’t even been an average passer thus far in his career.
Depending on the stat or the scout, Luck is mediocre or good. Or great. Or average.
What’s going on?
Even by the standards of individual football statistics, ANY/A is a rough metric. It doesn’t account for strength of schedule, and it treats all yardage as equal regardless of the situation. Other metrics, like ESPN’s Total QBR (in which Luck ranked ninth a year ago) and Brian Burke’s Expected Points Added (eighth) and Win Probability Added (sixth), regard Luck more highly, suggesting that he’s performing well in QB rushing and clutch play, which conventional yardage-based passing metrics don’t effectively measure.
Indeed, Luck’s running game is often overlooked. In its Defense-Adjusted Yards Above Replacement (DYAR) metric, Football Outsiders ranked Luck as the top rushing QB in the NFL last season. But even if we add Luck’s rushing DYAR to his passing tally over the past two seasons, he ranks 12th among quarterbacks in total value added above replacement.
The same goes for Pro Football Focus’s play-by-play grades — by their estimation, Luck was third among QBs in rushing value added above average. But those grades also rate Luck as barely better than an average QB as a passer. Again, if we combine passing and rushing (plus penalty avoidance) and convert those grades to assess value against replacement (instead of average), Luck has been the 14th-ranked quarterback over the past two seasons.
In both sets of value-over-replacement rankings, Luck narrowly places ahead of the Carolina Panthers’ Cam Newton, who makes for a particularly interesting point of comparison. Statistically, the two quarterbacks offered similar production a year ago, but their reputations couldn’t be more different. Luck is widely known as a “winner“; Newton has been dogged by criticism as a “stat-padder” who doesn’t deliver in big moments. Those sentiments probably played a large role when Sando’s execs placed Newton in the third tier despite having a quantitative résumé similar to Luck’s. Backing up that viewpoint, Christopher Price writes in Football Outsiders’ 2014 Almanac: “Newton is a very talented quarterback, but nobody in the league would take him over Andrew Luck right now.”
Of course, the biggest reason professed by the executives for their high placement of Luck was his team’s success despite a weak supporting cast, which gets at the fundamental limitations of evaluating individual NFL players with metrics and the eye test. “The evaluators think Luck has carried a subpar roster to a 22-10 record without much help,” Sando writes.
Football Outsiders has a great caveat about individual advanced stats that goes along these lines:
“In 2013, Andrew Luck had 650 DYAR. But what we are really saying is ‘In 2013, Andrew Luck, playing in Pep Hamilton’s offensive system with the Indianapolis offensive line blocking for him and Donald Brown and Trent Richardson providing rushing support, had 650 DYAR.’ “
The truth is that in football, perhaps more than any other sport, a player’s statistics depend greatly on the talent around him. And the common perception is that Luck’s supporting cast is similar to that which went 2-14 with Curtis Painter, Dan Orlovsky and Kerry Collins under center in 2011. That might not be entirely true, but it’s largely beyond our statistical capabilities to test assertions like those empirically.
We can attempt to measure the talent and experience level of isolated parts of an offense, but in the end we’re left with rough estimates. And what do we do if Luck’s early career ends up mirroring that of John Elway — a prestigious QB with surprisingly ordinary stats? We’ll be left wondering whether it was a coincidence if his stats jump when the 2024 versions of Rod Smith and Ed McCaffrey come along. (Or think of Tom Brady, the poster child for a quarterback’s statistical production not matching his reputation … until Randy Moss arrived in New England.)
So, in a certain sense, we still don’t know how good Luck is. He’s probably better than his raw efficiency rates suggest, but maybe he’s not quite as good as his top-tier ranking among NFL scouts and executives. It’s a fuzzy picture of individual value that NFL fans have to settle for while we wait for new ways to more effectively disentangle players’ contributions from one another.