The NFL Is Drafting Quarterbacks All Wrong

No position in professional sports is more important or more misunderstood than the quarterback. NFL scouts, coaches and general managers — the world’s foremost experts on football player evaluation — have been notoriously terrible at separating good QB prospects from the bad through the years. No franchise or GM has shown the ability to beat the draft over time, and economists Cade Massey and Richard Thaler have convincingly shown that the league’s lack of consistent draft success is likely due to overconfidence rather than an efficient market. Throw in the fact that young QBs are sometimes placed in schemes that fail to take advantage of their skills,³⁸ that red flags regarding character go unidentified or ignored³⁹ and that prospects often lack stable coaching environments, and there is no shortage of explanations for the recurring evaluation failures.

All of this uncertainty makes the NFL draft extremely exciting: You never know for certain who will be good and who will be an absolute bust. Last year, much of the pre-draft speculation surrounded where current Buffalo Bills starting QB Josh Allen — who is tall and can hit an upright from his knees from 50 yards away — would be selected. This year, when Oklahoma’s Kyler Murray decided to forgo a career in baseball for a chance to become a top pick in the 2019 NFL draft, his measurables captured attention in a different way. Murray, listed at 5-foot-10 and 194 pounds, is 7 inches shorter and more than 40 pounds lighter than Allen, and he’s the the smallest top QB prospect in recent memory. While some scouts and NFL decision makers think Murray’s odds for NFL success are long — or have him off their draft boards entirely because of his lack of size — there is strong evidence in the form of metrics and models that he is actually a good bet to succeed.

Like the rest of the league, practitioners of analytics have a pretty poor track record at predicting QB success. It wasn’t just Browns fans who were high on Johnny Manziel — many predictive performance metrics liked him as well. If some of the world’s best football talent evaluators are convinced that Murray’s height is at least a minor red flag, how can we be confident that a 5-foot-10 college QB will be productive in the NFL? When it comes to the draft, deep humility is warranted. Still, there are solid reasons to be excited about Murray.

Completion percentage is the performance measurable that best translates from college to the NFL. The metric’s shortcomings — players can pad their completion percentage with short, safe passes, for instance — are well-known. But even in its raw form, it’s a useful predictive tool.

Its kissing cousin in the pantheon of stats that translate from college to the pros is average depth of target: Passers who throw short (or deep) in college tend to continue that pattern in the NFL. These two metrics can be combined⁴⁰ to create an expected completion percentage, which helps correct the deficiencies in raw completion percentage. If you give more credit to players who routinely complete deeper passes — and dock passers who dump off and check down more frequently — you can get a clearer picture of a player’s true accuracy and decision-making.

Another important adjustment is to account for the level of competition a player faced. ESPN’s Total Quarterback Rating does this, and we’re doing it, too. For instance, passes in the Big Ten are completed at a lower rate than in the Big 12 and the Pac-12. We should boost players from conferences where it is tougher to complete a pass and ding players whose numbers are generated in conferences where passing is easier.

When we make these adjustments, and then subtract expected completion percentage from a QB’s actual completion percentage, we get a new metric: completion percentage over expected, or CPOE. An example: In 2011 at Wisconsin, Russell Wilson had a raw completion percentage of 73 percent. We would expect an average QB in the same conference who attempted the same number of passes at the same depths that Wilson attempted to have a completion percentage of just 57 percent. So Wilson posted an incredible CPOE of +16 percentage points in his last year of college. CPOE translates slightly less to the NFL than either raw completion percentage or average depth of target,⁴¹ but it does a substantially better job of predicting on-field production. In stat nerd parlance, we’ve traded a little stability for increased relevance.

The test of a good metric is that it is stable over time (for example from college to the NFL) and that it correlates with something important or valuable. Completion percentage over expected is slightly more stable than other advanced metrics like QBR. CPOE is also the best predictor of NFL yards per attempt. Since yards per attempt correlates well with NFL wins, and winning is both important and valuable, we’ve found a solid metric. It should help us identify NFL prospects likely to be good — so long as they are drafted and see enough playing time to accumulate 100 or more passing attempts.⁴²

But before we stuff the metric into a model and start ranking this year’s quarterback prospects, it’s worth asking why CPOE in college might be a good measure of QB skill. One possible explanation is that it’s measuring not just accuracy but also the signal from other qualities that are crucial to pro success. The ability to consistently find the open receiver and complete a pass to him requires a quarterback first to read a defense and then to throw on time and on target. Throwing with anticipation and football IQ are both crucial to playing in the NFL at a high level, and they are likely both a part of the success signal in the metric.

CPOE is also probably capturing the ability to execute a system efficiently. A quarterback who understands how each piece of the offense complements the others and constrains the opposing defense is a huge asset for his team. The term “system QB” has a negative connotation in player evaluation circles that is probably unwarranted. If a quarterback is operating at a high level, he is inseparable from the system he’s being asked to run. It’s also likely the case that the mental and physical abilities to run any system efficiently are traits that translate — even if only imperfectly — to the pro game.

CPOE also measures accuracy, of course — which many believe is the most important trait a QB can posses. Some coaches believe accuracy is an innate skill and not something that can be taught once a player has reached college. Others believe that mechanical flaws can be corrected if other traits like arm strength are present. The Bills clearly hold this view or they wouldn’t have drafted Allen, a player with an incredibly live arm but who had a college completion rate 9.2 percentage points below expected. But regardless of whether accuracy can be taught at the NFL level, all evaluators acknowledge its importance.

With all this in mind, I built a simple logistic regression model that attempts to identify players who will go on to establish a career mark of at least 7.1 yards per attempt in the NFL — the league average from 2009 to 2018. The model took into account CPOE and six other metrics, all calculated for the player’s college career.⁴³ There are 49 quarterbacks who have entered the NFL since 2012 who have also attempted at least 100 passes — except for small-school QBs for whom advanced college data wasn’t available. I randomly split those players into two sets and used one set to build the model and the second set to test to see if the model is any better than random chance at identifying which prospects will go on to play productive NFL football. Though the model is relatively simple — and it would be wonderful if the sample size were larger — the results are promising. The model correctly identified many players who went on to have NFL success and many more who didn’t. The best estimate for its generalized accuracy is that it will correctly identify a QB prospect as a hit or a bust in around 74 percent of cases.⁴⁴ The table below shows the results of the model, labeled Predict, and includes players’ college stats.

Humility is warranted at this moment, so let’s point and laugh at the failures first. After all, all models are universally wrong, but some can be useful. This one was wrong about Johnny Football, as it practically guaranteed Manziel to be an above-average NFL quarterback. What it didn’t know about was Johnny’s love of all-night parties and other off-field shenanigans. Kellen Moore, a lefty passer who had a decorated career at Boise State, is another hiccup for the model. Moore is an interesting case of a player who just barely reached the 100 passing attempt threshold and eclipsed 7.1 yards per attempt for his NFL career but still bounced around the league and never found success or even a starting job. So the model predicted his statistical success in yards per attempt but not his actual success on the field. The problem here is that our success metric — career yards per attempt over 7.1 — doesn’t perfectly discriminate between good and bad NFL QBs. Much like human evaluators, models can sometimes be right for the wrong reasons, and Moore is a prime example.⁴⁵

The model was also suspiciously bad at predicting Lamar Jackson, ranking him as an almost sure bust as a passer. Jackson’s career yards per attempt — most of those attempts coming in just seven games — is right at the 7.1 threshold, and while he is no one’s idea of Drew Brees, a success probability of zero seems an overly harsh assessment for a player that has clear talent — especially running the ball — and has already helped his team to the playoffs.

Still, the good outweighs the bad. The only other false negatives in the bunch are Kirk Cousins and Marcus Mariota, both of whom have career yards per attempt figures above 7.1. Meanwhile the low probabilities assigned to passers like Nate Peterman, Zach Mettenberger, Paxton Lynch, Geno Smith and Blake Bortles all appear reasonably prescient.

Looking forward and applying the model to the current draft class, we find a few surprises. Kyler Murray sits comfortably at the top with a 97 percent probability of being an above-average pro quarterback. Murray’s physical and statistical production comps with Russell Wilson are especially striking. Wilson and Murray had roughly the same yards per attempt in college, identical average depth of target and similar Total QBR.⁴⁶ Both are also under 6 feet tall and played baseball at a high level. As far as comps go for short QBs, you really can’t do any better.

Murray isn’t just a scrambler who excels working outside of the pocket and on broken plays, either. According to the ESPN Stats & Information Group, 91 percent of Murray’s 377 pass attempts in 2018 came inside the pocket, and 81.6 percent of those throws were on target and catchable. Murray faced five or more defensive backs on 82 percent of his passing attempts and threw a catchable pass 78.8 percent of the time. Against nickel and dime packages, he was even better when blitzed, with 79.1 percent of his passes charted as catchable when the defense brought pressure. And Murray didn’t just check down to the outlet receiver when the other team sent heat. Kyler pushed the ball downfield at depths of 20 yards or greater 21 percent of the time vs. a blitzing defender.

Meanwhile the other consensus first-round talent, Ohio State’s Dwayne Haskins, is viewed as much less of a sure thing by the model. While his CPOE is identical to Murray’s and his QBR is similar, the model rates his yards per attempt and low average depth of target as red flags that drag down his probability of success. Nickel is the base defense in the NFL, so a quarterback’s performance against it is important, and Haskins faced five or more defensive backs far less often than Murray, dropping back against nickel or dime on just 63 percent of his pass attempts. And when Haskins was blitzed out of those looks, he was not as adept at delivering on-target passes, with 76.4 percent charted as catchable despite only 6.6 percent traveling 20 yards or more in the air.

Other surprises from the consensus top-four prospects are the rankings of Duke’s Daniel Jones and Missouri’s Drew Lock — both of whom completed fewer passes than we would expect, and both of whom were assigned a low probability of NFL success. Teams should probably be very wary of both players. Since 2011, college QB prospects with completion percentages under expected — a list that includes Brock Osweiler, Trevor Siemian, Mike Glennon, Matt McGloin and Jacoby Brissett — have all failed to post career yards per attempt above the league average of 7.1. Meanwhile West Virginia’s Will Grier — a player few experts have mocked to go in the first round — looks to be the second-best QB prospect of the class. With his excellent college production and nearly prototypical size at 6-foot-3 and 217 pounds, Grier is a player whose stock could rise with a good performance on and off the field at the combine.

There are many weeks of interviews, testing and evaluation left to come for each of these prospects, and analytics are just one piece of the process. Models are certainly not a player’s destiny. Murray might end up profiling as a selfish diva who can’t play well with others. Lock could somehow morph into Patrick Mahomes. But ultimately the model and the metrics agree with Arizona Cardinals coach Kliff Kingsbury’s assessment that Murray is worthy of the top overall pick in the draft. Ship him off to a team with an early pick and a creative play-caller, and enjoy the air raid fever dream that follows.