The NFL Was Weird Last Season

The 2019 NFL season will be remembered for a host of unusual storylines. Early in the season, New York Jets quarterback Sam Darnold came down with mononucleosis and missed the next three games to sickness and concerns about a swollen spleen. Shortly after he returned, as New York was trounced 33-0 by the Patriots in Week 7, Darnold lost a toenail and admitted he was seeing ghosts live on the broadcast. The spookiness continued three weeks later on Nov. 4 when the Cowboys-Giants game was interrupted for a minute and a half by a black cat prowling around MetLife Stadium.

It was, by any accounts, a weird year. But the strangeness in 2019 wasn’t confined to human and feline drama.

Despite a Super Bowl-winning run¹ that showcased the robustness of a strong passing attack, the Kansas City Chiefs were one of only two NFL teams to pass more than they ran against stacked boxes on first down and 10 in open-field play with the game outcome still in doubt.² These situations — something we call FANS, or First down and 10 Against Neutral or Stacked boxes — are game states where the offense is faced with a large incentive to pass because of the structure of the defense.

Defensive structure is largely dictated by the personnel groups an offense sends on the field, so the offense controls when it gets these looks to a substantial degree. When a team sends out smaller, faster players like wide receivers, the defense matches with smaller, faster players of its own. And when the offense sends out two tight ends, the defense tends to crowd the box to fit the extra gaps created by the big men. Given the situation, we might expect teams to pass at least as much as they run just to keep the defense on its toes. After all, passing is so much more effective against these neutral and stacked boxes — especially on first down — that despite the talent disparities between NFL teams and their various strengths and weaknesses in personnel, 28 out of 32 teams had a higher expected points added per play on dropbacks than on runs.

It’s hard to overstate the advantage passing has over running on FANS. The difference in EPA per FANS play between rushing and passing across the league was almost a quarter of an expected point in 2019 — significantly more than the meager -0.05 EPA per play teams averaged rushing. Even the Jets and Darnold — spleen be damned — were wildly more effective passing vs. handing off.

Or consider the Baltimore Ravens, one of the most prolific rushing teams the NFL has ever seen. Baltimore was over three and a half times more effective passing on FANS than running, but the Ravens threw the ball just 30 percent of the time in those situations.

And while it’s probably unfair to criticize the Ravens’ run-pass splits on FANS since their rushing EPA per play was greater than nearly half the league, doing what the defense doesn’t expect is a consistently winning strategy in the NFL. Since at least 2008, our first year with data on defensive box counts, passing has completely dominated running on FANS. Despite claims about analytics going mainstream in the NFL, this obviously exploitable edge continues to go underutilized. In 2019, 63 percent of plays run on FANS were rushing attempts — a figure mostly unchanged over the past decade-plus.

It’s a bit of a mystery why exactly teams line up in a run look and then end up doing what the defense expects over 60 percent of the time. To make up for its overall (relative and absolute) inefficiency, the strategy would need to confer a massive psychological benefit to the offensive team.

Still, NFL teams persist. Interestingly, the efficiency on FANS pass plays doesn’t owe itself exclusively to play-action.³ Teams tend to throw deep more often when they do decide to pass on FANS, play-action or no, and deeper passes have a higher average value than their shorter, dink-and-dunk brethren.

If passing on FANS is the test, 2019 was a tough scene for analytics aficionados. Still, the news wasn’t all bad for fans of efficient football. Play-action fakes in the regular season were up 28 percent from 2008, the first year for which we have data. Non-play-action passing overall was up as well over the same period, but it rose just 3.6 percent. So play-action passes now account for a larger share of passing attempts than they did 12 years ago. And, crucially, that increase in volume has not led to the fake being less effective.

If there’s a point where the play-action fake will suddenly become less effective, the NFL has yet to find it. And because the effect is so pervasive and seemingly robust with respect to talent and context, it’s difficult to argue that any team would be wrong to make play-action a more prominent part of its offensive attack.

The same, unfortunately, can’t be said for fourth-down attempts. In 2019, NFL teams went for it when the game was still in doubt more than at any time in the past 12 years. Yet conversion rates on fourth-down attempts plummeted to their lowest level since 2013 and the third-lowest since 2008.

It’s a strange result for a play call that most analysts⁴ believe should be called even more frequently than it currently is. And while we’re dealing in small samples at the team level — and a leaguewide conversion rate of 53 percent is neither unprecedented nor a disaster — it should probably give rabid analytics fans at least a moment of pause. After all, the high variance nature of the play call is sort of the point of those arguing against going for it on fourth down more often. It’s easy to rationalize away negative variance when you aren’t the one on the sideline tilting its effects, all while trying to keep your team’s spirits up.

Evidence-based analysis that ignores the emotion of the game probably does itself few favors. Progress seldom comes easily in any competitive venture, much less in the neat linear trendlines that make for good charts. And as fans, we’re seemingly more open to the idea that a black cat stopping play in Week 7 helped cause a Cowboys comeback than we are to admitting that a failed fourth-down attempt gave our favorite team a better chance at winning.

Based on the most recent results, we should probably temper broad proclamations about the influence and effectiveness of quantitative analysis in the NFL. The edges afforded by analytics are small; even the best and most sound approaches aren’t appreciably different from the ad-hoc, irrational ones that numerate analysts tend to mock. Instead, it’s better to generate strong opinions based on the best available evidence, but to hold them weakly.

So when we scream for a pass on first and 10, for a coach to call more play-action or for a team to be more aggressive on fourth down, we’re not professing false confidence in a certain outcome. It’s a preference born of an intimate understanding of just how much we don’t know — our ignorance, quantified. And in the face of that uncertainty, it’s an appreciation for the bravery to make the unconventional choice.