The concept of constant evolution in sports isn’t exclusive to athletes and tactics. It’s central to the men and women who officiate the games as well.
Referee mechanics — where to stand, how to move, who makes which calls — weren’t taught uniformly in the NBA until the early 1980s, over 30 years into the league’s existence. The use of VCRs and videotapes for referee training wouldn’t begin in earnest until a few years later. The NBA didn’t even begin using three officials in its games full-time until 1988.
As the world around it has modernized, though, so has the NBA. Today’s league officiating department relies more than ever on a well-known sports buzzword: Analytics.
Every single call made by NBA referees — and many of those not made — is graded by impartial observers, then inputted into a vast database including every official in the league. This data is used in ongoing referee training and development, and it helps in determining ref promotions and playoff assignments. Teams are even given partial access to and are allowed to make some inquiries into specific calls.
This is no mom-and-pop setup. It’s a full-fledged operation involving more than two dozen full-time staffers, from former NBA officials to outside consultants and quantitative analysts. It’s also a realm into which the public has had very little window, even as analytics and technology have become larger and larger parts of the league’s officiating department in recent years.
FiveThirtyEight spoke with more than 25 people across the NBA, from league staffers in charge of data collection and use to front-office executives, to understand exactly how this referee-grading and analytics program works.
Many an NBA fan recognizes Secaucus, New Jersey, as home to the league’s Replay Center, but that’s just one small piece of league and referee operations taking place there. Secaucus is also the NBA’s hub for its referee review and grading program, which employs a staff of over 25 people in various roles.
A key figure here is Steven Angel, one of the NBA’s longest-tenured employees at nearly 14 years. Angel was part of a consulting firm that helped redesign the league’s referee data in the early 2000s when David Stern was commissioner; he was eventually brought on full-time as part of the officiating department, now occupying the role of senior vice president of game analytics and strategy.
When Angel started working with the league, the sophistication of referee reviews and grading systems matched the limited technology that was available. Around the turn of the millennium, the NBA started assigning observers to attend games in-person in their geographical area, then return home and break the same game down again via DVR later that night. Angel, with his consulting background, noticed potential problems with training and arena biases, and the system was brought entirely in-house in 2013.
When the NBA brass decided to begin sharing referee game grades and reports with individual franchises in 2015, they quickly realized that many teams were focused on different call and play types than the league’s own referee advisers — understandable given that teams have their own interests at heart, while advisers are focused on leaguewide referee performance. Standardizing these definitions across the board became a major piece of Angel’s department, one that persists to this day under the title of “Rules Clarity Project.”
Those involved work directly with the NBA’s competition committee — which comprises a group of owners, general managers, coaches, referees and players — plus representatives from all 30 teams and the referee’s union. Angel sums up the aim of this project simply: “What actually constitutes an error? The game is one long non-call, except when the whistle is blowing,” Angel told FiveThirtyEight. “Can we agree on what constitutes a foul?”
Angel’s team also handles the realm of integrity. It looks for any and all possible indicators of bias, whether conscious or subconscious. The name “Donaghy” is rarely uttered in league circles today, but there’s an obvious desire to protect against even the suggestion of impropriety — especially given the NBA’s own stated interest in the arena of sports gambling.
“We monitor gambling lines in Las Vegas,” Angel said. “We look to see if there’s any indication of manipulation or bias. That is hopefully a big, big waste of time, but we are still diligent in that effort.”
The biggest element Angel oversees, though, is the NBA’s staff of dedicated referee reviewers.
While certain roles in the league’s officiating department are held by former referees, game reviewers are not — again, the goal here is limiting any potential for bias. Rather, Angel and his staff look for “basketball-centric individuals” who have coached or played at some level and have a strong baseline of knowledge but don’t have specific connections with NBA franchises.
In the early years of this program, applicants for game reviewer jobs were given tests on their basketball acumen. Quickly, though, the league realized those tests were asking the wrong questions.
“We believe we can teach what [reviewers] need to look at,” Angel said. “What we need to find in them is the ability to sit and focus for long periods of time.”
Reviewing NBA referees is an arduous, painstaking task. Doing the job correctly might require watching the same three-second play clip over a dozen times at numerous angles. A single game review takes between six and eight hours, per multiple members of the department. Dedication and focus are just as important as actual basketball knowledge.
The interview process today is more focused on these sorts of skills. These are prestigious roles: The NBA’s staff of game reviewers numbers 15 at most, including 10 standard reviewers, three senior reviewers and a couple of specialized roles (such as one reviewer dedicated solely to gambling lines and related integrity concepts).
Game reviewers are trained on three specific components of the job:
- How to grade plays: What constitutes an infraction vs. what doesn’t? Which types of plays need to be included on game reports?
- Which ref is responsible: Reviewers grade whether an infraction took place, but also which referee on the floor was responsible for making (or not making) that call. This involves a detailed knowledge of NBA referee mechanics for each of the three positions an official occupies on the floor (lead, slot and trail).
- Use of the Game Review System technology.
The Game Review System, or GRS, is the technological foundation of modern NBA referee grading, one designed in-house and updated several times since the mid-2000s. The goal is to present reviewers with a standardized dashboard they access via computer, with processes that can be applied as evenly as possible to every single NBA game.
Again, there’s major emphasis here on which plays are included. The league could demand reviewers break down every single dribble in each game, labeling “no infraction” each time a player successfully moved without traveling; while that might allow the NBA to claim 99.99 percent accuracy on all call types, it wouldn’t be a feasible use of reviewer time — and the resulting data would be almost useless.
Instead, reviewers get a major assist here from modern technology: Second Spectrum camera tracking, which is present in all 29 NBA arenas, is integrated into GRS. This data is used to “pre-tag” various common events in a basketball game, such as shots, passes, drives, screens and dribble initiations. Instead of manually combing through every second of game action, hemming and hawing on which plays to cover and which to leave alone, reviewers have a standardized shorthand they can lean on.
Generally speaking, reviewers can categorize — or “tag” — a play in one of four ways:
- Infraction: An infraction of NBA rules clearly took place on the play in question.
- No Infraction: An infraction of NBA rules clearly did not take place.
- Potential Infraction: The call was not clear or conclusive. Two sub-categories, “Leaning Infraction” and “Leaning No Infraction,” are included in these “judgment calls” to make eventual datasets more robust.
- Enhanced Review: A conclusive decision on a call could have been reached only using enhanced video review and could not have been reasonably seen by a referee in real time. For instance, the league won’t punish a referee for missing a travel on a play in which slow-motion video revealed that a player lifted his pivot foot milliseconds before the ball left his hand for a dribble.
Reviewers have access to a minimum of nine broadcast angles for every play, plus often several additional views — and a full suite of video enhancement options at their disposal. In addition to entering one of those four tags for each play, reviewers also use video to determine which referee on the court should have been responsible. (Tracking technology also often plays a role in this task.)
NBA
A collaborative spirit is encouraged. Reviewers are in the same physical location in Secaucus and often canvass one another or senior staffers on tough plays. (COVID-19 protocols have interrupted parts of this in-person coordination over the past two seasons.)
After six to eight hours, a reviewer will have analyzed a single NBA game and all possible infractions within it. This process will often then be repeated by a senior reviewer on the staff; hundreds of NBA games each season each get 12 to 16 hours of review.
Each call and non-call will be automatically assigned to the referees who worked that game and compared to the reviewers’ tags. (It’s important to note, again, that reviewers are evaluating infractions, not referees themselves. Their job is to focus solely on the players on the court and whether rules were broken during the course of play; to avoid bias as much as possible, resulting grades are assigned to officials later and by other league staff.) This allows for corresponding accuracy to be determined. From here, this data will become part of each official’s existing record. Leaguewide, this dataset is massive: A single season will include around 500,000 refereeing data points, per Evan Wasch, the NBA’s executive vice president of basketball strategy and analytics.
Wasch oversees Angel’s department, which includes a group of data scientists whose role involves combing through this data in excruciating detail. They’re looking for trends across individual refs and and throughout the staff, among players and teams, and even within betting patterns. Wasch’s team will then work with other key NBA officials — including Monty McCutchen, senior vice president for referee development and training, and Mark Wunderlich, vice president of referee operations — on what they find.
This data analysis will often have a direct effect on ongoing referee training. For example, as the game has undergone a spacing revolution in the last decade, officiating analysts began noticing related positioning issues.
“[As] the game became more perimeter-oriented, we found that we were seeing errors in different places on the floor,” Wasch tells FiveThirtyEight. “Which in turn led us to work with Monty and Mark Wunderlich and his staff to rethink referee mechanics to make sure referees were in the right position to pick up plays.”
Again, collaboration is key. Wasch and his staff are regularly embedded in meetings and training sessions with Wunderlich’s group. If data can identify or strengthen an important training area for a single ref, a group of refs or even every ref in the league, all the better.
And they even review their own reviewers! Data is kept on how often senior review staff is forced to overrule an improper tag from an initial review upon their second pass; Angel and his staff address any error trends that show up here. If the issues persist, a reviewer might have to find a new job.
Coming Wednesday: How the ref-grading data is used by the league and its teams.
CORRECTION (Feb. 22, 2022, 3:27 p.m.): An earlier version of this article incorrectly characterized the department within the NBA that handles game analytics. It includes a group of data scientists who sift through the ref data looking for trends.