ST ALBANS, England — On a field 20 miles north of London, three people were camped on the edge of a field wearing USA Ultimate hoodies, notebooks open in front of them. They were the coaches of the U.S.’s under-23 women’s team, and they were scouting two of their biggest Ultimate Frisbee rivals, Canada and Colombia, who were about to play in a group-stage match of the 2015 world championships. The coaches barely even had any data on their own team — but there they were, scrounging for some on their future opponents.
Head coach Mike Whitaker and assistant coaches Carolyn Matthews and Lauren Boyle of the U.S. women’s team.
Carl Bialik
Ultimate should be fertile ground for analytics. The mostly amateur sport first blossomed at universities and remains popular with engineers, computer scientists, mathematicians and teachers — curious, creative nerds eager to break down the sport and share what they learn. Its profile is growing, too. This summer, the International Olympic Committee made the sport eligible to be included in a future Summer Olympics.
When I attended the under-23 world tournament in England this summer, I saw hundreds of the sport’s future stars coached by some of its brightest minds, but I also saw a sport missing something vital: detailed data.
It’s easy to take sports data for granted in an age when cameras track and quantify the movement of players and balls in baseball, basketball, tennis and soccer. The biggest challenges for analysts in those sports is how to wrangle and make sense of all that data and to get fans to look past traditional box-score numbers.
But in ultimate, there are hardly any traditional box-score numbers. Other sports have digitized stat-keeping even at the college or high-school level. But for ultimate, even at a relatively organized and well-run event like the under-23 worlds, the sport’s best young players checked opponents’ scores on schedules filled in by hand. Coaches — including my FiveThirtyEight colleague Jody Avirgan, an assistant coach for the U.S. men’s team — carried clipboards to log who played each point, with paper flapping in the wind and ink blurring in the rain. Players got a glimpse of what wealth can bring to a sport every time they walked past one of Watford FC’s brand-new 500,000 pound ($750,000) fields, but rope fences made clear that the Premier League team’s training ground was off-limits — as were stats as advanced and sophisticated as the EPL’s.
At best, ultimate box scores — such as those posted on the under-23 worlds website — contain just goals, assists and Ds (discs knocked down or intercepted). “That is Stone Age material to work with,” said Sean Childers, an ultimate player and co-author of a study on ultimate presented last year at the MIT Sloan Sports Analytics Conference, in an email. “Imagine a baseball or basketball box score from 50 years ago, but worse.”
Ultimate coaches dream of stats corresponding to some of their favorites from other sports. Several wished hockey assists — the pass that leads to the pass for the score — were tracked. Bob Krier, head coach of the U.S. men’s under-23 team, wants to see a shooting percentage for the most difficult passes into the end zone. Others want stats on “pulls,” ultimate’s version of kickoffs: Coaches suspect pulls matter a lot in helping a team set up its defense, both for how long they hang in the air and for where they land.
A catch-all metric for player value such as wins above replacement would be nice, too. But Martin Aguilera, who coached the U.S. mixed team at the under-23 championships this year, said, “We’re so far away from that.”
Many coaches said they look to basketball for stats they want to see for ultimate. On the surface, ultimate has more in common with football (passing toward a score in an end zone), soccer (a field sport with fluid positions and no play clock) and tennis (starting a point on offense is like serving, and scoring on a defensive point is called breaking). But ultimate has similar defensive principles to basketball, with players switching suddenly from offense to defense and both teams resetting after each score.
Plus, basketball has lots of cool data. Ultimate nerds speak with envy and awe about SportVU, the system of cameras that ring NBA arenas and produce data about where the players and ball are at every moment of each game. And they cite the shooting charts of FiveThirtyEight’s Kirk Goldsberry as models for charts they’d love to see, ones that would map success rates for players’ shots at the end zone by field position.
Other sports are also seeking better data than their traditional, limited box scores provide. In volleyball, “the official stat sheet is basically useless,” said Todd Dagenais, coach of the University of Central Florida women’s team. He’s seeking better stats to help his team but says there’s a dividend for spectators, too: A smarter sport is more fun to watch. “When an offense is run well, fans love that, which causes the defense to have to make more spectacular moves and more spectacular plays, which is also very entertaining,” he said.
Ultimate’s stats are stuck in the Stone Age in part because it takes a lot of work to get not a lot of insight. To improve on the kind of time-consuming, manual stat-keeping process that some coaches at the world championships were using, ultimate players developed an app to track players moving around the field. The Ultiapps Stat Tracker can generate heat maps showing the best scoring spots. Childers and a fellow researcher used data from the app to figure out where those spots are and which players were best at getting the disc there. What they found mostly reinforced basic tenets of the sport, like the importance of keeping the disc in the middle of the field. The heat map above, which is from the paper by Childers and Jeremy Weiss, shows a team’s likelihood of scoring from different points on the field. As a team moves closer and closer to the end zone (at the top of the chart), its chances of scoring increase (the higher the number, the better). The large dip in the 40-percent zone — shown as 0.4 — suggests that a team is just as likely to score from about 50 yards outside the end zone (marked as 20 on the heat map) in the middle of the field as they are from 35 but stuck on the sideline.
But data collected at one level of the sport with, say, little wind may not translate into a different level in windy conditions. Partly because of limitations like that one, teams mostly have stopped using the app to collect data.
“Teams liked our analysis but found collecting and inputting the data was too onerous to justify the time investment,” Childers said.
Part of would-be ultimate analysts’ challenge is that top ultimate players don’t play that many meaningful points in a season. Players might play during only eight or 10 points of a game because top teams are deep, usually with more than twice the number of players on the sideline as are on the field at any time. And the roster is rarely the same from tournament to tournament.
Even if everyone could agree on which new stats are needed in a sport like ultimate, a tough question remains: Whose job should it be to collect the stats? Tournaments are mostly run by volunteers focused on tasks such as ensuring players find the right field, have enough water and uphold the sport’s unique spirit of the game during play. That leaves coaches to keep any extra stats they’d want for analysis. But they’re also busy doing lots of other things during tournaments. It’s often easier to collect advanced stats during tryouts or practices instead.
For the under-23 tournament, U.K. mixed coaches had to choose 26 players from 93 who showed up at trials. They divided them into six groups and filmed them, rating them in 24 categories. None was scores, assists or Ds. The categories were more subtle: essential but hard-to-measure ultimate and interpersonal skills. One, for example, was “nicehead,” which gauged how well someone played with others. “What we didn’t want to do is pick very skilled players who couldn’t interact with other human beings,” coach Megan Hurst said. She and her fellow coach Felix Shardlow entered all the stats into a big spreadsheet and looked for players whose low ratings came in categories they could easily improve, like catching.
Riley Erickson records video of future opponents for the U.S. mixed team.
Carl Bialik
Aguilera thinks that more ultimate should be filmed and that more film should be watched. He filmed games at the under-23 worlds from atop a ladder he’d bought for 30 pounds ($45) just before the tournament. Many top college basketball players have seen hundreds of games by the time they get to campus. Incoming college ultimate players might have watched fewer than 20 ultimate games, Aguilera said.
Absent data, coaches have to rely on scouting to get ahead. Film analysis has become a hallmark of the best college and club programs in the country. And it was on display at the tournament in England, too.
Take, for example, the women’s final between the U.S. and Japan. Mike Whitaker, the head coach of the U.S. team who’d been scouting that Canada-Colombia game with his assistants near the start of the tournament, said that Japan used “advanced scouting more than any other team at the worlds.” The Japanese team brought personnel dedicated to the practice, which played a big role in the final’s outcome. He noticed Japan made adjustments to its defense after its group-stage game against the Americans (the U.S. won 17-13) and scouting other U.S. games.
Eri Hirai, Japan’s head coach, said the team tracked which players on other tournament teams threw the most long passes and which ones ran the most. Harai said this kind of scouting is standard practice in Japan. “It’s very important because we knew nothing about other teams before the tournament,” she said in an email interview conducted through a translator. By the end of the tournament, the Japanese team knew enough about the Americans to win the final in a big upset, 17-15. It was the only game any U.S. team lost in the tournament.
Read more: A Plea For More Frisbee Data From A U.S. Ultimate Coach
