Imagine you’re at home plate, your bat lifted behind your shoulder. You eye the pitcher as he sets up, carefully paying attention to not just the sights that play into the timing of your swing, but the sounds, too.
Researchers at Villanova believe those audiovisual cues will set apart the next generation of players. And this spring, the Villanova Wildcat baseball team will begin using a new virtual reality training system designed by Dr. Mark Jupina, an assistant professor of electrical and computer engineering at the university. The VR system matches batters and catchers against some of the best pitchers in Major League Baseball, but it’s intended use stretches far beyond just a virtual batting machine.
The system, dubbed PITCHvr, can pull in data from the MLB pitch database. (Its name was inspired by the PITCHf/x system that was used through 2016, and has since been replaced by TrackMan.) Jupina has used this data to recreate the motions of a pitched ball, such as its path, velocity, orientation, and spin. From this perspective, PITCHvr is not unlike other VR pitching platforms that have come in the past, such as Diamond FX, a virtual reality player performance and scouting tool that uses recreated pitches and sports vision to give baseball players extra reps.
What sets PITCHvr apart, however, is the addition of audio tags that help to not only train eyes but also ears as batters, catchers, and umpires prepare for pitches in the real world. Jupina’s algorithm generates a unique audio signature for each virtual pitch. When this audio is played alongside the pitch, the sound can assist users’ eyes in tracking the motion of the baseball.
As the Wildcats are preparing to use this system during their upcoming season, MLB teams are also becoming interested in the technology. At least one team so far has expressed a desire to use it for 2019 spring training, although Villanova declined to provide the name of the team since negotiations are ongoing.
Jupina, who played baseball through high school and coached for 11 years, leaned heavily for PITCHvr’s development on Wildcats head baseball coach Kevin Mulvey, a former standout pitcher for Villanova who reached the major leagues with the New York Mets, Minnesota Twins, and Arizona Diamondbacks.
While Jupina had the science and engineering background to build the complex algorithms that power PITCHvr, Mulvey offered the real-world experience of professional baseball and insight into how the system would best benefit his players.
To date the system has been primarily tested in Villanova’s room-sized virtual reality CAVE space, an 18-foot-wide, 10-foot-deep, and 7.5-foot high enclosure that provides immersive 3D experiences. But the system has also been adapted for the HTC Vive headset (the same headset that launched DiamondFX), which is the version that’ll be used in-season by the Wildcats and possibly by professional teams.
Whether stepping into the CAVE or donning a Vive headset, users are met with some of MLB’s top pitchers, including the Astros’ Justin Verlander, Red Sox closer Craig Kimbrel, the Yankees’ Aroldis Chapman, and the Indians’ Corey Kluber. Jupina even recreated Astros’ Lance McCullers’s nasty knuckle-curveball, which helped to deliver Houston an American League championship and World Series win in 2017.
“I recreated the same exact pitch in this virtual enforcement and they matched up well,” said Jupina. “That gave me the confidence that this looked accurate and realistic and began working with [Villanova head coach] Kevin to verify things.”
With the infrastructure now in place, Jupina is eyeing a range of other use cases for his system. In the future, pitches generated by PITCHvr could be made even faster and nastier than pitches thrown to date, which would help to further hone users’ audio and visual tracking instincts and possibly translate to more hits from the plate.
Jupina also is planning PITCHvr adaptations for specific positions, such as for catchers, umpires, and batters. He projects these updates will be ready for use over the next half-year. From there, the school could be able to explore potential licensing opportunities as well.
“I can see applications of a catcher or an umpire and have talked to the MLB office in charge of umpire development,” he said. “We could come up with an app where a person would try to catch the ball and the system would detect when their hands closed on the ball.”
The next step (and something Villanova researchers and students are already working on) is to use additional sensors and imaging technology to provide analytics on the bat’s motion and swing, enabling players to obtain metrics such as launch angle, velocity of the batted ball and distance traveled.
“Current trackers aren’t really sufficient … you need to use other types of sensors,” said Jupina.
Future iterations of the technology might integrate neurofeedback to read a users’ focus and stress levels. Jupina has held discussions with Narbis, a company that uses EEG sensors to measure brain waves, and has worked with the Villanova psychology department to use measurements of muscle tension to get an idea of someone’s focus or concentration level.
A tightened jaw or wrinkled forehead might indicate a batter on edge, for example. Perhaps one day, that feedback might pause the system, forcing users to relax before continuing, and conditioning them over time to step up to the plate with a clearer, more-focused mind. A similar neurotechnology has been used by the NBA’s Portland Trail Blazers.
According to Jupina, one MLB team has expressed interest in using that part of the technology along with inputted crowd noise and eye-tracking technology in its scouting process. While most professional-grade players might perform well at the plate or in batting cages during practice, their abilities might change when they’re in a high-pressure gameday situation.
“You throw them some practice pitching they’re going to crush that ball, so they’d rather see how they handle actual pitches and how well they’re tracking that ball with their eyes and what their state of mind is,” he said.
Further into the future, Jupina believes the technology could also be adapted into other sports that have fast-moving objects that have to be caught or hit, including tennis, hockey, and lacrosse.