AC34 LiveLine, harder than football & open sourced

Never mind the final Super Bowl score (and the unfortunate headlines), did you appreciate those slick digital field overlays like the yellow 1st & ten line? Frankly I'm still impressed with this technology even though it's not magic. HowStuffWorks explains pretty well the sensors, geometry, and video processing required to make that virtual line look like it's painted on the field. All it takes is precise knowledge of where the camera and line markers are in a 3D model of the stadium, plus the use of the turf color as a reverse mask so that the players -- who must never wear the same shade of green -- don't get overlaid. But once you understand that, the new LiveLine info inserted into America's Cup video seems practically impossible...

Consider the leap in sensor difficulty between tracking the location, pan, and tilt of a camera hung from a helicopter as opposed to those set on tripods around a stadium. Consider too how much larger the AC34 playing field is, which means that even small errors in camera or race/mark boat tracking can result in wacky overlays. And finally consider the communications and processing horsepower needed to make LiveLine live. No wonder than the seemingly endless gear racks and networking ganglia I spotted in San Diego around the complex of containers at the heart of what's known as ACTV...

I've been intrigued with LiveLine since I first heard it about last June, and learned more about the underlying high-precision telemetry when I got to San Diego. So I especially enjoyed lurking around the LiveLine booth (container) during an afternoon of the racing.
   Riding herd over all the technology are some talented humans, and the photo below shows the LiveLine chief operator at work. The gray switch box near his left hand lets him manage audio coms with the helicopter pilot and other nodes in the system, including the adjacent broadcast booth where LiveLine is mixed with the many other video feeds. Meanwhile the nearest PC screen shows the software he uses to select specific LiveLine overlays while the monitor above it shows the results and the one top left shows the mix going out on live ACTV. There was only minor grumbling about how the director should use more LiveLine!...

Then there's the mask man. The color of harbor water is not nearly as consistent as, say, AstroTurf, especially given the changing light angles from a helicopter, so his job is to constantly monitor the shades used to create the overlay mask. If you click the photo below larger you'll not only see how perfectly all the boats and land features are being excluded from the overlays, but also a window of "LiveLine Chatter Feed".
  AC34 seems to have numerous communications levels, including several text streams, both to accommodate those with limited screens -- like the racers, and the umpires on Jet skis -- and to serve as backup if the big pipes fail. In fact, the reason Stan Honey's laptop below was running AirView Spectrum Analyzer was because on this particular afternoon some unknown RF source was intermittently interfering with the wireless video feed from the LiveLine helicopter. Note that some of the same experts who live simulated previous Cup races to make it easier to understand on TV are working at the back of this same container, so LiveLine is even backed up graphically...

ACRM (remember how the umpiring and course management use the same underlying telemetry) and ACTV test all systems every race day morning, hoping to avoid such surprises. And the LiveLine team wasn't happy about the distorted copter feed, so Honey did a lot of troubleshooting, including calls to the big Navy base across the harbor and to a cruise ship that had just parked in a nearby berth. I felt weirdly lucky to witness how well he and the team worked the problem, but in fact it went away by itself. I wouldn't be surprised, though, if a pro-level directional antenna and spectrum analyzer gets added to the gear kit...

I was also pleased to meet Ken Milnes, the gray haired gent sitting with the LiveLine crew. He and Honey founded Sportvision and developed that yellow 1st & Ten Down Line together (and many other sports broadcast enhancements),  and he now serves AC34 as Live Graphics Project Manager. It's Milnes who developed the Streaming Data Interface Specification that's intended to let developers access AC34 telemetry, chat streams and more in real time.
   Yes, you read that right. Developers are free to write to write apps that might, say, put LiveLine style graphics onto the smart phones of AC spectators lining Newport or San Francisco harbor, or even overlay it on the phone's geo-located camera display. Logged data from San Diego and prior races is already available, and it's interesting to note that there's also a Specification for streaming the data to the race boats, presumably the 72-foot version. There's even a sample ACViewer app which can interface with a AC-served sample San Diego data that works right now (I tried it). Who's going to cook up some AC34 apps and what might they be besides enhanced viewing apps?
   The more you look into what AC34 is doing, the harder it is to deny the claim that sailboat racing is being reinvented before our eyes, with the AC World Series in particular perhaps becoming an enduring professional circuit. I'll close with a peek into the main ACTV broadcast booth (container) while Oracle Racing CEO Russell Coutts makes the case to some San Diego visitors.