Waymo is using AI to simulate autonomous vehicle camera data Posted on : May 21 - 2020

Waymo says it’s beginning to leverage AI to generate camera images for simulation by using sensor data collected by its self-driving vehicles. A recent paper coauthored by company researchers including principal scientist Dragomir Anguelov describes the technique, SurfelGAN, which uses texture-mapped surface elements to reconstruct scenes and camera viewpoints for positions and orientations.

Autonomous vehicle companies like Waymo use simulation environments to train, test, and validate their systems before those systems are deployed to real-world cars. There are countless ways to design simulators, including simulating mid-level object representations, but basic simulators omit cues critical for scene understanding, like pedestrian gestures and blinking lights. As for more complex simulators like Waymo’s CarCraft, they’re computationally demanding, because they attempt to model materials highly accurately to ensure sensors like lidars and radars behave realistically.

In SurfelGAN, Waymo proposes a simpler, data-driven approach for simulating sensor data. Drawing on feeds from real-world lidar sensors and cameras, the AI creates and preserves rich information about the 3D geometry, semantics, and appearance of all objects within the scene. Given the reconstruction, SurfelGAN renders the simulated scene from various distances and viewing angles.

“We’ve developed a new approach that allows us to generate realistic camera images for simulation directly using sensor data collected by a self-driving vehicle,” a Waymo spokesperson told VentureBeat via email. “In simulation, when a trajectory of a self-driving car and other agents (e.g. other cars, cyclists, and pedestrians) changes, the system generates realistic visual sensor data that helps us model the scene in the updated environment … Parts of the system are in production.”

SurfelGAN

SurfelGAN makes use of what’s called a texture-enhanced surfel map representation, a compact, easy-to-construct scene representation that preserves sensor information while retaining reasonable computational efficiency. Surfels — an abbreviated term for “surface element” — represent objects with discs holding lighting information. Waymo’s approach takes voxels (units of graphic information defining points in 3D space) captured by lidar scans and converts them into surfel discs with colors estimated from camera data, after which the surfels are post-processed to address variations in lighting and pose.

To handle dynamic objects like vehicles, SurfelGAN also employs annotations from the Waymo Open Dataset, Waymo’s open source corpus of self-driving vehicle sensor logs. Data from lidar scans of objects of interest are accumulated so that in simulation, Waymo can generate reconstructions of cars and pedestrians that can be placed in any location, albeit with imperfect geometry and texturing. View More