Simulation and Verification for Vision in the Loop

We aim to rigorously characterize the sensitivity of vision-in-the-loop driving controllers in increasingly complex visual tasks. While rooftop lidar provides a spectacular amount of high-rate geometric data about environment, there are a number of tasks in an autonomous driving system where camera-based vision will inevitably play a dominant role: dealing with lane markings and road signs, dealing with water/snow and other inclement weather conditions that can confuse a lidar, and even dealing with construction (orange cones), police officers, and pedestrians/animals. Furthermore, vision sensors are often fused with depth returns from a laser and other sensors as a part of the vehicle and obstacle estimation algorithms.

As an initial study, we will investigate the performance of a simple perception algorithm for lane detection and a simple controller for lane following, given visual scenes which capture some of the diversity of urban driving conditions here in Boston including complex on-road traffic markings at intersections and worn visual features.As the project progresses, we will attempt to simulate more and more of the visual world -- up to and including difficult volumetric effects such as fog or snow and dynamic obstacles such as pedestrians and other vehicles.