Learning Efficient and Effective Representations for Event-Based Cameras
Event-based cameras are vision sensors that attempt to emulate the functioning of biological retinas. An array of independent pixels mimics the photosensitive membrane of the retina by generating an output signal, i.e., an event, whenever the local brightness changes of a certain threshold. Like in the biological vision system, information is produced asynchronously and only when needed, resulting in a very efficient device with many advantages over traditional ones. These include very high temporal resolution and dynamic range, as well as a v¬ery low power consumption.
Driven by these benefits, many computer vision researchers have begun adapting traditional algorithms to this new event-based paradigm, first with hand-engineered algorithms and, more recently, machine learning ones. While the first demands each algorithm to be individually re-designed from the ground up, the second entails the more general effort of adapting learning frameworks to efficiently and effectively interface with events. These frameworks often rely on their ability to learn extracting meaningful features directly from experience, making even complex visual tasks easy to solve. How such systems perform on a particular task is often linked to how rich, informative, and general their internal representations are. However, as every single event carries very little visual information, designing novel mechanisms to extract effective representations while retaining the benefits of event-based cameras is challenging and remains an open research question to date.
In this research, we focus on bridging the gap between deep learning and event-based vision. We develop a framework that integrates event-driven processing within neural network layers without loss of performance. We propose a recurrent neural layer that learns to extract a task-specific event representation incrementally, driven by incoming events. And, finally, we show that domain adaptation techniques can be used to achieve good performance on real event-based data even when training is performed on simulated samples.
Back to Alumni