Foveating Cameras

FoveaCam: A MEMS Mirror Enabled Foveating Camera SaccadeCam: Adaptive Visual Attention for Monocular Depth Sensing
Brevin Tilmon, Eakta Jain, Silvia Ferrari, Sanjeev J. Koppal Brevin Tilmon and Sanjeev J. Koppal
ICCP 2020 / PAMI 2021 arXiv 2021
Most cameras today photograph their entire visual field. In contrast, decades of active vision research have proposed foveating camera designs, which allow for selective scene viewing. However, active vision’s impact is limited by slow options for mechanical camera movement. We propose a new design, called FoveaCam, and which works by capturing reflections off a tiny, fast moving mirror. FoveaCams can obtain high resolution imagery on multiple regions of interest, even if these are at different depths and viewing directions. We first discuss our prototype and optical calibration strategies. We then outline a control algorithm for the mirror to track target pairs. Finally, we demonstrate a practical application of the full system to enable eye tracking at a distance for frontal faces. Most monocular depth sensing methods use conventionally captured images that are created without considering scene content. In contrast, animal eyes have fast mechanical motions, called saccades, that control how the scene is imaged by the fovea, where resolution is highest. In this paper, we present the SaccadeCam framework for adaptively distributing resolution onto regions of interest in the scene. Our algorithm for adaptive resolution is a self-supervised network and we demonstrate results for end-to-end learning for monocular depth estimation. We also show preliminary results with a real SaccadeCam hardware prototype.

PAPERS:

ICCP 2020 | PAMI 2021

PAPERS:

arXiv 2021