Event camera-based 3D capture for mixed reality

VIT team presents paper at CVPR 2023.

A paper by researchers from the Vision and Imaging Technologies (VIT) department of Fraunhofer Heinrich Hertz Institute (HHI) has been accepted at the CVPR 2023 Workshop for Event-based Vision. The world-renowned workshop covers novel event-based cameras and algorithms that are significantly more energy-efficient, light-sensitive, and dynamic than conventional cameras. The paper, “X-maps: Direct Depth Lookup for Event-based Structured Light Systems” examines the use of event-based cameras to minimize latency in mixed reality environments and improve the user experience.

Unlike conventional cameras, event-based cameras do not capture complete images that then have to be processed and stored in a time- and energy-consuming manner. Instead, they generate continuous pixel streams with a temporal resolution of microseconds. This fundamentally different approach to signal generation has several advantages: event-based cameras have a high dynamic range and very low latencies. Because of these characteristics, they unlock opportunities that are unthinkable with conventional cameras (which have been the main sensor technology for the last 60 years).

Due to their low latency, these new sensors are particularly interesting for use in mixed reality technologies (XR), where even minimal delays in image processing can have a strong negative impact on the user experience.

In their paper, Fraunhofer HHI researchers investigate the combination of an event-based camera with a beamer to achieve real-time depth estimation for Spatial Augmented Reality (SAR) applications. SAR is also referred to as projection-based augmented reality and describes the superposition of virtual content on the physical environment by projection. This can create interactive, and immersive experiences in various application areas such as industry, culture, or healthcare. Interactive feedback and high frame rates are essential in this context.

The research team measures the 3D structure of the projected surface with the help of the event camera, adjusting the projection automatically. This process happens so quickly that the human eye can only perceive a coherent image and not a line-by-line appearance of the image. This is why adjusting the projection even in dynamic scenes with mobile elements is possible. Such a technique could also be used outside of XR when using a projector on a curved projection surface, for example, to adjust the projection to the surface in real-time. In addition, the paper presents new techniques for rapidly processing detected objects and events and for calibrating the nonlinear behavior.

Wissenschaftliche Ansprechpartner:

Martina Müller
Press Officer
Phone +49 30 31002 242
Email martina.mueller@hhi.fraunhofer.de

Dr.-Ing. Anna Hilsmann
Technical Contact
Phone +49 30 31002-569
Email anna.hilsmann@hhi.fraunhofer.de

Originalpublikation:

https://tub-rip.github.io/eventvision2023/papers/2023CVPRW_X-Maps_Direct_Depth_L…

Weitere Informationen:

https://tub-rip.github.io/eventvision2023/

http://www.hhi.fraunhofer.de