[Nature 2021] Towards Real-time Photorealistic 3D Holography with Deep Neural Networks

Nature 2021
Authors: Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, Wojciech Matusik
http://cgh.csail.mit.edu

The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality (AR/VR), human-computer interaction, education, and training. Computer-generated holography (CGH) enables high spatio-angular resolution 3D projection via numerical simulation of diffraction and interference. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography far from practical. Here, we demonstrate the first deep learning-based CGH pipeline capable of synthesizing a photorealistic color 3D hologram from a single RGB-Depth (RGB-D) image in real time. Our convolutional neural network (CNN) is extremely memory-efficient (below 620 KB) and runs at 60 Hz for 1920×1080 pixels resolution on a single consumer-grade graphics processing unit (GPU). Leveraging low-power on-device artificial intelligence (AI) acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 Hz) and edge (Google Edge TPU at 2 Hz) devices, promising real-time performance in future generation AR/VR mobile headsets. We enable this pipeline by introducing the first large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-D images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking high-resolution 3D holograms. Our learning-based approach and the first Fresnel hologram dataset will help unlock the full potential of holography and enable new applications in metasurface design, optical and acoustic tweezer-based microscopic manipulation, holographic microscopy, and single-exposure volumetric 3D printing.

Видео [Nature 2021] Towards Real-time Photorealistic 3D Holography with Deep Neural Networks канала MIT Computational Fabrication Group