Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display
Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. We present neural étendue expanders, offering a path to usable, wearable, and ultrathin augmented reality displays, by learning an étendue expanding element that effectively increases étendue by two orders of magnitude. Compared to existing state-of-the-art hand-engineered approaches, neural étendue expanders produce high-fidelity full-color holograms for complex scenes.
Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, Felix Heide
Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display
Nature Communications
The future of immersive augmented reality. A portable IMAX screen.
Holographic displays use optical elements that are small enough to fit into a regular pair of glasses, and they project images that are seamlessly integrated into a user’s normal field-of-view. Headsets that rely on a monitor, on the other hand, tend to be bulky since they need to accommodate a screen and the hardware necessary to operate it. This gives holographic displays the potential to become ubiquitous, transforming how we interact with our environments — everything from getting directions while driving, to monitoring a patient during surgery, to accessing plumbing instructions while doing a home repair.
Étendue, the single biggest barrier for holographic displays.
Étendue is a physical quantity associated with any optical system that upper bounds the product of field-of-view and the eyebox size. The spatial light modulators (SLM) that produce digital holograms possess a limited étendue that is orders of magnitude lower than that of analog recording devices, resulting in either a narrow field-of-view or an unusably small eyebox. Unfortunately, there is no foreseeable path towards building a high étendue electronic SLM in the near future due to fundamental issues such as pixel cross-talk, power consumption, and device size. Thus, compact passive elements that integrate with the SLM are poised to make the biggest impact on common practice in this active area of research and development.
High fidelity full-color holograms at 64× étendue expansion.
We turn towards artificial intelligence to learn an étendue expanding element, allowing us to close the gap in étendue without sacrificing other criteria such as hologram fidelity and form factor, which we validate experimentally. Here, we show captures of holograms generated with a display prototype with and without neural étendue expansion. The increase in étendue can be seen here as an increase in the field-of-view of the holographic display.
A very small custom-built piece of frosted glass.
The neural étendue expanders are physically realized as diffractive optical elements. The negative of the learned pattern is etched into a substrate which forms a stamp. The stamp is then pressed onto a resin mold that is mounted on a glass substrate. The resin itself contains the final pattern. We insert these diffractive optical elements, which are akin to frosted glass, into our display prototype.
Designed using a mix of AI tools and optical techniques.
We introduce a novel differentiable wavefront propagation model that treats the SLM and the étendue expanding element as a single shallow neural network. We teach this optical neural network to maximize hologram fidelity by training it on a dataset of natural images. Together, the differentiable physics model and neural design strategy produces an étendue expanding element that supports state of the art ultra-wide-angle holograms. Specifically, the neural étendue expanders learn to scatter light created by the spatial light modulator in a very precise way, pushing some elements of an image into frequency bands that are not easily perceived by the human eye. This improves the quality of the holographic image, making it larger and clearer and easily viewable for the user.
Departing from hand-engineered optical elements for étendue expansion.
Existing hand-engineered optical elements étendue expansion exhibit randomized scattering that is agnostic to the optical setup and the images to be displayed. As modern SLMs have limited degrees of freedom for wavefront shaping, the random modulation delivered by these scattering elements results in low-fidelity étendue expanded holograms. Here, we show comparisons between hand-engineered optical elements and neural étendue expanders for étendue expansion using our display prototype. Neural étendue expanders improves reconstruction fidelity by over 10 dB PSNR across all color channels.
Increasing the field-of-view without sacrificing the eyebox size and the image quality.
The field-of-view of a holographic display can be increased by using a diverging lens, also known as a quadratic phase profile. However, placing a lens into the system does not change the total étendue of the display because the eyebox size will shrink proportionally. In contrast, we validate that neural étendue expanders truly increase the total étendue of the holographic display. That is, we validate that neural étendue expanders support high-fidelity large field-of-view holograms without reducing the eyebox size. Here, we simulate the effect of varying user eye pupil positions and sizes. We observe that changing the eye pupil setting causes image content to dissapear when using a quadratic phase profile but not when using neural étendue expanders.
Related Publications
[1] Schiffers, Florian and Chakravarthula, Praneeth and Matsuda, Nathan and Kuo, Grace and Tseng, Ethan and Lanman, Douglas and Heide, Felix and Cossairt, Oliver. Stochastic Light Field Holography. International Conference on Computational Photography (ICCP), 2023
[2] Chakravarthula, Praneeth and Baek, Seung-Hwan and Schiffers, Florian and Tseng, Ethan and Kuo, Grace and Maimone, Andrew and Matsuda, Nathan and Cossairt, Oliver and Lanman, Douglas and Heide, Felix. Pupil-aware Holography. ACM Transactions on Graphics (TOG), 41(6):212, 2022