Thin On-Sensor Nanophotonic Array Cameras

  • Praneeth Chakravarthula

  • Jipeng Sun

  • Xiao Li

  • Chenyang Lei

  • Gene Chou

  • Mario Bijelic

  • Johannes Froech

  • Arka Majumdar

  • Felix Heide

SIGGRAPH Asia 2023

We propose a thin nanophotonic imager that employs a learned array of metalenses to capture a scene in-the-wild. We devise an array of lenses mounted directly on top of the sensor cover glass (top center). Each lens in our array (illustrated on the left) is a flat (700 nm thick) metasurface area of nano-antennas which we design to focus light across the visible spectrum. The peripheral elements capture images at slanted field angles, making it possible to capture a wide field of view of 100◦, more than twice as large as the most similar design from Tseng et al. [2021] shown on the right side. The proposed computational camera is capable of recovering images outside of lab conditions under broadband illumination. We illustrate here the matching physical image size as measured on the sensor, with the gray area on right illustrating same sensor size.

Thin On-Sensor Nanophotonic Array Cameras

Praneeth Chakravarthula, Jipeng Sun, Xiao Li, Chenyang Lei, Gene Chou,
Mario Bijelic, Johannes Froech, Arka Majumdar, Felix Heide

Thin On-Sensor Nanophotonic Array Cameras

SIGGRAPH Asia 2023

Replacing Compound Optics with Metalens Array

Today’s commodity camera systems rely on compound optics to map light originating from the scene to positions on the sensor where it gets recorded as an image. To record images without optical aberrations, i.e., deviations from Gauss’ linear model of optics, typical lens systems introduce increasingly complex stacks of optical elements which are responsible for the height of existing commodity cameras. In this work, we investigate flat nanophotonic computational cameras as an alternative that employs an array of skewed lenslets and a learned reconstruction approach.

Lens as a Combination of Prisms

Lens as a Combination of Prisms. A lens refracts an incoming set of parallel rays and bends them towards a focal point. With a piece-wise linear approximation, a lens can be thought of as a stack of infinitesimally small prisms. We depart from a single large lens with long back focal length, re-organize the prisms, and represent a large lens as a combination of appropriately chosen wedges and lenses with short focal length for high-quality camera imaging.

Ultra-thin and Compact Metalens Array

Our on-sensor imaging optic, which is smaller than a penny, consists of an array of metalenses as shown on the side. Each metalens is made of optimized nano-antennas of size 350 nm, significantly smaller than the wavelength of visible light. Our optimized metalens nano-antenna structures scatter light from the entire visible spectrum.


Probabilistic Image Reconstruction

We propose a deconvolution method that relies on the physics-based forward model (PSFs illustrated on the left) along with a learned probabilistic prior (diffusion model on the bottom right). The proposed method is an unrolled optimization method that alternates between inverse filtering steps using the calibrated PSFs, diffusion steps that sample from the natural image manifold with conditioning on the current iteration, and a merging step that combines the image estimates from all sub-apertures (green). The unrolled optimization method is trained in an end-to-end fashion with paired training samples captured with a co-axial reference camera (bottom and left).

Paired Data Acquisition

In our capture setup, we employ a plate beam splitter, which splits world light into two optical paths by 70% transmission and 30% reflection such that the setup can simultaneously capture real-world scenes with one camera in the transmission path that employs the designed metalens array and another camera in the reflection path that employs a conventional off-the-shelf lens (GT camera). The two cameras are aligned and calibrated to map one to the other captures, see text for details.

Experimental Evaluation on Broadband Scenes

Experimental evaluation of the proposed thin nanophotonic camera for broadband indoor & outdoor scenes. The proposed nanophotonic array optic with the probabilistic deconvolution method reconstructs the underlying latent image robustly in broadband environments.


Experimental indoor scene with a person holding a colorful storybook in a book store.



Experimental broadband outdoor scene with two pedestrians walking besides a bench close to a street.



Experimental scene in broadband outdoor lighting outside a coffee shop.



Experimental dynamic indoor scene with a group of people discussing in the hallway.



Experimental evaluation indoor scene with person holding a puppet in the toy store.



Optical Performance Analysis

We analyze the optical performance of the proposed nanophotonic array lens via its theoretical modulation transfer function (MTF), i.e., the ability of the array lens to transfer contrast at a given spatial frequency (resolution) from the object to the imaging sensor. The analysis validates that the proposed metalens exhibits a improved MTF under different angle of incidence (AoI) compared to the existing designs, approaching the efficiency of diffraction limit for normal incidence of light and achieving higher MTF values for larger incidence angles.


MTF for Different Angles of Incidence

MTF for different angles of incidence. The calculated MTF for various angle of incidence (AoI) for array MO with varying wedge phase profile. Thick solid lines correspond to the diffraction limit. Dashed lines correspond to calculated MTF curves for Tseng et al. Full thin lines correspond to the MO, presented in this work, with wedge phase as specified in the title. The x-axes represent line pairs per mm, within the specific resolution range for the used sensor, with a pixel size of 5.7 $\mu$m.


Siemens Star Calibration Measurement

Sensor measurement of the Siemens Star calibration pattern. Even without reconstruction, our metalens design significantly reduces scattering compared to the previous state-of-the-art design proposed by Tseng et al. across all wavelengths, thereby allowing for broadband imaging in-the-wild. When combined with probablistic deconvolution method, the proposed nanophotonic array camera robustly recovers the latent image.

Related Publications

[1] Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar, and Felix Heide. 2021. Neural Nano-Optics for High-quality Thin Lens Imaging . Nature communications 12.1 (2021): 6493.