Seeing Through Obstructions with Diffractive Cloaking

Learning Obstruction-aware Diffractive Optical Elements

We learn a DOE placed in front of a camera lens jointly with an optics-aware reconstruction network, to enable seeing through thin occluders close to the camera. To this end, we introduce a differentiable obstruction-aware image formation model that simulates the sensor capture of the proposed optical system.

The proposed camera (top left) modulates the latent scene at a long distance and the obstruction close to the observer (e.g., dirt on the wind shield) with the depth-dependent point-spread function of the learned optical element. The resulting sensor capture (top right) is fed into a feature deconvolution network (bottom) along with the depth-dependent point spread function (center). The entire computational imaging system, including optical and computational phases, is jointly learned (bottom left), resulting in a DOE that scatters light pertaining to nearby objects using a ring-shaped PSF, while preserving angular resolution for objects at optical infinity.

Learned Depth-dependent PSFs

We report the near-scene and far-scene PSFs and MTFs corresponding to the proposed learned optical design, and to other heuristic baseline optical designs that do not increase the aperture.

The proposed design achieves high MTF values at long distances corresponding to background scene depths, and disperses the radiance from nearby objects over the entire sensor. Specifically, the PSF at optical infinity resembles that of a conventional lens with a small focal spot, while the PSF for nearby distances resembles a large ring, with a radius prescribed by the maximum diffraction angle supported by the phase plate. Note that this PSF is obtained using our end-to-end learning approach, without using any design heuristics.

Experimental Prototype System

We implement the proposed method experimentally with the prototype system shown below. To this end, we fabricate the DOE designed for the front-facing automotive imaging task in a 16-level photolithography process on a fused silica wafer. To place the DOE on the aperture plane, instead of cutting an existing lens barrel, we relay the phase DOE to the aperture plane using a 4F system.

We measure the PSF of the entire optical system with (top row) and without (bottom row) the DOE. The measurements validate that the designed phase plate produces a ring-shaped PSF for close occluders, while resulting in a small spot size for far scene content. Manufacturing inaccuracies result in a 0th-order component (top left) at close distances, for which we compensate by providing the measured PSF to the reconstruction network.

Experimental Results

The proposed approach achieves a 1-5 dB PSNR gain over existing approaches when reconstructing occluded regions, by optically cloaking the obstruction while relying on the complementary reconstruction network to remove image aberrations, thus effectively allowing the occluded background to be “seen”.
The experimental captures below further validate that the proposed method is able to remove diverse obstructions and recover the hidden details, while existing image inpainting methods can merely attempt to “hallucinate” the missing content.