Collaborative On-Sensor Array Cameras

On-Sensor Video-Rate Broadband Metalens Camera

Prior work in metalens broadband imaging has predominantly focused on improving individual metalenses, yet single‐element diffractive optics remain fundamentally limited by chromatic dispersion. In this work, we optimized a 100-million nano-pillar metalens array collaboratively across the whole visible spectrum. This makes an on-sensor, real-time, high-fidelity, broadband metalens camera possible.

Collaborative Broadband Lens Array

We densely sampled the incident spectrum from 400 nm to 700 nm at 0.5 nm intervals to compute the point-spread functions (PSFs) of each sub-lens in our array. We devised an end-to-end design method to jointly optimize the meta-optic phase profiles and the downstream reconstruction algorithm. The resulting sub-lens PSFs exhibit complementary, wavelength-dependent aberrations that mutually compensate: when combined, these elements produce a composite PSF whose footprint remains small across the entire visible band.

End-to-End Collaborative Imaging

We optimized the metalens array using collaborative through a differentiable spatially-varying image formation model and a reconstruction algorithm. The forward model computes the PSFs with a neural metasurface proxy spanning the complete visible spectrum at wide incident angles, with computations distributed across multiple GPUs. After two-step alignment to compensate for parallax, we reconstruct the full visible spectrum at the resolution of the input sub-images, with a multi-image inverse filter method collaboratively combining wavelength spectra across the six collaborative measurements. The whole pipeline is fully differentiable, thereby enabling end-to-end optimization of the collaborative nanophotonic imaging system.

On-Sensor Experimental Prototype

Real-Time High-Fidelity Reconstruction without Hallucination

Our camera prototype improves on existing flat cameras and metalens cameras that uses diffusion-based image reconstruction in accuracy, speed, and absence of hallucinated content in the reconstructed results. We recover details such as the face of the person and the polka dots on the book, whereas the compared method instead blurs the face and mangles the dots on the book.

Robust Real-World Reconstruction in Diverse Conditions

We conducted real-world experiments across diverse and challenging conditions — including variations in illumination, depth parallax, and the presence of novel objects. We captured scenes featuring small text, human faces, significant depth variations, and low-contrast textures. These scenes were recorded for a wide range of illumination conditions such as low-light, artificial light, direct sunlight, and intense localized light sources. Below are representative measurement and reconstruction results, alongside aligned comparison measurements from a reference camera with a physical length of 50 mm.

Video-Rate Capture & Reconstruction

Our reconstruction runs at real-time rates. Compared to the diffusion-based reconstruction methods that take more than a minute to process a single image, our technique is 2000 times faster and runs at 35 FPS on the same test platform. Both approaches are implemented in PyTorch and do not leverage further optimizations. We rely on fast FFT-based inverse filtering with a subsequent denoising network that is lightweight since fusing six images already reduces noise.