Neural Single-Shot GHz FMCW Correlation Imaging

Optics Express 2024

Depth sensing is essential for 3D environmental perception across application domains, including autonomous driving, topographical mapping, and augmented and virtual reality (AR/VR). Traditional correlation time-of-flight (ToF) methods, while able to produce dense high-resolution depth maps, are plagued by phase wrapping artifacts which limit their effective depth range. Though multi-frequency methods can help reduce this problem by simultaneously solving for phase wrap counts in multiple wavelengths, this requires multiple measurements per pixel, necessitating additional hardware and imaging time. We introduce a 3D imaging method that requires a single per-point measurement by combining frequency-modulated continuous wave (FMCW) operation, all-optical correlation ToF imaging, and a specialized frequency-decoding network. Our system performs all-optical correlation imaging at GHz rates. The method is validated through both simulations and real-world experiments, comparing favorably to existing methods in all experiments.

Higher Frequency improves resolution in correlation ToF. Left: 100 MHz ToF system achieves cm-scale precision at 1 meter. Right: 10 GHz system resolves micron-scale textures. However, this comes at the cost of having to unwrap hundreds of phase wraps, which this work tackles.

Neural Single-Shot GHz FMCW Correlation Imaging

Cindy Pan, Noah Walsh, Yuxuan Zhang, Zheng Shi, Felix Heide
Optics Express 2024

Paper
Supplement
Bibtex
Code

Single-chirp GHz FMCW Correlation Imaging

Correlation Time-of-flight (ToF) methods excel in active depth sensing [2], using intensity-modulated signal interference to infer depth from phase shifts. High-frequency ToF systems need phase unwrapping algorithms for absolute depth due to phase shifts exceeding 2π. Single-frequency unwrapping recovers relative depth but requires scene assumptions. Multi-frequency unwrapping uses lower frequencies to unwrap high-frequency phases, enabling absolute depth reconstruction. We propose an depth estimation approach that integrates frequency modulated continuous wave (FMCW) operation with all-optical correlation ToF, enabling accurate absolute depth reconstruction in the GHz range from a single-chirp measurement, eliminating the need for multiple frequency measurements and effectively halving the capture time.

Our method begins with FMCW capture hardware emitting frequency-modulated signals into the scene. Reflected signals are captured, and theFMCWcorrelation signals are optically computed in the time domain. Phase shifts are then extracted before using FFT converting these signals to the frequency domain. Next, pixel-wise frequency decoding is applied to the frequency domain signals to determine absolute depths for each pixel. The final step involves a test-time optimization, which further refines the depth output by integrating the estimated depths with phase information.

All-Optical FMCW ToF Measurement Prototype

Building on Baek et al. 2022 [1], we implemented the proposed method with the all-optical FMCW prototype system shown below. (a) shows a schematic illustration of our set up and the photograph of the experimental setup is shown in (b), with light paths marked in green. The illumination module starts with a polarizing beam-splitter (PBS) converting a 3mW, 532 nm light beam into vertically polarized light. This light is then modulated by a half wave plate (HWP), a quarter wave plate (QWP), and an Electro-Optic Modulator (EOM), in sequence. After reflecting off a mirror, the sequence is repeated in reverse. An RF generator inputs a frequency chirped sinusoidal voltage (7.15 GHz center frequency, 20 MHz bandwidth) to the EOMs. The modulated signal travels to the scene and back to the detection module, which mirrors the illumination setup with synchronized HWPs, QWPs, and EOMs, and a mirror. The returned light is demodulated, mixed with a reference signal, and passed through a 1 MHz lowpass filter to extract the correlation signal.

Assessment

Using the above prototype, we validate the proposed system on real-world scenes containing target objects with different materials and complex geometric structures. In each case, we gathered 100×100 pixel measurements using our experimental prototype, assisted by a galvo scanning system. And we additionally compared to the single-frequency phase unwrapping method PUMA [3] using the same 7.15GHz measurement. Given that PUMA [3] exclusively reconstructs the relative wrap count, we modified its output to correspond with the depth range of the test scenes.

Related Work

[1] Seung-Hwan Baek, Noah Walsh, Ilya Chugunov, Zheng Shi, Felix Heide. Centimeter-Wave Free-Space Neural Time-of-Flight Imaging. SIGGRAPH 2022
[2] Ilya Chugunov, Seung-Hwan Baek, Qiang Fu, Wolfgang Heidrich, Felix Heide. Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging. CVPR 2021
[3] J. M. Bioucas-Dias and G. Valadao, “Phase unwrapping via graph cuts,” IEEE Trans. on Image Process. 16(3), 698–709 (2007).