Lidar Waveforms are Worth 40x128x33 Words

Lidar has become crucial for autonomous driving, providing high-resolution 3D scans that are key for accurate scene understanding. As shown on the right, lidar sensors measure the time-resolved full waveforms from the returning laser light, which a subsequent digital signal processor (DSP) converts to point clouds by identifying peaks in the waveform. Conventional automotive lidar DSP pipelines process each waveform individually, ignoring potentially valuable context from neighboring waveforms. As a result, lidar point clouds are prone to artifacts from low signal-to-noise ratio (SNR) regions, highly reflective objects, and environmental conditions like fog. In this work, we propose a neural DSP that jointly processes full waveforms using a transformer architecture leveraging features from adjacent waveforms to generate high-fidelity multi-echo point clouds. Trained on synthetic and real data, the method improves Chamfer distance by 32cm and 20cm compared to conventional peak finding and existing transient imaging approaches, respectively. This translates to maximum range improvements of up to 17m in fog and 14m in challenging real-world conditions

Quantitative Results

As shown in the Table on the right, we use Chamfer Distance (CD) and Recall to assess the quality of the predicted point clouds. To assess the robustness in challenging conditions, we evaluate two signal-to-noise ratio (SNR) ranges: difficult low-SNR conditions (0-2) and easier high-SNR conditions (2-4). The quantitative results confirm the above qualitative findings.  Existing transient imaging methods by Lindell et al. and Peng et al. improve reconstruction in low SNR (0–2) scenarios. At high SNR (2–4), their denoising strategy offers however no advantage over conventional peak finding (Conv.). In contrast, the proposed neural DSP outperforms baselines across all SNR ranges and excels in fog. Notably, the neural DSP has never seen any real fog captures during training but was only trained with synthetic fog data, validating the benefit and high-degree of realism of our presented waveform simulation model.

Super-Resolution Waveform Cues

Current lidar sensors suffer from significantly lower resolution compared to, e.g., cameras. However, the full waveform contains information useful to increase the resolution without the need for more pixels. For example, multiple peaks in the waveform suggest that a higher-resolution lidar with smaller beam divergence would likely resolve distinct points. Conversely, a single strong peak is an indicator that a higher-resolution system would detect multiple points at the same distance.
As shown on the right, extending our neural DSP to exploit these cues to render super-resolution point clouds is beneficial for distinguishing small hazardous items from the road, as shown on the right. The best-performing baseline ILN suffers from artifacts and flying pixels.