Gated Fields: Learning Scene Reconstruction from Gated Videos

Neural Gated Fields

The neural gated field approach is tailored for reconstructing scenes from gated videos using a neural field representation. Active gated slices, alongside a passive slice, capture the scene, integrating photometric cues and scene priors. Neural fields, such as those for ambient light, surface reflection, and shadow effects, are conditioned on spatial embedding, facilitating joint estimation of light and material properties. A physics-based model handles illumination and shadow modeling, employing a higher-order Gaussian function to compute illumination intensity. Shadow effects are directly estimated from the density field.

\[
\tilde{I}_{k}(\mathbf{r}) = \sum_{j=0}^{N} w_j \Big( \underbrace{\alpha_j \tilde{C}_{j} \psi_j |\mathbf{n}_j \cdot \omega_j | \iota_j}_{\substack{\text{Active} \\ \text{Component}}} +\underbrace{\Lambda_j}_{\substack{\text{Passive} \\ \text{Component}}} \Big) +\mathcal{D}_k
\]

During training, the model is supervised through various loss functions. Photometric loss ensures alignment between predicted and ground truth images, while volume density regularization and shadow estimation refine scene representation. Normals consistency and reflectance regularization ensure spatial coherence and accurate separation of scene components. These losses collectively constitute the total training loss, guiding optimization. Additionally, the approach integrates LiDAR rendering for enhanced depth and intensity estimation, further enriching scene reconstruction capabilities.

Results: 2D Depth Estimation

The evaluation of Gated Fields’ depth synthesis quality for unseen camera poses relies on accumulated LiDAR point cloud data as ground truth, enabling unbiased evaluation up to 160 meters. Comparative analysis against nine depth estimation methods, including RGB-based and LiDAR-based approaches, highlights Gated Fields’ superior performance, particularly in night conditions. The limitations of RGB-based methods in low-light settings underscore the advantage of gated imaging’s ability to capture meaningful 3D representations consistently across lighting conditions.

Results: 3D Geometry Reconstruction

We evaluate the 3D scene reconstruction capabilities of Gated Fields using accumulated LiDAR point cloud data as ground truth. Employing voxelized occupancy grids, we compare against RGB and RGB-LiDAR fusion methods, computing metrics such as IoU, Precision, and Recall. Results show Gated Fields outperforming RGB baselines by an average of 15% in IoU. While RGB-LiDAR fusion methods partially enhance results, they struggle with finer surface details and low-light conditions. In contrast, Gated Fields leverage gated imaging, illumination, and depth cues for reconstructing finer geometries consistently across lighting conditions.

Dataset

The dataset created in this study comprises 10 static sequences captured in diverse day and night conditions across North America. Equipped with a NIR gated stereo camera setup, RGB stereo camera, LiDAR sensor, and GNSS with IMU, the test vehicle records synchronized high-resolution images and depth data. Each gated camera operates at 1280×720 resolution and 120Hz, while the RGB cameras provide 12-bit HDR images at 1920×1080 resolution and 30Hz frame rate. LiDAR sensor data is aggregated to construct a dense ground truth pointcloud, refined using LIO-SAM for precise scene representation. With 2650 samples split into training, validation, and test sets, the dataset offers diverse scenes for scene reconstruction tasks, supported by visualizations and detailed data aggregation methods in the Supplementary Material.