ScatterNeRF: Seeing Through Fog with Physically-Based
Inverse Neural Rendering

Disentangled Scene Representation

In this work, we depart from both feed-forward dehazing methods and fully synthetic training data, and we address this challenge as an inverse rendering problem. Instead of predicting clean images from RGB frames, we propose to learn a neural scene representation that explains foggy images with a physically accurate forward rendering process. Specifically, we propose a disentangled representation for the scattering volume and the scene objects, and we learn the scene reconstruction with physics-inspired losses. The total image formation model for the original hazed scene is therfore defined as:

$$\mathbf{C}_F(\mathbf{r}) =\ \int_{t_n}^{t_f} T_c(t) \underbrace{\left(T_p(t) \sigma_p(\mathbf{r}(t)) \mathbf{c}_p(\mathbf{r}(t)) \right)}_{\text{Fog Contribution}} \nonumber \quad + T_p(t)\underbrace{\left(T_c(t)\sigma_c(\mathbf{r}(t)) \mathbf{c}_c(\mathbf{r}(t)) \right) }_{\text{Clear scene contribution}}dt.$$

Once this representation is fitted to a scene, this allows us to render novel views with real-world physics-based scattering, disentangle appearance and geometry without scattering (i.e., reconstruct the dehazed scene), and estimate physical scattering parameters accurately, as shown in the figure below.

Fog-Unbiased Sampling Instead of sampling across the whole volume density we adapt the approach to our decomposed volume densities $\sigma_c$ and $\sigma_F$. As $\sigma_F$ is regularized to be approximately constant across the scene, the re-sampling procedure is not going to be performed using the scene weights $w_F = T_F (1-\exp(\sigma_F \delta))$ but rather using the clear scene $w_c = T_c (1-\exp(\sigma_c \delta) )$. We apply this approach to follow an importance sampling and reconstruct scene objects by sampling close to object boundaries, as exemplified in the figure to the right.

In-the-Wild Experiments

ScatterNeRF Dataset

To evaluate the proposed method, we collect both an automotive In-the-Wild foggy dataset and a controlled fog dataset.

In-the-Wild Dataset

In total, we collect 2678 In-the-Wild foggy images throughout nine different scenarios.The sensor set for the In-the-Wild dataset consists of an Aptina-0230 stereo camera pair and a Velodyne HDL64S3D laser scanner. Camera poses are estimated with the hierarchical localization approach, a structure from motion pipeline optimized for robustness to changing conditions. The training and testing split was done by randomly choosing 30% of the images as test set.Each scene contains between 150 and 300 images.

Controlled-Environment Dataset

The controlled fog dataset is captured in a fog chamber where fog with varying visibilities can be generated. We capture 903 images in a large-scale fog chamber with clear ground-truth and two different fog densities. Further ground-truth depths are captured through a Leica ScanStation P30 laser scanner. Each point cloud consists of approximately 157M points and we accumulate multiple point clouds from different positions to reduce occlusions and increase resolution.