All-photon Polarimetric Time-of-Flight Imaging
We propose all-photon polarized ToF imaging. We extract the temporal-polarimetric characteristics of both surface and sub-surface reflections relying on a novel temporal-polarimetric BRDF model. This enables us to infer scene parameters (depth, normals, and material parameters) from all photons fitting through the aperture in an analysis-by-synthesis fashion.
Time-of-flight (ToF) sensors provide an image modality fueling diverse applications, including LiDAR in autonomous driving, robotics, and augmented reality. Conventional ToF imaging methods estimate the depth by sending pulses of light into a scene and measuring the ToF of the first-arriving photons directly reflected from a scene surface without any temporal delay. As such, all photons following this first response are typically considered as unwanted noise. In this paper, we depart from the principle of using first-arriving photons and propose an all-photon ToF imaging method by incorporating the temporal-polarimetric analysis of first- and late-arriving photons, which possess rich scene information including geometry and material. To this end, we propose a novel temporal-polarimetric reflectance model, an efficient capture method, and a reconstruction method that exploits the temporal-polarimetric changes of light reflected by the surface and sub-surface reflection. The proposed all-photon polarimetric ToF imaging method allows for acquiring depth, surface normals, and material parameters of a scene by utilizing all photons captured by the system, whereas conventional ToF imaging only obtains coarse depth from the first-arriving photons. We validate our method in simulation and experimentally with a prototype.
All-photon Polarimetric ToF Imaging
In contrast to conventional ToF imaging methods that aim to measure the first-reflected light, we propose to capture and exploit both early and late photons and untangle them using their polarization state. We analyze temporal-polarimetric characteristics of surface and sub-surface reflections that enable us to infer scene parameters (depth, normals, and material parameters) from all captured photons.
We propose a novel temporal-polarimetric BRDF model and solve for scene parameters in an inverse rendering fashion using differentiable reconstruction. As shown on the left, we fit our model to temporal polarimetric measurements to recover depth, normalsm and material properties.
Temporal-polarimetric BRDF Model
To make differentiable reconstruction possible, we introduce the first temporal-polarimetric reflectance model which explains the total reflection (left) as a sum of the temporal polarization change in surface (middle) and sub-surface reflections (right). Here, we visualize an example reflectance function in the form of a temporal-polarimetric Mueller matrix. Time-varying Mueller matrix elements are shown in the second and third rows for diagonal and non-diagonal elements, respectively.
Energy-efficient Polarimetric ToF Imaging
(a) For a polarimetric capture, we rotate a HWP instead of a LP to preserve the light intensity of the polarized source. (b)We learn the rotation angles of the polarizing optics with uniform initialization which allows us to reconstruct polarimetric reflectance with fewer captures than conventional ellipsometry methods. We visualize the Poincar´e sphere of the polarizing (c) and analyzing (d) Stokes vectors before and after learning.
We built (a) an experimental prototype for polarimetric ToF imaging, allowing us to efficiently capture temporal-polarimetric images. Its optical path is shown in (b). A pulsed laser emits photons, and we modulate their polarization states using a half-wave plate (HWP) and a rotating quarter-wave plate (QWP). The photons are then redirected to the scene by a galvo mirror. After light transport, our imaging system receives the returning photons and analyzes their polarization state using a rotating QWP and a linear polarizer (LP). A SPAD sensor records the intensity of incident photons with picosecond temporal resolution.
Temporal-polarimetric BRDF Analysis
The proposed BRDF model is a parametric representation that is capable of describing temporal-polarimetric reflections for surface (orange arrows) and sub-surface scattering (pink arrows). To validate the proposed model, we measure the temporal-polarimetric response of a spherical object (top) using our prototype. Our parametric BRDF model (bottom) accurately describes the measurements in terms of surface reflection (middle) and subsurface reflection (right) at different time stamps.
We evaluate all-photon ToF imaging on synthetic data simulating the entire process of (a) temporal-polarimetric
capture, (b) ellipsometric reconstruction, and (c) scene reconstruction. The proposed method accurately estimates (c) per-object material parameters, (d) travel distance, and (e) surface normals.
We validate all-photon polarimetric ToF imaging on real-world scenes. (a) Conventional peak-finding ToF imaging methods acquire scene depth from first-reflected photons only, resulting in inaccurate geometry reflected in imperfect depth and surface normals estimates. In contrast, the proposed method exploits all photons, enabling (b) accurate geometry reconstruction and the recovery of material parameters by (c) decomposing light transport into surface and sub-surface components.
 Seung Hwan Baek, Felix Heide., Polarimetric Spatio-Temporal Light Transport Probing., ACM Transactions on Graphics (TOG), Proc. SIGGRAPH Asia 2021
 Seung Hwan Baek, Tizian Zeltner, Hyun Jin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, Min H. Kim., Image-Based Acquisition and Modeling of Polarimetric Reflectance., ACM Transactions on Graphics (TOG), Proc. SIGGRAPH 2020