Polarimetric Spatio-Temporal Light Transport Probing
SIGGRAPH Asia 2021
We propose a computational light transport method that decomposes light transport into all polarization states, spatial and temporal dimensions. The proposed method allows us to jointly analyze the geometry and materials found in a scene.
Light emitted from a source into a scene can undergo complex interactions with multiple scene surfaces of different material types before being reflected towards a detector. During this transport every surface reflection and propagation is encoded in the properties of the photons that ultimately reach the detector, including travel time, direction, intensity, wavelength and polarization. Conventional imaging systems typically capture intensity by integrating over all other dimensions of the incident light into a single quantity, hiding this rich scene information in the accumulated measurements. Existing methods are capable of untangling these measurements into their spatial and temporal dimensions, fueling geometric scene understanding tasks. However, examining polarimetric material properties jointly with geometric properties is an open challenge that could enable unprecedented capabilities beyond geometric scene understanding, allowing to incorporate material-dependent semantics and imaging through complex transport, such as macroscopic scattering. In this work, we close this gap, and propose a computational light-transport imaging method that captures the spatially- and temporally-resolved complete polarimetric response of a scene, which encode rich material properties. We introduce a 7D tensor formulation of light transports and instantiate our theory in two imaging prototypes: spatio-polarimetric imaging and coaxial temporal-polarimetric imaging. This allows us, for the first time, to decompose scene light transport into polarization, time, and spatial directions that unveils scene properties hidden to conventional methods. We demonstrate five applications of our method on untangling multi-bounce light transport, shape reconstruction with subsurface scattering, breaking metamerism with polarization, spatio-polarimetric decomposition of crystals, and seeing through scattering media, while outperforming conventional methods in all experiments.
Paper
Seung-Hwan Baek and Felix Heide
Polarimetric Spatio-Temporal Light Transport Probing
SIGGRAPH Asia 2021
Video Summary
Supplementary video that describes the proposed method and showcases selected spatio-polarimetric probing experiments.
Tensor-Based Light Transport Model
We formulate a tensor-based light transport model over the space, time, and polarization dimensions of the transport. Previous light transport models are designed for individual dimensions [O’Toole et al. 2012, Velten et al. 2013, Collett 2005, see manuscript] based on linear modeling as a matrix-vector multiplication. In contrast, we define the light transport as a 7D tensor consisting of 2D camera space, 2D illumination space, 1D travel time, and 2D polarization change.
Analyzing Polarimetric Light Transport Tensors
Multi-Dimensional Path Tracing
To facilitate the understanding of the multi-dimensional light transport tensor, we develop a multi-dimensional path tracer capable of rendering light with its spatial, temporal, and polarimetric resolutions. We analyze the light-transport tensor by slicing different dimensions of space, time, and polarization. The polarimetric light transport reveals material-dependent scene responses, for instance, untangling reflections from diffuse surface points (red circles) and mirrors (yellow circles), and, in fine-grained analysis, allows us to separate single-bounce specular reflection from scattered specular reflection (cyan arrows). With an additional spatial dimension, spatio-polarimetric light transport separates single-bounce specular reflection from multi-bounce specular reflection (green circles) visualized for the orange scanline. Capturing the temporal-polarimetric light transport makes it possible to untangle time-varying global illumination into diffuse and specular components (magenta circles) visualized for the purple scanline.
Low-Dimensional Polarimetric Embeddings
We perform PCA analysis for the polarimetric light transport of real-world materials [Baek et al. 2020]. We found that (a) polarimetric BRDFs with intensity normalization exhibit low-rank structure. (b) To perform PCA analysis, we propose a novel arctangent mapping that handles both negative and positive values of polarimetric light transport.
Learning Acquisition with Rotating Optical Ellipsometry
Rotating optical ellipsometry is an existing technique for acquiring polarimetric light transport of a real-world scene [Collett 2005, Azzam et al. 1978]. This requires capturing many images with diverse configurations at different rotation angles of quarter-wave plates, resulting in long acquisition times that pose a problem for multi-dimensional imaging as we aim to acquire not only polarimetric information but also spatial and temporal dimensions jointly. We present a learning-based approach to finding the optimal rotation angles of polarizing optics. Our learned ellipsometry outperforms the conventional method (DRR) and can be applied to intensity and polarizer-array sensors (L-QRPR and L-QRPR-P respectively). The proposed method substantially reduces the number of required captures by more than 3× without scarifying reconstruction accuracy. Refer to the manuscript for details.
Spatio-Polarimetric and Temporal-Polarimetric Imagers
We propose two novel imaging systems that capture spatio-polarimetric light transport and coaxial temporal-polarimetric light transport. The spatio-polarimetric system acquires the complete polarimetric scene response for every combination of camera and projector pixels over trichromatic spectral bands, using camera-projector masks and rotating QWPs. The temporal-polarimetric setup captures the coaxial spatial dimension at picoseconds temporal resolution for a narrow spectral band, also with the full polarimetric resolution. We scan the transport using the 2-axis galvo mirror and the rotating QWPs.
Spatio-Polarimetric Imaging
The proposed method is capable of untangling complex reflections by exploiting the spatial and polarimetric dimensions of light transport. Our method enables separating the interreflection between the coins only and direct reflections, or capturing interreflection between the coins and the floor.
Temporal-Polarimetric Imaging
We decompose mingled temporal transport of a statue with dielectric surface material and a metallic dragon into their individual reflection components.
Shape Reconstruction with Subsurface Scattering
Reconstructing the shape of an object in the presence of strong subsurface scattering is challenging for conventional structured light methods. We exploit spatio-polarimetric information to obtain accurate geometry of the dragon statue.
Seeing through Scattering Media
Scattering media, such as milky water and fog, can significantly decrease scene visibility. While previous methods attempted to image through scattering media by relying on either polarization or space, we jointly exploit the joint spatio-polarimetric dimensions, resulting in improved visibility.
Transport Analysis for Semi-Transparent Crystals
The proposed spatio-polarimetric imaging method allows for capturing surface details of crystalline objects and identifying their optical anisotropy (birefringence).
Breaking Metamerism with Polarization
Spatio-polarimetric imaging can distinguish real from fake fruits by capturing sub-surface scattered light.
Related Publications
[1] Seung-Hwan Baek, Tizian Zeltner, Hyun Jin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, Min H. Kim. Image-Based Acquisition and Modeling of Polarimetric Reflectance. ACM Transactions on Graphics (TOG), 2020
[2] Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, Min H. Kim. Simultaneous Acquisition of Polarimetric SVBRDF and Normals. ACM Transactions on Graphics (TOG), 2018
[3] Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, Kiriakos N. Kutulakos. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Transactions on Graphics (TOG), 2014