Spectral and Polarization Vision: Spectro-Polarimetric Real-World Dataset

CVPR 2024 (Highlight)

Image datasets are essential not only in validating existing methods in computer vision but also in developing new methods. Many image datasets exist, consisting of trichromatic intensity images taken with RGB cameras, which are designed to replicate human vision. However, polarization and spectrum, the wave properties of light that animals in harsh environments and with limited brain capacity often rely on, remain underrepresented in existing datasets. Although there are previous spectro-polarimetric datasets, they have insufficient object diversity, limited illumination conditions, linear-only polarization data, and inadequate image count. Here, we introduce two spectro-polarimetric datasets, consisting of trichromatic Stokes images and hyperspectral Stokes images. These datasets encompass both linear and circular polarization; they introduce multiple spectral channels; and they feature a broad selection of real-world scenes. With our dataset in hand, we analyze the spectro-polarimetric image statistics, develop efficient representations of such high-dimensional data, and evaluate spectral dependency of shape-from-polarization methods. As such, the proposed dataset promises a foundation for data-driven spectro-polarimetric imaging and vision research.

Spectral and Polarization Vision: Spectro-Polarimetric Real-World Dataset

Yujin Jeon, Eunsue Choi, Youngchan Kim, Khalid Omer, Felix Heide, Seung-Hwan Baek

CVPR 2024

Acquisition of Spectro-Polarimetric Image Dataset

We acquire the data for the proposed dataset using two imaging systems. First, the trichromatic Stokes camera by Tu et al (2020) incorporates on-sensor quarter- wave plates (QWPs) and linear polarizers (LPs) [9]. This allows for single-shot capture of trichromatic Stokes images, enabling convenient data collection on diverse scenes. The resolution of a trichromatic Stokes image is 2100 (width) × 1920 (height) × 3 (RGB) × 4 (Stokes elements).

Second, the hyperspectral Stokes camera from Kim et al (2023) captures images by sequentially scanning 21 spectral channels from 450 nm to 650 nm in 10 nm increments with a LCTF which functions as a LP. For each spectral channel, we capture images by rotating a QWP with a fixed LP. This system enables a detailed analysis of the interplay between wavelength and polarization. The resolution of a hyperspectral Stokes image is 612 (width) × 512 (height) × 21 (wavelengths) × 4 (Stokes elements)

Scene Diversity in Dataset

We present trichromatic and hyperspectral Stokes datasets of which labels are presented above. We propose a comprehensive spectro-polarimetric dataset that encompasses: (1) Full Stokes polarimetric data, including both linear and circular polarization states, represented by Stokes vectors for each pixel and wavelength. (2) A diverse range of spectral channels, facilitating in-depth exploration of the interplay between spectrum and polarization. (3) A broad array of real-world scenes, crucial for extracting meaningful statistics and relationships within spectro-polarimetric images.

Dataset Samples

We collected the data based on nine different labels, encompassing location and time (indoor, outdoor daytime, outdoor night), scene types (object-oriented and scene-oriented), illumination sources (white light, incandescent light, clear sunlight and cloudy conditions).

Related Publications

[1] Seung-Hwan Baek and Felix Heide. Polarimetric spatio-temporal light transport probing. ACM ToG SIGGRAPH 2021
[2] Youngchan Kim, Wonjoon Jin, Sunghyun Cho, and Seung-Hwan Baek. Neural spectro-polarimetric fields. SIGGRAPH Asia 2023