Automotive Datasets

Reliable perception in all weather conditions remains one of the most critical challenges in autonomous driving. Adverse weather conditions such as snow, heavy rain, and fog can severely degrade the ability of current systems to sense and interpret their surroundings. This has long been considered one of the last remaining technical challenges preventing self-driving cars from reaching the market. To address this, we present a suite of multimodal datasets designed to promote robust environmental perception under a wide range of conditions. An overview of the dataset is presented below:

Object detection dataset in challenging adverse weather conditions

Sensor setup

To capture the dataset, we have equipped a test vehicle with sensors covering the visible, mm-wave, NIR, and FIR band, see Figure below. We measure intensity, depth, and weather conditions. For more information, please refer to our dataset paper.

Videos

Dataset Overview

We present an imaging framework which converts three images from a low-cost CMOS gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements.The main contributions are:

• We propose a learning-based approach for estimating dense depth from gated images, without the need for dense depth labels for training.
• We validate the proposed method in simulation and on real-world measurements acquired with a prototype system in challenging automotive scenarios. We show that the method recovers dense depth up to 80m with depth accuracy comparable to scanning lidar.
• Dataset: We provide the first long-range gated dataset, covering over 4,000km driving throughout Northern Europe.

Dataset Overview

We present a self-supervised imaging framework, which estimates dense depth from a set of three gated images without any ground truth depth for supervision. By predicting scene albedo, depth and ambient illumination, we reconstruct the input gated images and enforce cycle consistency as can be seen in the title picture. In addition, we use view synthesis to introduce temporal consistency between adjacent gated images.
The main contributions are:

• We propose the first self-supervised method for depth estimation from gated images that learns depth prediction without ground truth depth.
• We validate that the proposed method outperforms previous state of the art including fully supervised methods in challenging weather conditions as dense fog or snowfall where lidar data is too cluttered to be used for direct supervision.
• Dataset: We provide the first temporal gated imaging dataset, consisting of around 130,000 samples, captured in clear and adverse weather conditions as an extension of the labeled Seeing Through Fog dataset which provides detailed scene annotations, object labels and corresponding multi modal data.

Dataset Overview

We introduce an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25” (arcsecond), akin to a 50 megapixel camera with per-pixel depth available. The main contributions of this dataset are:

• High-resolution ground truth depth: annotated ground truth of angular resolution 25” – an order of magnitude higher than existing lidar datasets with angular resolution of 300”.
• Fine-Grained Evaluation under defined conditions: 1,600 samples of four automotive scenarios recorded in defined weather (clear, rain, fog) and illumination (daytime, night) conditions.
• Multi-modal sensors: In addition to monocular camera images, we provide a calibrated sensor setup with a stereo camera and a lidar scanner.

We model four typical automotive outdoor scenarios from the KITTI dataset. Specifically, we setup the following four realistic scenarios: pedestrian zone, residential area, construction area and highway. In total, this benchmark consists of 10 randomly selected samples of each scenario (day/night) in clear, light rain, heavy rain and 17 visibility levels in fog (20-100 m in 5 m steps), resulting in 1,600 samples in total.

Sensor Setup

To acquire realistic automotive sensor data, which serves as input for the depth evaluation methods assessed in this work, we equipped a research vehicle with a RGB stereo camera (Aptina AR0230, 1920×1024, 12bit) and a lidar (Velodyne HDL64-S3, 905 nm). All sensors run in a robot operating system (ROS) environment and are time-synchronized by a pulse per second (PPS) signal provided by a proprietary inertial measurement unit (IMU).

Videos