CarlAnomaly
A CARLA-based Dataset for Multi-Modal Anomaly Detection
in Autonomous Driving Scenarios
Multi-Modal
- 4 RGB-D Cameras
- LiDAR
- IMU
- GNSS
Anomalies
- Various Types
- Manually Verified
- Diverse Environmental Conditions
Large Scale
- 728 scenarios
- 491,100 annotated timesteps
- 520 anomaly test clips
- 1.3 TB
Autonomous vehicles operate in open-world environments where rare, unexpected events are unavoidable — unknown obstacles, erratic behavior from other drivers, sudden weather changes, or sensor anomalies. Detecting these situations is a critical safety requirement, yet existing benchmarks either lack anomaly annotations or are limited to single images without temporal or multimodal context.
CarlAnomaly fills this gap. Generated in the CARLA simulator, it provides synchronized multimodal sensor data for 728 human-verified driving scenarios with ground-truth anomaly annotations at four levels of granularity.
| Property | CarlAnomaly |
|---|---|
| Sensor modalities | RGB cameras (×4), depth maps, instance segmentation, semantic LiDAR, GNSS, IMU, weather, collisions, control signals |
| Anomaly categories | Unknown objects · Contextual · Temporal · Covariate shifts |
| Evaluation tiers | Sample · Sensor · Timestep · Scenario |
| Training split | 101 normal scenarios, 5 min each, 6 CARLA towns |
| Test split | 627 scenarios (107 normal + 520 anomalous), 30 sec each |
| Human verification | Every scenario reviewed independently by two annotators |
@inproceedings{
}
1
2
3
4
5
6
7
8
9
10
11
12
13CarlAnomaly contains 1.3 TB of synchronized multi-modal driving data across 728 CARLA scenarios and 491,100 annotated timesteps. Anomalies fall into four categories:
- Unknown objects — categories absent from the training distribution, such as turned-off traffic lights.
- Contextual anomalies — familiar objects in implausible contexts, such as debris suddenly appearing on the road.
- Temporal anomalies — inconsistencies that only emerge over time: erratic drivers, vanishing vehicles, or abrupt weather changes.
- Covariate shifts — changes in sensor input distribution, such as flickering streetlights altering night-time illumination.