CarlAnomaly is primarily a benchmark for anomaly detection in autonomous driving. The dataset covers four categories of anomalies and supports evaluation at four levels of granularity — from individual pixels to entire driving scenarios. The dense ground-truth labels also enable secondary perception tasks such as segmentation and object detection.

Anomaly Detection

CarlAnomaly covers four categories of anomalies:

• Unknown objects — instances of categories absent from the training distribution, e.g. turned-off traffic lights that never appear during training.
• Contextual anomalies — familiar objects in implausible or safety-critical contexts, e.g. debris suddenly spawning on the road.
• Temporal anomalies — inconsistencies that only become apparent over time, such as a vehicle vanishing, erratic steering, or an abrupt weather transition.
• Covariate shifts — changes in the sensor input distribution without a change in scene semantics, e.g. flickering streetlights altering night-time illumination.

Evaluation is structured across four hierarchical tiers. Scores aggregate from finer to coarser levels: pixel or point scores are reduced to a per-sensor score, sensor scores are combined into a per-timestep score, and timestep scores determine the scenario-level decision. Anomaly types without meaningful spatial extent — vanishing actors, instant weather changes, streetlight flickering — enter the hierarchy at the timestep level rather than the sample level:

Sample Level

At the sample level, detectors assign an anomaly score to each individual sensor sample — a pixel in a camera image or a point in a LiDAR point cloud. This enables fine-grained localization of anomalous regions, analogous to anomaly segmentation.

Ground-truth per-pixel masks are provided for camera images and per-point labels for LiDAR. Anomalies that affect the global scene state — such as sudden weather changes or streetlight flickering — do not have meaningful spatial labels and are only evaluated at higher tiers.

Sensor Level

At the sensor level, a single anomaly score is assigned to one sensor reading — one camera image or one LiDAR point cloud. This evaluates whether a given observation contains any evidence of an anomaly, without requiring fine-grained localization.

Timestep Level

At the timestep level, all synchronized sensor readings from a single point in time are jointly evaluated as one multi-modal observation. This supports anomalies that affect multiple sensors simultaneously, evolve over time, or cannot be attributed to a specific pixel or point — such as a vanishing actor or an abrupt weather transition.

Scenario Level

At the scenario level, one anomaly score is assigned to an entire 30-second driving scenario. The task is to classify whether the scenario contains any anomalous event, regardless of when or where it occurs. This tier is particularly relevant for temporal and behavioral anomalies that only become apparent across many timesteps.

Secondary Tasks

The dense ground-truth labels generated in simulation support several additional perception tasks. Models are trained on the normal training split and evaluated on the test set.

Semantic Segmentation

Per-pixel semantic class labels are available for all four cameras. A segmentation model trained on the normal training split provides a natural baseline for appearance-based anomaly detection via logit confidence scores.

Instance Segmentation

Each pixel carries both a semantic class label and a unique instance identifier, enabling instance and panoptic segmentation.

Point-Cloud Segmentation

The semantic LiDAR sensor provides per-point class labels and instance IDs consistent with the camera segmentation masks, supporting 3D scene understanding.

Object Detection

3D bounding boxes for relevant traffic participants are provided in KITTI format, enabling 3D object detection and supporting geometry-based anomaly detection approaches.