KITScenes Multimodal: A Grouped Sensor Dataset in FiftyOne

A single frame from a modern robotaxi's sensor suite is not one file. It's ten.

Nine synchronized global-shutter cameras. A fused point cloud built from seven lidars and three radars. An HD map is reprojected into every image. A predicted path. A GNSS trace. All of it describes the exact same tenth of a second, from a different sensor's perspective.

Collected that way, a “frame” is a scavenger hunt. You open the front camera in one viewer, the point cloud in another, the map in a third, and you reconstruct the moment in your head instead of looking at it. Even something as simple as checking whether a predicted lane boundary is actually wrong requires lining up three files by timestamp before you can look at anything.

This is exactly the problem KITScenes Multimodal and FiftyOne solve together. KITScenes Multimodal, built by the Institute of Measurement and Control Systems (MRT) at the Karlsruhe Institute of Technology and the FZI Research Center for Information Technology, packs one of the densest sensor rigs in any public driving dataset. This post walks through a FiftyOne build that groups all ten of those sensors under one id per driving moment, so a frame is one sample again.

Key Takeaways

KITScenes Multimodal packs 72.5 Mpx across nine global-shutter cameras, seven lidars averaging roughly 906K points per frame with an effective range beyond 400 m, and three 4D imaging radars into one synchronized rig. This FiftyOne build groups all ten of those slices under a single group id, so a driving instant is one sample, not ten files to reconcile by hand.
The included Lanelet2 HD maps are reprojected directly into every ring-camera image as the hd_map field, with a subtype attribute that goes all the way down to the specific German traffic-sign code. You check map accuracy against pixels, not a rasterized top-down view.
lidar_depth fuses the return of all seven lidars and three radars into a per-camera depth heatmap, giving you dense ground truth exactly where monocular depth models are known to degrade: past 200 m.
The fused lidar slice keeps ground segmentation (isground) and Doppler velocity (range_rate) as per-point scalar fields, so you filter the point cloud interactively in the App instead of writing a preprocessing script first.
This build is an early-release preview: 4 validation scenes from Frankfurt, no dynamic 3D boxes or tracks yet, and the seamseg instance boxes are model predictions, not human annotations. Treat it as a pipeline-development sandbox, not a benchmark leaderboard submission.

What is KITScenes Multimodal?

Six KITScenes Multimodal ring-camera views with lidar_depth overlays, alongside the reprojection-accurate Lanelet2 HD map for the same driving instant.

Public driving datasets have carried the field a long way, but the paper behind KITScenes Multimodal makes its motivation blunt: “Existing autonomous driving datasets have enabled major progress, but fall short in sensor fidelity, map completeness, or geographic diversity.” nuScenes, Waymo Open Dataset, and Argoverse 2 remain the reference benchmarks, and most high-fidelity, map-annotated datasets skew toward North America and Asia.

KITScenes Multimodal is KIT-MRT's answer: a European urban dataset recorded across Karlsruhe, Frankfurt, and Sindelfingen, built around a sensor suite that is unusually dense on every axis at once.

The camera rig alone puts up 72.5 Mpx per frame, more than twice the next-closest public dataset: six 7.1 Mpx global-shutter surround cameras for 360° coverage, one 16.2 Mpx long-range forward camera, and a tilted forward stereo pair, all hardware-synchronized for pixel-accurate temporal alignment. Underneath that, seven lidars average around 906K points per frame (peaking above 1.2M) with an effective range past 400 m, “nearly doubling the next-best dataset.” Three Continental ARS548 4D imaging radars add Doppler velocity and weather-robust returns the lidar and cameras can't provide on their own.

The HD maps are the other half of the pitch. Annotated in Lanelet2 format across 62 km², with 29 road-feature classes and traffic signs classified against a 220-class taxonomy (120 of which are actually observed in this build), the maps carry full topological connectivity, every lane, sign, and light is linked to the lanes it governs, and they've been validated in closed-loop autonomous driving trials on the open-source Autoware stack. That's the claim behind “production-grade”: these maps are good enough to drive on.

Good to know. How is this different from nuScenes or Waymo Open Dataset?

Mostly in sensor density and map completeness rather than scale. KITScenes Multimodal's full corpus is 1,007 scenarios across 162 km, smaller than nuScenes or Waymo in raw hours. What it adds is fidelity per frame: more than double the camera resolution, roughly triple the lidar point density, lidar range beyond 400 m versus the ~100–200 m typical of prior datasets, and HD maps with full lane-to-sign-to-light topology rather than lane geometry alone. The paper's own framing: it's built to close “a clear need for datasets that combine high-fidelity sensing, complete geospatial context, and dense 3D annotations,” not to out-scale the largest corpora.

What's in this FiftyOne build of KITScenes Multimodal

This build packages the validation split as a grouped FiftyOne dataset: every sensor for a given moment lives in one group, and the 3D lidar/radar point cloud sits alongside the camera images.

KITScenes Multimodal at a glance: the FiftyOne build's dataset name, media type, sample and group counts, scene coverage, capture rate, and license.

KITScenes Multimodal at a glance: the FiftyOne build's dataset name, media type, sample and group counts, scene coverage, capture rate, and license.
Dataset name	kitscenes-multimodal
Media type	group (grouped dataset)
Samples	6,800
Frames (groups)	680
Scenes	4 (validation split, Frankfurt)
Group slices	9 cameras + 1 fused 3D lidar/radar slice
Capture rate	10 Hz
License	CC-BY-NC-4.0

A group is one timestamped frame holding 10 samples: the 9 camera images plus the fused point cloud. 680 groups × 10 slices = 6,800 samples.

The 10 group slices in each frame: nine camera images (six ring cameras plus three long-range and stereo cameras) and one fused 3D lidar/radar point cloud.

The 10 group slices in each frame: nine camera images (six ring cameras plus three long-range and stereo cameras) and one fused 3D lidar/radar point cloud.
Slice	Media	Role
camera_ring_front	image	Forward ring camera (default view)
camera_ring_front_left / _right	image	Ring cameras, front-left / front-right
camera_ring_rear	image	Rear ring camera
camera_ring_rear_left / _right	image	Ring cameras, rear-left / rear-right
camera_base_front_center	image	16.2 Mpx long-range front camera
camera_base_front_left_rect / _right_rect	image	Rectified front stereo pair
lidar	3d	Fused point cloud: 7 lidars + 3 radars, ego frame

Good to know. What's the difference between the full KITScenes Multimodal release and this FiftyOne build?

This build is a preview: 4 validation scenes from Frankfurt (680 frames), meant for pipeline exploration rather than final benchmark reporting, which matches how KIT-MRT itself describes the early release. The full corpus behind the paper is 1,007 scenarios (534 train / 117 val / 206 test / 127 held-out test-e2e) across three cities and 162 km. This build also doesn't include dynamic 3D boxes, tracks, or instance segmentation for moving agents, the source release omits those for now, and the seamseg boxes you do get are model predictions, not human annotations. If you want a smaller entry point still, there's a single-scene preview repo (KIT-MRT/KITScenes-Multimodal-Sample) too.

Step 1: Load it and see what a group actually holds

Once loaded, every timestamp is a group of 10 samples instead of 10 unrelated files. The App opens on camera_ring_front by default; switch slices from the same panel to move between cameras and the fused point cloud without losing your place in the scene.

FiftyOne App showing camera_ring_front for the kitscenes-multimodal dataset, colored by the lidar_depth heatmap field, with the labels sidebar listing lidar_depth, hd_map, ego_trajectory, and seamseg.

That yellow-to-purple overlay is lidar_depth, dense depth fused from all seven lidars and projected into the image, near in yellow, far in purple. It's on by default the moment you toggle the field in the sidebar; no reprojection code required.

Step 2: Walk the 360° ring, then drop into the point cloud

The six camera_ring_* slices form the surround view; the three camera_base_* slices are the long-range and stereo cameras. Stepping between them on the same group keeps you anchored to one instant, so if something is ambiguous in camera_ring_front, the adjacent ring cameras usually resolve it.

The lidar slice is where the sensor suite actually shows its density. It's a single .fo3d scene per frame fusing all seven lidars (motion-deskewed) and three radars (ego-motion compensated) into one ego-frame point cloud, shaded by intensity in the App. Each point carries scalar fields you can filter on directly:

Lidar points: intensity (reflectivity) and isground (per-point ground flag).
Radar points: intensity (RCS) and range_rate (Doppler velocity), courtesy of the three Continental ARS548 4D imaging radars, each covering 300 m at up to 0.22 m range resolution and a −400 to 200 km/h velocity window.

Turn off isground and you're looking at only the obstacles and structure above the road, no ground plane fit required beforehand.

Step 3: Check the map against the pixels, not a bird's-eye view

hd_map reprojects Lanelet2 elements straight into the six ring cameras: lane markings, road borders, road markings, poles, traffic signs, and traffic lights, as Polylines. Each polyline carries a coarse label and a subtype attribute holding the fine-grained Lanelet2 class, down to lane-marking style (dashed, solid, dashed_solid) or the specific German traffic-sign code (de206, for instance).

That's the payoff of “reprojection-accurate”: instead of comparing a predicted map against a rasterized top-down ground truth, you look at whether the map element lines up with the actual sign or lane line in the image.

Six camera views of a Frankfurt street with HD-map polylines overlaid (road border, curbstone, drivable area, dashed/solid/virtual lane lines, centerlines, crosswalks, stop lines) and a legend of the Lanelet2 style categories.

Good to know. What is Lanelet2, and why does topology matter here?

Lanelet2 is a map format built around lanelets, atomic lane segments with topological links to their neighbors, the signs that govern them, and the lights that control them. KITScenes Multimodal's maps carry that connectivity end to end across 62 km² and 29 road-feature classes, which is why the paper calls them the most complete public HD maps available: every sign and light is explicitly tied to the lane it governs, not just geometrically nearby.

Step 4: Get depth where cameras alone run out

The lidar_depth heatmap on camera_ring_front, near-to-far depth fused from all seven lidars and projected into the image, with near returns in yellow and far in purple.

lidar_depth is an 8-bit depth heatmap on all nine cameras, fused from the full lidar rig and encoded near-to-far. It exists because monocular depth is a known failure mode past a certain range, and this dataset's own benchmark quantifies exactly where: split into 0–100 m, 100–200 m, and beyond-200 m bins, current methods hold up reasonably well close in and fall apart past 200 m, “no method achieves reliable performance beyond 200 m,” per the paper's own results. lidar_depth gives you dense ground truth precisely in the range band where that's hardest to get any other way.

Step 5: Check where the vehicle is headed

Every sample carries the ego pose: ego_translation, ego_quaternion, ego_yaw_deg, plus GNSS fields (location, altitude, gnss_fix_status, ego_speed). That's a redundant GNSS/INS setup under the hood, accurate to roughly 0.6 cm horizontally under RTK, so the trajectory you reconstruct from these fields is trustworthy, not just plausible.

On top of that, camera_ring_front carries ego_trajectory, a Keypoints field labeled ego_path that projects the vehicle's actual future waypoints onto the road ahead. Step through a scene with both the pose fields and ego_trajectory visible and you get the car's intended path and its measured one, in the same frame.

Step 6: See what's already been detected

seamseg detections (boxes plus masks) on a KITScenes Multimodal street scene, in the Mapillary Vistas taxonomy, shown as model predictions rather than ground truth.

seamseg gives you Detections, boxes plus masks, on camera_ring_front and camera_ring_rear, in the Mapillary Vistas “thing” taxonomy: vehicles (Car, Truck, Bus, Trailer, Other Vehicle), vulnerable road users (Person, Bicyclist, Motorcyclist, Other Rider), and street furniture (Traffic Light, Traffic Sign (Front/Back), Pole, Utility Pole, Bench, Fire Hydrant, and more).

Worth repeating plainly: these are model predictions, not human annotations. Use them as a starting point for filtering interesting frames, not as ground truth for evaluation.

Step 7: Build the views you'll actually reuse

Three saved views ship with the dataset already:

The three saved views that ship with the dataset, and what each one groups: forward ring camera, rear ring camera, and the fused lidar slice by scene.

The three saved views that ship with the dataset, and what each one groups: forward ring camera, rear ring camera, and the fused lidar slice by scene.
View	What it shows
ring_front_by_scene_frame	Forward ring camera, grouped by (scene_id, frame) — 680 groups
ring_rear_by_scene_frame	Rear ring camera, grouped by (scene_id, frame) — 680 groups
lidar_by_scene	Fused lidar slice grouped by scene_id — 4 groups, one per scene

A grouped saved view in the FiftyOne App showing four KITScenes Multimodal scenes at once, with the ego trajectory drawn on each.

Building your own is a few lines. Say you want every frame where the HD map places a traffic light in view, a fast proxy for intersections:

Tag by eye as you scrub through the App, or threshold on a field like ego_speed to isolate stopped-traffic frames programmatically and refine from there. Either way, the view gets saved once and reused across every future session, instead of re-deriving “the busy intersections” from scratch each time you need them.

The four KITScenes Multimodal benchmarks behind this preview

The paper attached to KITScenes Multimodal introduces four benchmarks, and this FiftyOne build is the exploration layer that sits in front of all of them:

Online HD map construction. Predicting the full relational Lanelet2 map, lane markings, centerlines, road infrastructure, traffic lights, traffic signs, road markings, not just lane boundaries, scored as Average Precision at Chamfer-distance thresholds of 0.5, 1.0, and 1.5 m. The paper's own baselines (MapTRv2, SDTagNet) show “a large performance drop on our complete formulation compared to existing benchmarks, revealing a gap hidden by the currently limited task scope.”

Long-range monocular depth estimation. Stratified into 0–100 m, 100–200 m, and beyond-200 m bins against lidar-fused ground truth. The result worth remembering: the model that wins overall degrades hardest past 100 m, and the model that ranks last overall is actually the strongest long-range estimator, “no method achieves reliable performance beyond 200 m.”

Novel view synthesis. Reprojected HD-map traffic signs, lidar-checked for occlusion, are used to score synthesized views at seven lateral offsets from −3 m to +3 m. Recall drops over 80% relative at the ±3 m extremes, and even at zero lateral offset, evaluating at full sensor resolution instead of model output resolution costs a 27.8% relative recall drop versus 7.6% at low resolution, a gap current image-quality metrics like PSNR and SSIM don't surface.

End-to-end driving. The same scenes evaluated across three input tiers, single front camera, full 360° surround, and the complete multimodal suite, letting you isolate exactly what each sensor adds to a driving policy rather than comparing across incompatible setups.

Keep it honest. This build is 680 frames from 4 Frankfurt validation scenes, a slice of the full 1,007-scenario corpus (534 train / 117 val / 206 test / 127 held-out test-e2e) behind those benchmark numbers. It's built for looking at the data and developing a pipeline, not for reproducing leaderboard results, and it doesn't yet carry the dynamic 3D annotations the end-to-end benchmark ultimately depends on.

Why curation still matters even on a preview build

Model development gets more interesting exactly where the data gets rare: the busy intersection, the occluded pedestrian, the sign the map disagrees with the pixels on. Nothing about a 680-frame preview changes that pattern, only the scale of it.

Save the view once, whether it's every frame with a traffic light in frame, every frame with a wide lidar_depth heatmap gap, or every frame where seamseg flags something the HD map doesn't expect, and every future session starts from the curated set instead of the full 6,800 samples. That's the whole difference between staring at leaderboard metrics and actually knowing why a model missed something.

Try it yourself

FiftyOne build (Hugging Face): Voxel51/kitscenes-multimodal
Original dataset (Hugging Face): KIT-MRT/KITScenes-Multimodal
Single-scene preview (Hugging Face): KIT-MRT/KITScenes-Multimodal-Sample
Python devkit (GitHub): KIT-MRT/kitscenes
Paper: The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset
Project page: kitscenes.com/multimodal
FiftyOne docs: docs.voxel51.com

Frequently Asked Questions

What's actually inside one sample of this build?

One group is one timestamped driving instant with 10 samples underneath it: nine camera images (camera_ring_front, its five ring neighbors, and three camera_base_* long-range/stereo cameras) plus one fused lidar slice combining seven lidars and three radars. All ten share scene_id, frame, and timestamp, so they're always in sync.

Is seamseg ground truth I can evaluate against?

No. seamseg is instance-detection model output in the Mapillary Vistas taxonomy, not a human annotation. Use it to find interesting frames or as a baseline to compare against, not as the labels you score a model against.

Can I reproduce the paper's four benchmark numbers from this build?

Not directly. This build is the 680-frame, 4-scene validation preview meant for exploring the data and building a pipeline. The benchmark numbers in the paper come from the full 1,007-scenario corpus with its dedicated train/val/test/test-e2e splits, including a geo-disjoint held-out test set the paper describes as the first of its kind in online HD map perception.

What's missing from the current release?

Dynamic 3D bounding boxes, object tracks, and instance segmentation for moving agents aren't in this release yet, the source dataset doesn't include them at this stage. seamseg boxes are 2D, camera-space, and model-predicted.

What license governs this build?

CC-BY-NC-4.0, matching the source dataset: non-commercial use, attribution required.

Nine Cameras, Seven LiDARs, Three Radars, One Group ID: Exploring KITScenes Multimodal in FiftyOne

Talk to an AI expert