3D LiDAR Annotation: Creating Bounding Boxes in Point Cloud Datasets with FiftyOne

As LiDAR and point cloud datasets become increasingly important in autonomous driving, robotics, and spatial AI, accurate 3D bounding box annotation workflows are essential for building high-quality computer vision systems. Unlike 2D annotations, 3D object representations must capture spatial position, orientation, and physical dimensions within a three-dimensional coordinate system. This additional depth information enables more precise scene understanding and object localization across complex environments.

In this blog, you’ll learn the fundamentals of creating and visualizing 3D object detections in the open source version of FiftyOne using point cloud data. We’ll walk through how to construct a point cloud bounding box with location, rotation, and dimension parameters, manipulate detections in 3D space, and explore visualization techniques for 3D lidar annotation workflows. Whether you are evaluating autonomous vehicle datasets or experimenting with a 3D bounding box annotation tool, this guide provides a practical introduction to working with LiDAR-based annotations in FiftyOne.

3D detection can mean a variety of things in computer vision depending on the context. In this blog we are looking at 3D detections as they pertain to 3D LiDAR annotation or point cloud spaces. These kinds of detections are common in automotive and topography datasets and leverage the insightful knowledge of the different depths of the environment.

However, not all objects in real-world point clouds are simple or “clean” geometric shapes. In addition to standard vehicles and pedestrians, datasets often include uncommon 3D objects such as construction equipment, traffic barriers, utility poles, debris, trailers, and irregularly shaped machinery. These objects can be partially occluded, unevenly distributed in point clouds, or difficult to align with a perfect cuboid representation.

In these cases, 3D bounding box annotation becomes more nuanced. Annotators must rely more heavily on accurate estimation of object orientation, center positioning, and approximate physical dimensions rather than relying on idealized shapes. This is especially important in 3D LiDAR annotation workflows, where point cloud density and sensor perspective can significantly affect how clearly an object is represented.

In FiftyOne’s open source software, detections are a single label for either 2D or 3D, it is how we define them that makes the difference. Let's start with 2D and see how you can build a detection label.

You can see that all you need is the label and the bounding box to construct your label. In 3D object detections, you also provide a label, but instead of a 2D bounding box, you now provide the details in order to build a 3D bounding box, accounting for the extra dimension.

3D bounding box detections are defined in FiftyOne with three input parameters: location, rotation, and dimensions. Location refers to the location of the bounding box on the point cloud coordinate system. The location is the absolute center of the bounding box. Rotation is the amount the bounding box is rotated among its axes. Rotation is also given as [x, y, z] where x is the rotation around the x axis in a space of [-pi, pi]. Finally, dimensions are the width, length, and height of the bounding box.

Lets try making your first 3D bounding box.

You start by loading in a dataset that has point clouds as a part of it. quickstart-groups is a subset of a KITTI formatted dataset. Feel free to explore the point clouds by clicking on one of the groups and looking at the existing detections.

Next, you can add a very basic bounding box at the center of your point cloud. Switch to the pcd group slice to add the detection to the point cloud, then add the detection. You create a view at the end to show only the sample you’re interested in.

Below you can see the results.

With your first box placed, you can play around with some of the input parameters to get comfortable with the FiftyOne 3D visualizer. Lets modify the location as well as the wlh to get more of a grasp of the coordinate system.

You can see how the box has transformed and shifted along the coordinate plane. Feel free to tweak these numbers until you get a comfortable feel. A good foundation of the coordinate system will help troubleshoot any potential problems in the future.

Rotating the bounding box is simple, for each given axis, define in radians how much you want your box to rotate around the circle. Play around with the tweaking rotation numbers to get a handle on situating your boxes.

Pro Tip: Save rotation for last if you are trying to align your boxes. Different standards exist for 3D detections, so be confident you aren't accidentally rotating too much that now your width and length have flipped.

Similar to the last example, you can define polygons with a set of points to create a 3D polyline. To create a 3D polyline, all you need to do is define the points on the grid in a list. To close a shape, specify the first vertice again. You can add these lines to your sample and display your new shape.

Ending with a final tool you can use when working with 3D data is Orthographic Projections. You have probably noticed when running the app that the LiDAR samples have no thumbnails. You can compute orthographic projections of our LiDAR space to create thumbnails and add a new slice to our dataset to curate by. This will give you a birds eye view of your data to look through with tons of options to change coloring or rendering.

Effective 3D bounding box annotation is foundational for modern perception systems that rely on LiDAR and point cloud data. By accurately defining object location, dimensions, and orientation in 3D space, you can build richer datasets for applications including autonomous driving, robotics, mapping, and augmented reality. Understanding how to create and manipulate a point cloud bounding box also improves dataset validation, model debugging, and annotation quality assurance workflows.

With FiftyOne’s open source software, your team can streamline lidar annotation workflows through interactive visualization, 3D polyline support, and orthographic projections for easier dataset curation. As a flexible 3D bounding box annotation tool, FiftyOne enables you to inspect, refine, and manage complex 3D datasets more efficiently. By mastering the core concepts covered in this tutorial, you’ll be better prepared to build scalable and accurate 3D lidar annotation pipelines for real-world computer vision applications. If you're looking to scale to production-grade, collaborative, cloud-native enterprise workloads, check out FiftyOne Enterprise.