How to annotate data in FiftyOne: from one sample to 27,000
Sep 10, 2026
• 10 min read
I'm one of the founding machine learning engineers at Voxel51, so I helped kick-start FiftyOne. For a lot of that time I worked primarily on customer success, and more recently I'm also a product manager at Voxel51 leading our annotation efforts. It's a great intersection: I get to talk to our customers, listen to what they're using FiftyOne for and what their pain points are, and then build against that.
In the early days I built a bunch of the annotation integrations we had with other annotation tools. Now we're taking all of that and building annotation natively in FiftyOne. The goal is to tighten the data flywheel so you no longer have that dangling thread of needing to go out to an external annotation tool while you're in the curation and evaluation loop.
This post pulls together three short videos from our FiftyOne Questions series. Each one answers a single question, and together they walk from what FiftyOne is, to what you can annotate in it, to actually annotating a dataset step by step.
Key takeaways
- FiftyOne is a data flywheel platform that tightens the loop of curation, annotation, and evaluation for whatever type of physical AI model you're building.
- In-app annotation in FiftyOne open source covers per-sample annotation across label types, including classifications, bounding boxes, segmentation masks, polylines, and 3D cuboids.
- A label schema in FiftyOne needs, at minimum, a field with a name, a label type, and at least one class. You can add custom attributes to any label, such as a 0 to 1 occlusion slider.
- Click-to-segment with the Segment Anything Model 2 (SAM 2) lets you annotate complex shapes in a couple of clicks, then refine the mask with positive and negative keypoints.
- Voxel51, our enterprise offering, picks up where per-sample annotation stops scaling, with annotation workflows, auto labeling models and agents, and an annotation workforce.
How does visualization work in FiftyOne?
FiftyOne is really about the data flywheel. It's a platform that helps you tighten the loop of curation, annotation, and evaluation for whatever type of physical AI model you're building, and it's agnostic to the specific type of data you're working with.
If you're working on automotive perception, you can have object detections in road scene images. If you're working in manufacturing, you can train defect detection models that detect and segment defects on parts from the line. If you're in the medical domain, you can classify different types of medical imagery. If you’re in the robotics space, you can train policies for different embodiments.
You can also work across modalities. That means not just images but different types of sensors: 3D, time series, images, videos, and any other type of data you're working with. If you're on a robotics task with multiple cameras on your robot, depth maps, 3D points, and multiple joint states, you can bring all of that into FiftyOne for visualization.
It doesn't stop at visualization, though. What makes FiftyOne especially useful is that you bring your entire dataset into the tool and then query it. You can identify interesting subsets by filtering on all kinds of custom metadata you may have. With a click of a button, you can find all of your robot episodes where the robot was folding a cloth in half, jump straight into visualizing that subset or that individual episode, and explore all of the samples, episodes, or clips in your data. You can then use these findings to inform what to annotate or how to train or evaluate the next iteration of your model.
If you have any type of AI system that takes in sensor inputs and tries to do something interesting with them, then FiftyOne is the best building ground for productionizing that system. The companies that perform the best and get their work into production the fastest have the most efficient flywheel. That flywheel lets them take in all of their input data, find the most important subsets to annotate quickly, get data into the hands of annotators, bring in auto labeling models, evaluate the models they train on that data, quickly identify failure modes, and then curate more data to account for the failure modes which kicks off the next turn of the flywheel.
What can I annotate in FiftyOne?
All of the basic visualization features in FiftyOne are open source. If you've got a few images on your laptop, a few robot episodes, some videos, or some images of your manufacturing parts, you can visualize all of that in FiftyOne and curate the kind of small datasets that live locally on your machine.
That extends into annotation. If you're an individual or a hobbyist working on a project, FiftyOne supports annotating across all of the label types we support for visualization.
Take the manufacturing domain. If you're trying to detect a defect, say a little black spot on a part, and you're generating a segmentation mask, you can jump into the Annotate tab for any individual sample and annotate it there. For segmentation, FiftyOne has integrations with SAM so you can click one point and generate a segmentation mask, plus tools to refine it with brushes and polylines and lots of different configurations for the label.
It's not just one label type. If you're working on 3D perception for autonomous vehicles, you might have 3D cuboids. FiftyOne has an entire annotation experience for 3D labels where you can draw boxes around vehicles and automatically project those boxes into image slices.
On top of spatial editing of bounding boxes, segmentations, and polylines, you can also annotate the label schemas of all of these objects. For a lot of use cases, knowing where things are in the world is only half the problem. The other half is knowing what they are and all of the characteristics of the objects or defects you've identified. In FiftyOne you can define very sophisticated label schemas and ontologies: what classes you're annotating, what attributes each class has, and how those attributes get presented to annotators. Vehicles, for example, might have a make attribute stored as a string, and once you add that to the schema you select any vehicle detection and annotate its make as an attribute of the bounding box.
Everything above is per-sample annotation you can do in the open source. It also extends to custom workflows. If you're training a vision-language-action (VLA) model on robot tasks, you're trying to define a policy that takes in visual and sensor inputs and outputs the robot's action. That's a relatively new type of model in the physical AI space, and it comes with different types of annotations. You might have a caption generated by a vision-language model (VLM) as part of the VLA, and you want to correct it.
In the video I'm using a custom plugin I put together to visualize and edit captions a VLM generated for a robot episode. The caption says the outcome was a success: the robot picked up the brick and placed it in the drawer with no collisions or slippage. But I want to note that the robot also opened the drawer, so I update the caption. That lets me refine the VLM and the chain-of-thought reasoning that's part of these VLAs.
There's a lot of built-in annotation functionality in FiftyOne, but where it gets especially exciting is the pluggability. You can build your own annotation workflows. Conceptually, that's possible in the open source at the per-sample or per-episode level. The limit is scale. Me on my laptop going through 100 samples is already going to take a while. When I've got a dataset of 100,000 samples, I can't do that anymore. That's where Voxel51 comes in: bringing in models for auto labeling, including agents, and bringing in an annotation workforce so a team of people can collaborate on annotation workloads.
How do I annotate a dataset in FiftyOne?
Let's get hands on. Say you've loaded a dataset of cell images, some of which are infected with malaria. The goal is to train a model that detects, with some amount of confidence, whether malaria parasites are present, so it could be used to help diagnose the patient the images came from.
Some of the images are uninfected. Others have splotches on the cell that indicate parasites. This is a case where Voxel51 would be useful for setting up an annotation workflow and bringing in expert annotators. But once you get hands on, what does that actually look like?
Step 1: Configure the label schema
The first step is configuring the schema of the annotations you're doing. Click on an image and go to the Annotate tab. As the manager of this dataset, I can configure a schema here: the types of labels we're annotating and the classes and attributes available on those fields.
The dataset already has fields, including some metadata we've computed. I want to add new annotations, so I add a new field. This is where you think about the annotation project and the type of model you're training. You could annotate for a classification model, classifying every image as parasitized or uninfected. Or you could train a bounding box detector or a segmentation model that predicts where parasites are in the image, which could feed more fine-grained classification down the road. All of that is possible in FiftyOne.
For classification, I create a field, set its type to classification, and add the classes: parasitized and uninfected. That's the bare-bones requirement for a label schema: a field with a name, a type, and at least one class. This is also where you configure custom attributes for the labels.
Then I create a second field for the segmentation use case, named segmentation, with type detection. In FiftyOne, detection covers both bounding boxes and instance segmentation, so it's either a box or a box with a mask, and you choose that later on a label-by-label basis. I add a class, parasite, and an attribute, occlusion level, as a floating-point value with a slider UI from 0 to 1. For every segmentation mask, that says how visible the parasite is. You can configure as many attributes as you want and exactly how they show while annotating.
Confirm the attributes and create the field. Now the label schema is ready and both fields are active for annotation. If you want to work on one at a time, you can hide a field, but here I'll keep both active and annotate both the image-level classification and the pixel-level segmentation.
Step 2: Annotate a sample
The sidebar updates once there's an active label schema. There are no labels yet, but there are buttons for each label type you can add.
For the classification, click the button and label the image as parasitized. Done, the classification field is populated.
For the segmentation mask, create a new mask. There are a few tools:
| Tool | What it does | When I reach for it |
|---|---|---|
| Brush | Shade in a region. Configurable size and shape, add or remove from the mask. | Small, circular or square regions. A circular brush works well for parasites. |
| Pen (polygon) | Draw a polygon around a region to create a polygon-based mask. | Fewer clicks than brushing for some shapes. |
| AI annotation | Click a keypoint and run Segment Anything to segment the region automatically. Refine with positive and negative keypoints. | The most efficient option, especially for complex shapes. |
I shade the first parasite with the brush and set its occlusion level to about 0.75, since it's pretty visible. The second one is less visible, so I draw it with the pen and set it closer to 0.5.
The most efficient of all is AI annotation. Click a button and it runs Segment Anything to automatically segment regions of the image. There's a little configuration. You can run SAM 2 tiny, which runs directly in your browser and is available in the open source as well. It's a much smaller model, though, because it’s running on the client hardware. In Voxel51 we have GPU compute backing the deployment, so we can run larger models server side.
Click a keypoint and the image encodes, which takes a couple of seconds the first time. From there, iteratively refine. It got most of the parasite but missed a piece, so I click another positive keypoint. There's still a little pink region around it, so I add a couple more positive keypoints. If it grabs a region I don't want, I click on the existing mask and it places a red keypoint to remove that section. With just a couple of clicks, I've got a pretty complex shape that matches the parasite much more accurately than the quick polygon or brush. Using all of these together is often a good idea: start with SAM 2, then go into the mask and refine the edges the model didn't perfectly get. This one isn't very occluded, so I set a low occlusion value.
Step 3: Repeat
Now I can see all of the classification and segmentation labels on this sample, and I can move through the rest. Uninfected, no masks. Infected: parasite mask, click, done. Two parasites with a small region between them, a quick fix, set occlusion, next sample.
When you have 27,000 samples: annotation at scale in Voxel51
There are 27,000 samples in this dataset. Working through them one at a time in the Annotate tab is fine for getting a feel for the schema and the tools, but it's not something I'd want to do myself, and it's not how a team ships a model.
This is where Voxel51 picks up. Everything I've shown so far is per-sample annotation you can do in the open source. Voxel51 adds support for annotation workflows, so instead of one person clicking through a dataset, you can bring in expert annotators and a team of people collaborating on the same annotation workloads.
It also changes what you can run. In the walkthrough, SAM 2 tiny ran in the browser because that's the compute a laptop has. Voxel51 has GPU compute backing the deployment, which is why I could switch to SAM 2 large and get a mask that matched the parasite much more accurately. That same backing lets you bring in models for auto labeling, including agents for auto labeling, so annotators start from predictions instead of a blank sample.
The point of all of this is the flywheel. Annotate the subset that matters, get labels into a model, evaluate it, find the failure modes, and use that to decide what the next batch of data to annotate should be. Doing that without leaving FiftyOne is the whole reason we're building annotation natively.
FAQ
Per-sample annotation across FiftyOne's label types, including classifications, bounding boxes, segmentation masks, polylines, and 3D cuboids, plus label schema attributes on those labels. You can also build custom annotation plugins, such as editing VLM-generated captions.
At minimum, a field with a name, a label type, and at least one class. You can add custom attributes, such as a floating point occlusion level with a slider UI.
Yes. Click a keypoint and SAM 2 segments the region, then refine with positive and negative keypoints. SAM 2 tiny runs in-browser in the open source. Voxel51 runs larger models such as SAM 2 large on GPU compute.
Open source covers per-sample annotation on datasets that live on your machine. Voxel51 adds annotation workflows, auto labeling models and agents, an annotation workforce, and GPU-backed models.
Loading related posts...
© 2026 Voxel51 All Rights Reserved
