A dataset is an organized collection of examples, such as images, video, or point clouds together with their labels and metadata, that a machine learning model is trained, validated, and tested on. It is the unit teams curate, annotate, and evaluate against throughout the pipeline.
A dataset is the raw material of supervised learning: the examples plus the labels a model learns to predict, usually carrying metadata like source, capture conditions, and which split each example belongs to. It is conventionally divided into training, validation, and test sets so you can fit a model, tune it, and then measure it honestly on data it has never seen. In visual AI the samples are images, video, or point clouds and the labels are boxes, masks, classes, or keypoints, and standard formats like COCO, YOLO, and KITTI define how those samples and labels are stored and shared.
Key takeaways
A dataset is examples plus labels plus metadata, the input to both training and evaluation.
It is usually split into training, validation, and test sets to fit, tune, and measure a model without leakage.
In visual AI the samples are images, video, and point clouds, and formats like COCO and YOLO standardize how labels are stored.
Parts and splits
Samples: the raw items, such as images, frames, or point clouds.
Labels: the annotations the model learns to predict.
Metadata: source, capture conditions, splits, and attributes you can slice by.
Splits: training to fit, validation to tune, and a held-out test set for the final measure.
How it works, and how FiftyOne fits
The dataset is FiftyOne's core object: it holds your samples, labels, and metadata in one place, so you can query, slice, visualize, and evaluate them, and import or export the standard formats. A one-line entry point is import fiftyone as fo, then load or build a fo.Dataset from there. See it in practice in the FiftyOne dataset quickstart in the docs.
Dataset vs related terms
Definitions for dataset and its related terms
Definitions for dataset and its related terms
Term
What it is
Dataset
The full collection of samples, labels, and metadata
Sample
A single example within the dataset
Split
A partition of the dataset, such as train, validation, or test
Benchmark
A standard dataset used to compare models
Why it matters
A dataset is not neutral, its composition is a series of decisions that becomes the model's entire worldview. Information-gain insight: the most common and most damaging dataset problem is leakage between splits. Near-duplicate frames, which are everywhere in video, landing in both the training and test sets make a model look far more accurate than it is, because it is being tested on data it effectively saw in training. So the dataset quietly lies to you about performance. Class imbalance and coverage gaps do the same in reverse, becoming the model's blind spots. How a dataset is split and curated matters as much as how big it is.
Frequently asked questions
What is a dataset in machine learning?
An organized collection of examples with their labels and metadata, used to train and evaluate a model.
What are training, validation, and test splits?
Partitions of a dataset used to fit the model, tune it, and then measure it on data it has not seen.
What formats are visual datasets stored in?
Common ones include COCO, YOLO, and KITTI, which standardize how samples and labels are recorded.