Coreset selection is the process of finding a small, representative subset of a dataset that trains a model to near-full-dataset accuracy at a fraction of the cost. Until now, state-of-the-art coreset methods required labeled data. ZCore (Zero-Shot Coreset Selection), an open source method from Voxel51's research team, selects valuable data with no labels, no dataset-specific training, and no domain expertise. In our tests on robot data, it cut dataset volume 95% while covering every setting in the original dataset.
Zero-Shot Coreset Selection for Robotics and Visual AI Development
I’m amazed by the massive amount of data deployed vision systems can generate. Two robots I recently worked with, HSR and Digit, each have four or more cameras that can collect data at 15 Hz or faster. Using back-of-the-envelope calculations, you realize that a single robot (not even the fleet) can collect ImageNet-scale data in just a few hours and Florence-2-scale data in a few weeks (1.3M and 126M images, respectively). What’s more, all of that data is relevant to the environment and tasks that the robot is performing and can be used to improve its own perception capabilities. In an age of data-hungry visual AI, “data is the new oil,” but robot and AV systems are its renewable source.
Key Takeaways
ZCore (Zero-Shot Coreset Selection) selects the most valuable subset of a visual dataset without labels, dataset-specific training, or domain expertise.
Coreset selection cuts annotation and compute costs by training models on a small representative subset instead of the full dataset.
On ImageNet, ZCore coreset-trained models outperform previous label-based coresets at a 10% data rate, removing annotation costs for 1.15 million images.
On robot data from HSR, ZCore found a 95% reduction that still covered every setting in the full dataset.
ZCore is open source and runs on any image folder with a single command using FiftyOne.
"Data is the new oil" — Clive Robert Humby, Mathematician & Entrepreneur
Notably, refining and operationalizing all of this data is cost-prohibitive. The computational cost to train a single state-of-the-art deep learning model in various fields doubles every 3.4 months due to increasingly large models and datasets, requiring as much as hundreds of thousands of GPU hours. Also, despite progress in weak- and self-supervision, many production models train using full supervision on labeled data, adding annotation costs. But we can’t possibly label ImageNet-scale data every few hours on a per-robot basis (although annotation service providers would love that contract).
Rest assured, we can “tame” this influx of data. Typically, domain experts determine which data are most valuable, but this takes effort away from other development (yet another cost). Alternatively, coreset selection algorithms find a representative subset of data to train a model with lower cost and minimal impact on performance. However, previous state-of-the-art coreset methods can only select data that were pre-labeled. Voxel51 recently announced the open source solution, Zero-Shot Coreset Selection (ZCore), which selects valuable data without any labels, data-specific training, or domain expertise. ZCore offers an efficient, automated solution to sift through massive data to find hidden gems.
How ZCore Selection Works (Try it!)
State-of-the-art coreset methods use carefully designed criteria to quantify the importance of each data example using ground-truth labels and dataset-specific training (e.g., gradient dynamics from training on all of the data). Alternatively, ZCore uses existing foundation models to generate a zero-shot embedding space for unlabeled data and quantifies the relative importance of each example based on overall coverage and redundancy within the embedding distribution (see our paper for full details). On ImageNet, ZCore coreset-trained models are more accurate than previous label-based coresets at a 10% data rate, effectively removing annotation costs for 1.15 million images.
ZCore removes the label, training, and expertise requirements of previous state-of-the-art coreset selection methods.
ZCore removes the label, training, and expertise requirements of previous state-of-the-art coreset selection methods.
Requirement
Previous coreset methods
ZCore
Ground-truth labels
Required
Not needed
Dataset-specific training
Required (e.g., gradient dynamics)
Not needed
Domain expertise
Often required
Not needed
ImageNet at 10% data rate
Lower accuracy
Higher accuracy, zero annotation cost for 1.15M images
Our paper provides many results on image-based datasets, but does not explore datasets from robots. So, let's do that here. We provide a ZCore demo below using HSR data from the lab combined with ODMD, ODMS, and TFOD Benchmark examples. To demo along on your data, run this single command from our repo:
This customizable script handles the entire embedding generation, ZCore selection, and the FiftyOne visualization process we use here.
In the demo above, image data varies across multiple robot cameras, environments, and tasks. In the embeddings view, clusters of images indicate similar or redundant data samples. On the other hand, each cluster represents a unique concept or setting in the broader dataset (i.e., objects, environments, and or viewpoints).
By running ZCore on this data, the individual value of each image within the context of the entire dataset is determined automatically. For example, the HSR grasp camera sequence approaching the pan is unique relative to the rest of the dataset, but only two images are needed to represent such a simple concept. Thus, the best image in that cluster is given a high ZCore score, the second best image is found at the opposite end of the sequence, and the remaining, redundant data have a low score. Across the entire robot dataset, ZCore automatically makes similar determinations without any labels, training, or domain expertise. Notably, in experiments using several public datasets, we find that ZCore selections lead to reliable model performance across a variety of dataset scales and applications (e.g., satellite image classification).
ZCore on Robot Data: A 95% Reduction
We threshold the ZCore score across all data to determine which images are included in our core set of data – either for model training, fine-tuning, or evaluation. In the demo above, we show threshold results for the top 1, 2, 4, 20, and 85 images. Notably, the top 85 ZCore images cover the dataset’s unique concepts without redundancy. Quickly optimizing the data volume means less manual review, lower annotation effort, less compute for model training or evaluation, and huge cost savings as a result.
Summary
Robotics, AVs, and other deployment platforms provide a renewable data supply for visual AI development. However, the sheer volume of data produced by these systems makes it cost-prohibitive to use all of it. Our research team at Voxel51 developed a tool to automatically select valuable subsets of data to efficiently manage scale without any data labels, training, or domain expertise. Using the ZCore technique, we quickly found a 95% reduction in robot data that covered all the settings of the initial, full dataset.
This tool is open source, available on GitHub, and is also available in the Enterprise version of FiftyOne soon. Give it a try and let us know what you think: https://github.com/voxel51/zcore.
Acknowledgments
Thank you to Jason Corso, Kirti Joshi, Manushree Gangwar, and Michelle Brinich for their feedback and suggestions on this blog!
Questions? We have answers.
Coreset selection finds a small, representative subset of a dataset that trains a model with minimal impact on performance and much lower cost. Previous state-of-the-art methods could only select from pre-labeled data.
ZCore uses foundation models to generate a zero-shot embedding space for unlabeled data, then scores each example by coverage and redundancy within the embedding distribution. No labels, dataset-specific training, or domain expertise required.
On robot data, ZCore found a 95% reduction that covered all settings in the full dataset. On ImageNet, ZCore-trained models beat label-based coresets using only 10% of the data.
Yes. The code is on GitHub at voxel51/zcore, and you can run it on your own image folder with a single script after installing FiftyOne.
