Foundation model labeling is auto-labeling that uses large, general-purpose pretrained models, such as SAM, CLIP, or vision-language models, to annotate data, often from a text or visual prompt and without any task-specific training. It brings broad, off-the-shelf knowledge to your labeling.
Foundation models are large networks pretrained on huge, diverse datasets, so they carry broad visual and semantic knowledge out of the box. Foundation model labeling puts that knowledge to work generating annotations: you prompt a model, with a click, a box, or a text phrase like "label every forklift," and it produces labels without being trained on your specific task. SAM produces a mask from a click, vision-language models classify or detect from a text prompt, and grounding models localize objects named in language. The result is labeling that starts from general competence rather than a model you had to train first.
Key takeaways
It uses large pretrained models, SAM, CLIP, VLMs, grounding models, to label, often zero-shot.
No task-specific training is required, the model's general knowledge does the work.
It is strongest on common concepts and weakest on niche or domain-specific classes the foundation model never saw well.
Foundation models used for labeling
The main kinds of foundation model used to label visual data, and what each produces.
The main kinds of foundation model used to label visual data, and what each produces.
Model type
What it does
Examples
Promptable segmentation
Masks an object from a click, box, or point
SAM, SAM 2, SAM 3
Classification and retrieval
Matches images to text
CLIP-style models
Open-vocabulary detection
Localizes objects named in a text prompt
Grounding models
Vision-language models
Describe, tag, or answer questions about an image
VLMs
How it works, and how FiftyOne fits
You point a foundation model at your data with a prompt, and its outputs become candidate labels for review. In FiftyOne, you apply foundation models from the model zoo or your own, store their predictions, and curate which are reliable enough to keep, sending the uncertain ones to humans.
Why it matters
Foundation model labeling collapses the old chicken-and-egg problem: you no longer need labeled data to train a labeler before you can label. The catch is the domain gap. Foundation models are confident and fluent on common, web-like imagery and quietly unreliable on specialized domains, medical scans, industrial defects, sensor data, that were underrepresented in pretraining. So they are a fast first pass, not a final answer, and the rarer or more specialized your data, the more human verification you need.
Frequently asked questions
What models are used for foundation model labeling?
SAM for masks, CLIP for classification and retrieval, grounding models for open-vocabulary detection, and vision-language models for tagging and description.
Is foundation model labeling the same as zero-shot labeling?
Closely related, foundation models are what make zero-shot labeling possible, labeling classes they were never explicitly trained on.
When does it work less well?
On specialized domains underrepresented in pretraining, where the model is confidently wrong.