Image embeddings
An image embedding is a compact vector representation of an image produced by a neural network, positioning visually or semantically similar images near each other in a high-dimensional space. It is the backbone of similarity search, clustering, deduplication, and data curation.
In this article
Building visual or physical AI?
Let's talk.
What are image embeddings?
An image embedding is what you get when a neural network reduces an image to a fixed-length list of numbers, a vector, that captures its content rather than its raw pixels. Models trained on large image collections, a pretrained CNN, a vision transformer, or a multimodal model like CLIP, learn to place similar images close together in this vector space and dissimilar ones far apart. That single property, distance equals similarity, is what makes embeddings so useful: you can search, group, and compare images by meaning instead of by file. Crucially, embeddings work on unlabeled data, so they are how teams make sense of a raw dataset before a single annotation exists.
Key takeaways
- An embedding is a vector that represents an image's content, with similar images landing near each other.
- Embeddings power similarity search, clustering, deduplication, outlier detection, and curation, all without labels.
- Their quality depends on the model that produced them, an embedding is only as good as the model's fit to your domain.
How embeddings are produced
- CNN or vision-transformer features: the activations from a pretrained vision model.
- Multimodal embeddings: CLIP-style models that put images and text in one shared space, so you can search images with words.
- Task-tuned embeddings: a model fine-tuned on your domain, which separates your classes more cleanly than a generic one.
How it works
You run a model over your images, store the resulting vectors, and then reduce them to 2D to see the structure of your data. That turns an undifferentiated pile of images into a map you can navigate.
What an embedding plot shows, and what it hides
A 2D embedding plot is the fastest way to see the shape of an unlabeled dataset. It's also a projection. Methods like UMAP and t-SNE squeeze hundreds of dimensions into two, which keeps local neighborhoods mostly intact and can distort the rest. Distances between clusters, cluster sizes, and the empty space between them often mean very little.
Read the plot as a map of where to look, not as a measurement:
- Use clusters to find groups, then open the samples to see what they actually share. On satellite imagery that's often location, season, or viewing angle, not the objects you care about.
- Use isolated points to find outliers, like corrupted tiles or rare scenes, and confirm them by eye.
- Use similarity search for the real question. It runs on the full embeddings, so "find more like this" isn't affected by the projection.
That first point matters for your splits. If tiles from the same location cluster tightly and land in both train and test, your evaluation will look better than the real world will be.
RarePlanes 5k
5,000 images
WorldView-3 satellite tiles of 112 airfields with aircraft boxes, outlines, and ten attributes per plane.
Try this: Open the embeddings plot and check whether the clusters follow the aircraft or the airfields, then run a similarity search from one tile to see what the model treats as its nearest neighbors.
Open RarePlanes in Voxel51
Free with a work or .edu email.
Go deeper
Image embedding vs related representations
| Representation | What it is | Typical use |
|---|---|---|
| Image embedding | A compact vector of the whole image's content | Similarity, clustering, curation |
| Feature map | The spatial activations inside a network | Detection and segmentation heads |
| Logits / prediction | The model's class scores for an image | Classification output |
Why it matters
Embeddings are the engine of data curation, which is where most model improvement actually comes from. An embedding inherits the blind spots of the model that produced it, and this quietly breaks curation when the model is a poor fit. A generic ImageNet embedding run over medical scans or LiDAR-derived imagery will cluster by irrelevant surface features, so "find similar" and "select diverse" return the wrong samples, and the curation looks principled while selecting noise. The highest-leverage decision is not the search interface, it is choosing an embedding model close enough to your domain that distance in the space means what you think it means.
Frequently asked questions
What is an image embedding?
A vector representation of an image's content, where similar images sit close together in the vector space.
What are image embeddings used for?
Similarity search, clustering, deduplication, finding outliers, and curating which data to label, all on unlabeled images.
How are image embeddings created?
By running images through a pretrained or fine-tuned vision model, sometimes a multimodal model, and keeping the resulting vectors.
Related terms
Learn more
Compare embedding models head to head in Finding the Best Embedding Model for Image Classification, and use embeddings to mine rare cases in Few-Shot Image Retrieval in FiftyOne.
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 3, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved