Embedding and semantic search
An embedding is a numerical representation that places data such as images, text, or sensor readings as points in a vector space, so that similar items sit close together. Semantic search uses these embeddings to find data by meaning rather than by exact keywords or metadata, retrieving items that are conceptually similar to a query. Together they make it possible to explore and organize large, unstructured datasets by content.
In this article
Building visual or physical AI?
Let's talk.
What is embedding and semantic search?
An embedding is a way of turning a piece of data into a list of numbers, a vector, that captures its meaning. The trick is that the numbers are arranged so that similar things end up near each other in the resulting space. Two photos of the same kind of scene, or two sentences that express the same idea, will have embeddings that sit close together, while unrelated items will be far apart. This lets a computer measure how similar two things are just by measuring the distance between their vectors, even for messy, unstructured data like images and text.
Semantic search is the natural application of this idea. Instead of matching exact words or relying on manually assigned tags, semantic search embeds a query and then finds the items whose embeddings are closest to it, returning results that are similar in meaning. Ask for "a dog running on a beach" and semantic search can surface matching images even if none of them were ever labeled with those words. This shift from matching symbols to matching meaning is what makes it possible to search and organize large collections of data by what the data actually contains.
Key takeaways
- An embedding represents data as a vector, positioned so that similar items are close together in the space.
- Semantic search uses embeddings to retrieve data by meaning rather than by exact keywords or tags.
- Together they make large, unstructured datasets explorable and organizable by content, not just by metadata.
How it works
Embeddings are produced by models trained so that related inputs map to nearby vectors and unrelated inputs map to distant ones. Once data is embedded, each item becomes a point in a high-dimensional space, and similarity is measured by how close two points are. Semantic search embeds the query the same way and then looks for the nearest points, often using specialized indexes that can find nearest neighbors quickly even across millions of items. Because a shared embedding space can be built across modalities, it is possible to search images with text, or find examples similar to a given image, which is what makes embeddings so useful for exploring multimodal data.
When semantic search needs a second filter
Semantic search is good at fuzzy, visual questions. "Cyclist at dusk" returns cyclists in low light even if no image was ever tagged "dusk." It's weaker at exact ones. Embeddings don't reliably encode counts, small details, or facts that live in metadata, like a timestamp, a sensor ID, or whether your model got the frame right. Ask for "three pedestrians crossing" and you'll get crossings, with however many people happened to be there.
Real data questions usually mix both kinds. "Low-light frames with a cyclist the detector missed" is one fuzzy condition and one exact one. That's why production search pairs vector similarity with structured filters.
Voxel51 Search combines semantic retrieval with lexical search, metadata filters, and evaluation results on the same dataset. You can start from a natural-language query, then narrow by labels, fields, or model errors, without exporting anything.
PandaSet Detection
9,888 images
PandaSet driving camera frames with ground truth boxes and an evaluated RT-DETR detector.
Try this: Search for "cyclist at dusk," then filter the results to frames where the evaluated detector missed an object.
Open PandaSet in Voxel51
Free with a work or .edu email.
Go deeper
Why it matters
Embeddings and semantic search turn a large, unlabeled pile of data into something you can actually navigate by meaning, which is transformative for working with real-world datasets. For physical AI, where datasets are enormous and mostly unlabeled, this is a key enabler of curation, letting practitioners find similar scenes, surface rare cases, and understand what their data contains. Anyone trying to make sense of large multimodal collections benefits, because searching by content is often the only practical way to find the specific examples that matter.
For how search works on robot recordings, such as asking for episodes where the gripper drops the object instead of writing a filter, see natural-language search across robot recordings.
Frequently asked questions
What exactly is an embedding?
An embedding is a numerical vector that represents a piece of data in a way that captures its meaning, arranged so that similar items have nearby vectors. It lets a computer reason about similarity for unstructured data like images and text by comparing distances in the vector space.
How is semantic search different from keyword search?
Keyword search matches exact words or tags, so it misses relevant results that use different wording or were never labeled. Semantic search matches meaning by comparing embeddings, so it can find conceptually similar items even when the exact words do not appear.
How does this help with curating data?
Because similar items sit close in embedding space, you can find scenes like a known example, surface unusual or rare cases, and organize data by content. This makes it far easier to explore large, mostly unlabeled datasets and to target the examples worth labeling or investigating.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 11, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved