What is a multimodal embedding?
A multimodal embedding lives in a space that several data types share. A model like CLIP embeds images and text into the same coordinates, trained so a photo of a dog and the sentence "a photo of a dog" land close together. Newer models extend the shared space to video, audio, and sensor data.
The shared space is what makes cross-modal search possible: a text query becomes a vector, and its nearest image vectors are the search results.
Key takeaways
- Multimodal embeddings place different data types in one shared space.
- Matching content across modalities lands close together.
- They enable text-to-image search and cross-modal comparison.
How it works
A separate encoder per modality maps each input type into the common space, and contrastive training on paired data, such as images with captions, pulls matching pairs together across modalities. Once trained, any modality can query any other: text finds images, images find video moments, and combinations can be averaged or filtered together.
Why it matters
Multimodal embeddings turned natural language into a universal interface for visual data, letting anyone search millions of images by describing what they want. For physical AI, extending the shared space to sensor streams and robot data promises the same leap: querying multimodal logs with plain language instead of hand-built filters.
Frequently asked questions
What models produce multimodal embeddings?
CLIP-family vision-language models are the best known, with newer multimodal embedding models covering video, audio, and documents in one space.
How is a multimodal embedding different from an image embedding?
An image embedding only encodes images. A multimodal embedding shares its space with other modalities, so text and images can be compared directly.
Related terms
Go deeper