EmbeddingGemma 2 in FiftyOne: Search Video by Text, Sound, or Your Webcam

Key takeaways

• EmbeddingGemma 2 (740M parameters, Apache 2.0) embeds text, images, video, and audio into one 768-dimensional space, and its Matryoshka training lets you truncate vectors to 512, 256, or 128 dimensions.
• In FiftyOne, EmbeddingGemma 2 is a remote zoo model: register it once, then model.embed_dataset() fills one vector field per modality.
• With EmbeddingGemma 2, VGGSound ends up with three vector fields (eg2_video, eg2_audio, eg2_av) and three LanceDB similarity indexes built with fob.compute_similarity(..., backend="lancedb").
• Because EmbeddingGemma 2 puts every modality in one space, a text query can search all three indexes, and a clip’s soundtrack can search other clips’ frames.
• The same EmbeddingGemma 2 vectors drive uniqueness, representativeness, UMAP layouts, and zero-shot classification of VGGSound’s 309 sound classes.
• The Cross-Modal Retrieval panel installs with two fiftyone plugin commands and lets you search by text, selected samples, or media you upload or record. You can also try it in a Hugging Face Space without installing anything.

What is EmbeddingGemma 2?

1. One space. You can compare any vector to any other, so cosine similarity works between a sentence and a sound.
2. Matryoshka dimensions. The first 512, 256, or 128 numbers of a vector are a usable embedding on their own, so you can use smaller indexes without retraining anything.
3. An 8K-token context. Per the launch post, that fits up to 5.5 minutes of audio, 29 images, or 58 video frames in a single input.

What can you do with EmbeddingGemma 2?

• Find clips by describing them. “Rain on a window” against a pile of unlabeled video.
• Search across modalities. A sound finds pictures, and a picture finds sounds. In a grouped dataset, a video’s vector can pull the music that fits.
• Pick what to label or drop. Uniqueness and representativeness scores computed from the vectors rank clips by how redundant or typical they are.
• Label without training. Compare each clip’s vector to the text of your class names for zero-shot classification.
• Search with your own media. Record a clap, a photo, or a few seconds of video and use it as the query.

The dataset: VGGSound

The VGGSound dataset in the FiftyOne App, with label counts, uniqueness and representativeness scores, and an embeddings view.

Step 1: Register the model

import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
from fiftyone import ViewField as F

# One-time: point FiftyOne at the remote zoo model source
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/EmbeddingGemma2", overwrite=True
)

# Load the vision and audio encoders so one model handles video, audio, and both
model = foz.load_zoo_model(
    "google/embeddinggemma-2",
    media_type="video",
    modalities=["vision", "audio"],
)

Step 2: Load VGGSound

from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("harpreetsahota/VGGSound")

Step 3: Embed the clips three ways

The three EmbeddingGemma 2 vector fields on VGGSound, explored as UMAP layouts in the Embeddings panel.

# Frames only: sampled at 1 fps, up to 32 frames spread over the clip
model.embed_dataset(dataset, "eg2_video", modality="video")

# Soundtrack only: the first 30 seconds, resampled to 16 kHz mono
model.embed_dataset(dataset, "eg2_audio", modality="audio")

# Frames and soundtrack together, as one vector
model.embed_dataset(dataset, "eg2_av", modality="video+audio")

Step 4: Build a LanceDB index for each field

index_settings = {
    "eg2_video": {"media_type": "video"},
    "eg2_audio": {"media_type": "audio"},
    "eg2_av": {"media_type": "video", "modalities": ["vision", "audio"]},
}

for key, model_kwargs in index_settings.items():
    fob.compute_similarity(
        dataset,
        embeddings=key,                   # reuse the stored vectors
        model="google/embeddinggemma-2",  # pass the zoo model name so text queries work
        model_kwargs=model_kwargs,
        backend="lancedb",
        brain_key=key,
    )

{
    "similarity_backends": {
        "lancedb": {"uri": "~/fiftyone/lancedb"}
    }
}

Why LanceDB for a multimodal index?

• Embedded, no server. It runs inside the FiftyOne process and stores tables as files, on local disk or in cloud storage.
• Built for ML data. Tables use the Lance columnar format, designed for fast random access to large vectors.
• Scales past memory. Tables live on disk, so an index doesn’t have to fit in RAM.
• One table per index. A video, an audio, and a video + audio index of the same clips don’t interfere with each other.

Step 5: Search with text

Text queries in the Cross-Modal Retrieval panel return matches from the video, audio, and video + audio indexes at once.

view = dataset.sort_by_similarity("church bell ringing", k=10, brain_key="eg2_av")

session = fo.launch_app(view)

Step 6: Let a sound search videos

Audio and video queries in the Cross-Modal Retrieval panel find VGGSound clips by what they show and how they sound.

# A clip labeled "female singing"
clip = dataset.match(F("ground_truth.label") == "female singing").first()

# Its soundtrack vector queries the frames-only index; the clip itself is excluded
matches = dataset.exclude([clip.id]).sort_by_similarity(
    clip["eg2_audio"], k=10, brain_key="eg2_video"
)

Step 7: Find the clips worth your time

# Uniqueness: high scores mark clips with few close neighbors
fob.compute_uniqueness(
    dataset, embeddings="eg2_av", uniqueness_field="eg2_av_uniqueness"
)

# Representativeness: high scores mark clips near the center of a cluster
fob.compute_representativeness(
    dataset,
    embeddings="eg2_av",
    representativeness_field="eg2_av_representativeness",
    method="cluster-center",
)

# A 2D layout you can explore in the Embeddings panel (needs umap-learn)
fob.compute_visualization(
    dataset, embeddings="eg2_av", method="umap", brain_key="eg2_av_viz", seed=51
)

Step 8: Classify without training anything

# Use the 309 VGGSound class names as the label set
model.classes = dataset.distinct("ground_truth.label")

# Compare stored vectors to the class text; no media is decoded again
model.classify_dataset(dataset, "eg2_zeroshot", embeddings_field="eg2_av")

# Score the zero-shot labels against ground truth
dataset.evaluate_classifications(
    "eg2_zeroshot", gt_field="ground_truth", eval_key="eg2_eval"
)

Try it without code: the Cross-Modal Retrieval panel

Install

fiftyone plugins download https://github.com/harpreetsahota204/cross_modal_similarity_search
fiftyone plugins requirements @harpreetsahota/cross-modal-retrieval --install

• FiftyOne 1.22.0 or later
• lancedb, transformers>=5.18, torch, av, and huggingface-hub, all installed by the second command
• The EmbeddingGemma 2 weights in your model zoo. Download them once (about 1.5 GB) so your first search doesn’t stall:

import fiftyone.zoo as foz

foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/EmbeddingGemma2", overwrite=True
)
foz.download_zoo_model("google/embeddinggemma-2")

Don’t want to install anything?

Open the panel and build an index

The Build Cross-Modal Index form, where you choose what part of each sample becomes its EmbeddingGemma 2 vector.

Search with text

Text searches in the Cross-Modal Retrieval panel, with a colored chip for each index's results.

Search with other samples

Searching with selected samples, which the panel averages into one query.

Search with your own media

• Upload an image, video, audio, or MCAP file.
• The microphone records up to 30 seconds of audio.
• The webcam records up to 30 seconds of video with sound or takes a single photo.

A microphone recording searches the VGGSound clips directly, without being added to the dataset.

Tune the search and keep your results

Saved searches on the Cross-Modal Retrieval panel's home page, ready to reopen, clone, or delete.

Keep it honest

• This is one shard of VGGSound. Everything here uses 10,000 of the 199,467 clips.
• Read scores as a ranking. Text-to-clip similarity sits in a narrow band, and matching clips score only slightly higher than the rest. Compare scores within one search, not across searches.
• Audio-only is the weakest index on VGGSound. The class names sound sources that are visible in the frames, so the video and video + audio indexes find them more reliably. Audio alone does better when the classes are defined by sound.
• 128 dimensions costs quality. The model card reports a large drop for multimodal tasks at 128. Use 768 or 512 unless storage forces your hand.
• Retrieval figures in this post come from a 100-clip sample. The church bell and sound-to-frames charts illustrate behavior and aren’t benchmark results.

Try it yourself

• EmbeddingGemma 2 in FiftyOne Space: the panel on VGGSound, no install
• Cross-Modal Retrieval plugin: the panel and operators
• EmbeddingGemma 2 FiftyOne integration: the remote zoo model, with a notebook
• VGGSound on Hugging Face: 10,000 clips with vectors, uniqueness, and representativeness scores
• EmbeddingGemma 2 model card and Google’s launch post
• FiftyOne Brain similarity docs