Can an AI trained on pop songs and podcasts hear the tree of life in whale calls? Turns out: pretty much, yes. We’ve built a Jupyter Notebook that lets you hear it AND see it.
The science behind whale song embeddings
It starts with a preprint — "Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models" (Rincón Yepes, 2026) — and its companion repo. The research found that general-purpose audio models like CLAP and AST encode the evolutionary relationships between species about as well as a bioacoustics specialist does. And curiously, even though nobody trained them on animals, phylogeny, or anything of the sort.
Key takeaways
Whale song embeddings from general-purpose audio models like CLAP and AST encode evolutionary relationships between species about as well as a bioacoustics specialist model does (Rincón Yepes, 2026).
The demo embeds roughly 1,700 clips across 32 species from the Watkins Marine Mammal Sound Database four ways: CLAP, AST, BEATs-bio, and an MFCC baseline.
FiftyOne's embeddings visualizer projects the audio vectors to a 2-D plot where clades and families visibly separate when colored by taxonomy.
The fiftyone-audio-toolkit plugin renders every audio clip as a live, in-browser spectrogram with playback in the FiftyOne App.
CLAP supports text-to-audio search in FiftyOne, so typing "barking seal" surfaces matching clips with no example audio required.
The sounds come from the Watkins Marine Mammal Sound Database (Woods Hole Oceanographic Institution) — a historic archive spanning seven decades, including the first-ever underwater recording of a marine mammal, a beluga taped in 1949. The demo uses a clean subset: ~1,700 clips across 32 species. Each clip is embedded four ways — CLAP, AST, the specialist BEATs-bio, and a hand-crafted MFCC baseline — so you can compare what each model "hears."
Pairing it with this project is what makes an abstract statistic tangible. The paper reports a correlation coefficient; FiftyOne lets you watch the family tree assemble itself in a scatter plot, flip between models to see which ones "get it," and click a dot to hear the actual animal. A number becomes an experience.
See it, hear it, search it
We built a demo notebook that wires everything using open-source FiftyOne and the fiftyone-audio-toolkit plugin:
Spectrograms in the grid. Every clip renders as a live, in-browser spectrogram.
Play it in the app. Double-click a tile and a player opens with a playhead sweeping across the spectrogram. Listen to a killer whale, then a bearded seal.
Watch models disagree. Flip the embeddings plot between CLAP, AST, BEATs-bio, and MFCC. Color by clade and the whale/seal split appears; switch to the cetaceans and color by family; dolphins, baleen whales, and belugas pull cleanly apart. MFCC? Mush. That contrast is the paper's point.
The four embedding models in the demo and what each reveals about the Watkins marine mammal clips.
The four embedding models in the demo and what each reveals about the Watkins marine mammal clips.
Embedding
What it is
What you see in the plot
CLAP
General-purpose audio-language model
Clade and family structure separates, also powers text search
AST
General-purpose audio transformer
Comparable evolutionary structure
BEATs-bio
Bioacoustics specialist
The specialist reference the generalists match
MFCC
Hand-crafted acoustic features
Mush
Search by sound — or by words. Type "barking seal" and CLAP turns your text into a query embedding, then surfaces the clips that match. No example audio required.
Yes. CLAP and AST encode the phylogenetic relationships between marine mammal and bird species about as well as a bioacoustics specialist model, despite never being trained on animal sounds.
Use the embeddings visualizer to project high-dimensional audio vectors down to a 2-D plot, color the points by any field, and lasso clusters that stay synced to the media grid.
A historic Woods Hole Oceanographic Institution archive spanning seven decades, including the first-ever underwater recording of a marine mammal, a beluga taped in 1949. The demo uses a clean subset of about 1,700 clips across 32 species.
Yes. With CLAP and the fiftyone-audio-toolkit plugin, a text prompt like "barking seal" becomes a query embedding that surfaces the closest-matching clips.
