Exploring FathomNet Plankton Data with FiftyOne
Author
Chief Community Officer
Jimmy is the Chief Community Officer at Voxel51, where he leads developer relations and growth. With 25+ years of experience connecting developers and enterprises to cutting-edge ML and open-source solutions, Jimmy drives impactful content, events, and education that fuel awareness, adoption, and community engagement.
See all articles by Jimmy Guerrero
MLOps Research Engineer, MBARI
Laura Chrobak is an MLOps Research Engineer at Monterey Bay Aquarium Research Institute.
See all articles by Laura Chrobak
In this article
Talk to an AI expert
Ocean imagery is one of the richest and least-explored frontiers in computer vision. In this post I walk through a small, self-contained demo that pulls two plankton collections from FathomNet into FiftyOne, then uses embeddings, similarity search, and zero-shot classification to turn a folder of 224×224 crops into something you can actually reason about. Everything runs on a laptop, and the whole thing lives in one Jupyter notebook.
Key takeaways
- FathomNet is an open-source, expertly curated ocean image database from MBARI and collaborators. At its 2022 launch it held roughly 84,000 images, 175,000 localizations, and 2,244 concepts.
- The demo loads 200 FathomNet plankton crops from two collections into FiftyOne with the fathomnet Python client. It all runs on a laptop in one Jupyter notebook.
- When CLIP embeddings of FathomNet plankton are projected to 2D in FiftyOne, morphologically similar taxa cluster together. Generic buckets such as Detritus and Diatom smear across the named-species clusters.
- CLIP classifies FathomNet plankton zero-shot with no training. FiftyOne's clickable confusion matrix then loads the exact images behind each error, such as Chaetoceros predicted as Skeletonema.
- FiftyOne's uniqueness score surfaces redundant FathomNet plankton crops that are candidates for pruning before training.
FathomNet: An ocean of labeled images
FathomNet is an open-source, expertly curated image database built to bring modern machine learning to ocean science. It was created by the Monterey Bay Aquarium Research Institute (MBARI) together with collaborators including CVision AI, the Ocean Discovery League, National Geographic Society, and the National Oceanic and Atmospheric Administration (NOAA), and it aggregates decades of underwater visual data into a single machine-learning-ready resource. As of its 2022 launch snapshot, FathomNet held roughly 84,000 images with over 175,000 localizations spanning 2,244 concepts, drawn from 200+ separate collections.
Why does this matter? The ocean is changing fast, and researchers collect visual data far faster than anyone can annotate it by hand. Terrestrial vision benefited from massive labeled corpora such as ImageNet and COCO that could be scraped largely automatically. The ocean has no such luxury: the imagery comes from remotely operated vehicles (ROVs), autonomous underwater vehicles (AUVs), and drop cameras representing enormous amounts of ship time and specialized instrumentation. FathomNet fills that gap with expert taxonomic labels down to the genus and species level — the kind of fine-grained, trustworthy annotation that automated scraping can't produce.
For this demo we used two FathomNet collections of plankton imagery: 200 images total, each a 224×224 crop with a single expert-assigned concept, spanning roughly 18 distinct taxa — named species such as Chaetoceros, Pseudo-nitzschia, Skeletonema, and Thalassiosira, alongside broader buckets such as Nano plankton, Detritus, and generic Diatom. It's a compact, well-posed fine-grained classification problem, which turns out to be an ideal way to show off a dataset-exploration tool without waiting on a GPU cluster.
FiftyOne: A lens for your data
FiftyOne is an open-source toolkit for building, curating, and understanding visual datasets. Instead of squinting at files in a folder or trusting aggregate accuracy numbers, you load your data into a FiftyOne
Dataset and get an interactive app for browsing images and labels, a Python API for slicing and querying, and a "Brain" module for embeddings, similarity, uniqueness, and more.Pairing FiftyOne with FathomNet is a natural fit. FathomNet gives you high-quality, fine-grained labels; FiftyOne gives you the lens to actually see what's in them. Fine-grained marine taxa are visually subtle — the difference between two diatom genera can be a matter of chain structure or cell shape — so a tool that lets you cluster by visual similarity, surface the model's confusions, and click straight from a confusion-matrix cell to the offending images is worth far more than a single accuracy score. The two halves reinforce each other: good labels make the exploration meaningful, and good exploration makes the labels legible.
How to explore FathomNet plankton images in FiftyOne
The demo notebook is fully self-contained. It spins up a dedicated virtual environment, registers a Jupyter kernel, and then walks through the interesting parts:
Loading from the FathomNet API
It pulls both collections via the
fathomnet Python client, hydrates each image's metadata, downloads the crops locally, and builds a single FiftyOne dataset with each collection preserved as a field and tag.Interactive exploration
The FiftyOne App renders the crops as a browsable contact sheet with sidebar filtering by concept and collection — an immediate way to eyeball intra-class variation and spot mislabels.
Filtering FathomNet plankton crops by label in the FiftyOne App sidebar, then opening a single Ciliate sample
Embeddings and 2D visualization
CLIP image embeddings are projected to 2D so morphologically similar plankton land near each other. This is where the generic buckets visibly smear across the named-species clusters.
Filtering FathomNet plankton crops by label in the FiftyOne App sidebar, then opening a single Ciliate sample
Similarity search
A similarity index lets you select any image and retrieve its nearest visual neighbors — "find me more plankton that look like this."
Running an image similarity search in the FiftyOne App that returns the 25 FathomNet plankton crops most similar to a selected sample
Zero-shot classification
CLIP classifies each crop against the concept list using an
”a microscope photo of {concept}” prompt template, with no training required.
Switching between saved views such as misclassifications in the FiftyOne App, then inspecting a CLIP zero-shot prediction on a FathomNet plankton crop
Evaluation with an interactive confusion matrix
The zero-shot predictions are scored against the expert labels, and the confusion matrix is clickable: pick a cell and the app loads exactly those images, so a confusion such as Chaetoceros → Skeletonema becomes something you can inspect rather than guess about.
Clicking a confusion matrix cell in the FiftyOne Model Evaluation panel to load the FathomNet plankton crops CLIP misclassified
Uniqueness and near-duplicates
A uniqueness score surfaces redundant crops that are candidates for pruning before training.
Sorting FathomNet plankton crops by uniqueness in the FiftyOne App to surface near-duplicate Nano plankton images
Next steps and resources
If you want to go further with this setup:
- Swap in a stronger model. Replace zero-shot CLIP with a fine-tuned classifier and re-run the evaluation to watch the confusion matrix tighten up.
- Scale up. These were two small collections; FathomNet has thousands. The same loading pattern extends to larger, bounding-box-annotated collections where FiftyOne's detection evaluation and geolocation map plots come into play.
- Contribute back. FathomNet is community-driven, and mislabels you surface during exploration are exactly the kind of feedback the project benefits from.
Useful links:
- FathomNet Program is at www.fathomnet.org
- FathomNet Database is at database.fathomnet.org
- FathomNet at MBARI: https://www.mbari.org/data/fathomnet/
- FathomNet Python client (
fathomnet-py): https://github.com/fathomnet/fathomnet-py - The FathomNet paper (Scientific Reports, 2022): https://www.nature.com/articles/s41598-022-19939-2
- The demo notebook
- FiftyOne documentation: https://docs.voxel51.com/
- FiftyOne Brain (embeddings, similarity, uniqueness): https://docs.voxel51.com/user_guide/brain.html
FAQ
FathomNet is an open-source, expertly curated image database for ocean science. MBARI created it with collaborators including CVision AI, the Ocean Discovery League, National Geographic Society, and NOAA. Its labels go down to the genus and species level.
You use the fathomnet Python client (fathomnet-py) to pull collections, hydrate metadata, and download images. Then you build a FiftyOne dataset that keeps each collection as a field and a tag. The demo notebook does this end to end.
Yes. CLIP can zero-shot classify plankton crops against a concept list using a prompt template such as "a microscope photo of {concept}." Scoring the predictions against expert labels in FiftyOne shows where it gets taxa confused.
You browse and filter by concept in the FiftyOne App, check where samples fall in the 2D embeddings plot, and click into confusion matrix cells to inspect specific images. FathomNet is community-driven, so you can report the mislabels you find back to the project.
No. Everything in this demo runs on a laptop.
Chief Community Officer
Jimmy is the Chief Community Officer at Voxel51, where he leads developer relations and growth. With 25+ years of experience connecting developers and enterprises to cutting-edge ML and open-source solutions, Jimmy drives impactful content, events, and education that fuel awareness, adoption, and community engagement.
See all articles by Jimmy Guerrero
MLOps Research Engineer, MBARI
Laura Chrobak is an MLOps Research Engineer at Monterey Bay Aquarium Research Institute.
See all articles by Laura Chrobak
Talk to an AI expert
Loading related posts...
© 2026 Voxel51 All Rights Reserved
