Recapping the Journey from Zero to Plugin Hero
Back in June, with the release of FiftyOne 0.21, the open source computer vision library FiftyOne became more flexible, customizable, and extensible than ever with the introduction of operators. With this simple, yet powerful feature, we unlocked limitless possibilities for anyone to customize their FiftyOne experience exactly the way they want.
Nevertheless, I set out with the intention of building one plugin each week for ten weeks. My initial goal was to build the custom functionality I wanted in the FiftyOne App, wrapping many of my commonly executed code blocks in Python operators so I could run these commands without digging up each snippet every time I wanted to use it. However, my Ten Weeks of Plugins journey turned into so much more.
In this post, I want to summarize all of the plugins I managed to build throughout this journey, and share the lessons I learned along the way. The plugins below are presented in chronological order (week by week). Very roughly, the plugins and lessons learned build upon each other, but if you’d like, feel free to skip ahead to any topic that interests you.
- 🌩️Image Quality Issues
- 📈Concept Interpolation
- 🎨Build Your Own AI Art Gallery
- 📲Data Ingestion with Twilio
- ❔Visual Question Answering
- 📺In-App Youtube Player
- 🪞Image Deduplication
- 🔡Optical Character Recognition with PyTesseract
- 🔑Keyword Search
- 🔮Zero-Shot Prediction
- 🏃Active Learning
- ⏪Reverse Image Search
- 🌌Concept Space Traversal
- 🛠️Plugin Management and Development
- 🔊Audio Retrieval with ImageBind
- 🔎Semantic Document Search
🌩 Image Quality Issues (Week 0)
GitHub Repo: https://github.com/jacobmarks/image-quality-issues
One of the most pervasive problems in computer vision is constructing a high quality dataset to use for model training. Some aspects of dataset construction are subtle. But some issues are clear as day.
This plugin allows you to find common image quality issues plaguing your dataset, including low entropy (low information content), low or high contrast, bright and dark images, and images that are blurry. You can specify the threshold for designating an image as having a certain issue, or use preset defaults. Relevant computations for given diagnoses will be run if necessary.
- Streamline Common Workflows: whether you are computing entropy, brightness, or a custom metric, you are best served by wrapping this code in a Python operator which you can access from the FiftyOne App!
- Interactivity Retains Flexibility: instead of choosing a set threshold for diagnosing an image with a certain issue, it is much more powerful and flexible to use a slider (or some other input mechanism) to let the user specify the value.
📈 Concept Interpolation (Week 0)
GitHub Repo: https://github.com/jacobmarks/concept-interpolation
With natural language image search, you can type in a text prompt and find the images in your dataset that most closely match that description. Under the hood, this type of query leverages a multimodal embedding model like OpenAI’s CLIP, which can embed text and images in the same vector space.
Concept Interpolation takes this a step further and allows you to interpolate between two “concepts” by way of their embeddings. Pick two text prompts — for instance “sunny” and “rainy”, and choose a point along the line connecting them. The Concept Interpolation plugin generates the weighted average of the embeddings for the two prompts, and searches for the images whose embeddings most closely match this new vector.
- Custom Operator Icons: you can use any SVG as an icon for an operator in FiftyOne. Open the SVG file with a text editor to edit its contents, specify the fill colors, and remove unwanted text.
🎨 Build Your Own AI Art Gallery (Week 1)
GitHub Repo: https://github.com/jacobmarks/ai-art-gallery
Text-to-image models like Stable Diffusion and DALLE-2 can create some stunning AI-generated artwork. Once you generate these images, however, the process of curating your own AI art gallery — keeping track of the creation date, model, prompt, and hyperparameters — can be quite laborious.
This plugin automates all of this gruntwork. Run a variety of text-to-image models directly from the FiftyOne App, including Stable Diffusion, SDXL, Latent Consistency Models, and DALLE-2, and have the resulting images and the relevant metadata automatically added to your dataset/collection!
- Custom Component Properties: you can pass in a dictionary of componentProps as an argument to any view. This essentially puts the power of Material UI Components in your hands.
- Plugins and Environment Variables: you can gracefully handle different cases based on whether or not the user has certain packages installed. Use importlib’s
find_spec, check environment variables with
os.environ, and lazily import with FiftyOne’s
For more details, check out the blog post, Build Your Own AI Art Gallery.
📲 Data Ingestion with Twilio (Week 1)
Have you ever happened across a real-life scene you wish you could capture and add into one of your datasets? With FiftyOne and Twilio (the programmable communication company), now you can! Take a picture (or multiple), and send them to your Twilio phone number. Next time you open the FiftyOne App to your desired dataset, use the
add_twilio_images operator to add the photos you sent directly to the dataset. You can also filter by message content, only adding images from messages whose text contains a given text string!
- Python Plugins Can Be Simple: I built the majority of this plugin within two hours at the Twilio Signal Creator Summit Hackathon. Since then, I’ve given it a few upgrades, but the core code has remained the same. FiftyOne Python plugins are easy enough to build that you can create a single-purpose plugin within a few hours.
❔Visual Question Answering (Week 2)
GitHub Repo: https://github.com/jacobmarks/vqa-plugin
Do you ever wish you could have a conversation with the images in your dataset? With this Visual Question Answering (VQA) plugin, you can ask your images anything. Select an image from the sample grid, enter a question (text prompt), and receive a text-based response from your model of choice. This plugin gives you access to four cutting edge multimodal models: Adept’s Fuyu-8b, Llava-13b, BLIP2, and a Vision-Language Transformer (ViLT) fine-tuned on the VQA2 dataset.
- Using Selected Samples: within an operator’s
ctx.selectedwill give you access to the list of currently samples samples in the FiftyOne App — the same as
- Returning Informative Outputs: you can return a JSON-style dictionary from
execute()and use these variables in
For more details, check out the blog post, Ask Your Images Anything.
📺 In-App Youtube Player (Week 3)
On the surface, this plugin is about enabling you to play YouTube videos in a custom panel with the FiftyOne App. Choose from the preset list of videos, or enter the URL for another YouTube video, and the in-app YouTube embed will update automatically. This could be useful if you want to watch a FiftyOne tutorial and follow along in real time, without leaving the app.
Delving deeper, however, this plugin is meant to showcase the simplicity of building custom user journeys and interactive workflows within the FiftyOne App! You can define custom panels as React components and fill the canvas of these panels with whatever Material UI components and interactive logic you’d like!
viewBoxfor the icons to appear centered and sized as desired.
- Trigger Composes Operators: The
ctx.trigger()method within an operator’s
For more details, check out the blog post, Build Custom Computer Vision Applications.
🪞Image Deduplication (Week 4)
Duplicate data can lead to a plethora of problems, from longer than necessary training times, to models over-indexing on specific samples rather than learning larger patterns. But deduplicating your dataset doesn’t have to be complicated!
This image deduplication plugin makes it easy to find both exact duplicates (via file hashes) and approximate duplicates (via image embeddings). Once you’ve found these duplicates, the plugin provides operators for displaying groups of duplicates, removing all duplicates, and deduplicating — removing all except one representative from each group.
- Splitting Code into Submodules: separation of concerns is good. To extract logically related sections of code into separate Python modules and import them in a FiftyOne plugin, you can use FiftyOne’s
add_sys_pathutil in conjunction with the
os.pathmethods to locate the module to be imported.
- Loading a View: you can trigger another operator from an operator’s
execute()method. To load a view, use the built-in
load_viewoperator, passing in a serialized representation of the
DatasetViewto be loaded.
- Icons for Operators: setting the
iconattribute to an operator’s
OperatorConfigallows for a custom icon to appear next to the operator in the operator list that shows up when you press “`”.
For more details, check out the blog post, Double Trouble: Eliminate Image Duplicates with FiftyOne.
🔡 Optical Character Recognition with PyTesseract (Week 5)
GitHub Repo: https://github.com/jacobmarks/pytesseract-ocr-plugin
Optical Character Recognition (OCR) is the task of identifying and localizing text (characters) in a document. It can be useful for extracting insights from everything from license plates to legal documents, medical records, and even handwritten notes.
This plugin leverages the most popular OCR engine, Tesseract, via its Python bindings (PyTesseract), allowing you to recognize text in the images in your dataset. Choose whether to save word predictions, contiguous blocks of text, or both, and whether to execute in real-time or delegate as a job. OCR text predictions are converted to and stored in
Detection labels on samples.
- Menu Buttons in Python Plugins: you can turn your Python operator into a button in the FiftyOne App with the
For more details, check out the blog post, Optical Character Recognition with PyTesseract.
🔑 Keyword Search (Week 5)
GitHub Repo: https://github.com/jacobmarks/keyword-search-plugin
Motivated by the need to search through the blocks of text generated by the OCR plugin, the Keyword Search plugin makes it easy to search your dataset for specific text strings without writing any custom code. The plugin finds all top-level strings and lists of strings, as well as strings and lists of strings within detection labels, and applies the appropriate view stage to search for the query text within the specified field. It acts as a wrapper for the
match_labels() view stages with the
contains_str() view expression, exposing the option for case sensitive or insensitive search.
The plugin uses a cache manager to cache the name of the last field queried, and sets that as the default field to be queried moving forward. This serves as a proof of principle for caching user preferences in Python plugins.
- Caching in Python Plugins: you can’t do any caching within the
__init__.pyfile itself, because this file gets reloaded and its variables reset every time the operator is used. Create a separate Python file for caching and import the caching function from this module in
- Plugging in Your Plugins: the keyword search plugin and the OCR plugin together are way more valuable than each is on its own. Make your plugins flexible enough to work well together, so you can create complex workflows.
For more details, check out the blog post, Optical Character Recognition with PyTesseract.
🔮 Zero-Shot Prediction (Week 6)
One of the most costly and time-consuming parts of machine learning and computer vision work is the annotation of ground truth data. With foundation models like CLIP and SAM, we can accelerate this process by pre-labeling our data.
This plugin provides a unified interface for zero-shot or open-vocabulary prediction, which can be used to greatly streamline annotation workflows. Whether you are interested in classification, object detection, instance segmentation, or semantic segmentation, this plugin has you covered.
Select your task and model, input your desired class names — either directly or by pointing to a labels file — and generate zero-shot predictions. You can run the zero-shot models on selected samples, your current view or the entire dataset, and the plugin can be used in conjunction with FiftyOne’s annotation integrations (CVAT, Labelbox, Label Studio).
- Keep up with FiftyOne’s Plugin Features: this plugin utilized two features which did not exist at the beginning of the ten week journey: delegated operators and the file explorer. New features are being added all the time, so stay up to date!
- Reduce Boilerplate Code: if you have multiple operators with very similar information or execution flows, you can extract these as more general functions.
For more details, check out the blog post, Zero-Shot Prediction Plugin for FiftyOne.
🏃Active Learning (Week 7)
GitHub Repo: https://github.com/jacobmarks/active-learning-plugin
Active Learning is a set of techniques for getting to a set of ground truth labels faster and more efficiently. Starting from some preliminary labeled/tagged examples, active learning generates hypotheses for the rest of the samples in the dataset, and cleverly determines which samples for the human to label next based on the certainty of the learning model’s existing hypotheses.
This plugin wraps the functionality of the modAL library in a simple interface within the FiftyOne App. You can create a “learner” from an initial set of labels, query the learner for new samples to label, and teach the learner with the newly labeled examples.
The plugin can be used in conjunction with the zero-shot prediction plugin, as well as FiftyOne’s annotation integrations (CVAT, Labelbox, Label Studio). Currently it only supports classification tasks, but could readily be extended to support other computer vision labeling tasks.
- Caching Is King: sometimes you need to cache more than just user preferences. As in the keyword search plugin, you can cache variables in modules that are imported by the
- Don’t Reinvent the Wheel: for many workflows, there are decent pre-existing solutions. It is often better to wrap the functionality from a purpose-built library as a FiftyOne plugin than to try to recreate all of the functionality from scratch.
For more details, check out the blog post, Supercharge Your Annotation Workflow with Active Learning.
⏪ Reverse Image Search (Week 8)
Since March 2023 when we added native vector search functionality into FiftyOne, it has been possible to select an image from your dataset and perform a similarity search to find the
k most similar images within the dataset. This plugin extends this functionality, allowing you to query your dataset with external images. With reverse image search, you can now drag and drop images from your local filesystem, or enter the URL of an image, and find the most similar images in your dataset to these queries!
- Handling Remote Images: you can work with images from the web without storing them to disk. Use Python’s `requests` library to get the data, then create a file-like object using
io.BytesIO, and treat the resulting object like you would a file on disk.
BytesIOto create the file-like object.
For more details, check out the blog post, Reverse Image Search Plugin for FiftyOne.
🌌 Concept Space Traversal (Week 9)
💪💪 Joint effort with Ibrahim Manjra 💪💪
Similar in spirit to Concept Interpolation, but more flexible and interactive, Concept Space Traversal allows you to combine text and image concepts and dynamically move through the space of embeddings. Select a starting image, and iteratively add text concepts in desired amounts. On the backend, the text and image embedding vectors will be combined to create a new “destination” vector, which is then used to query the dataset’s vector similarity index. By adding text prompts, you can move through the space of “concepts” in your dataset.
For more details, check out the blog post, Concept Traversal Plugin for FiftyOne.
🛠️ Plugin Management and Development (Week 10)
💪💪 Joint effort with Brian Moore 💪💪
We learned a lot of lessons from building plugins for ten weeks. This plugin packages those lessons into utilities to make plugin management and development easier. On the management side, you can install new plugins and enable/disable installed plugins from within the FiftyOne App. On the development side, this plugin helps you to interactively build input components for Python operators, and walks you through the process of scaffolding the code for Python plugins, even creating the files for you!
- Dynamically Updating Code Blocks: to make code blocks that update based on user interactions, make the identifier for code block component dependent on each of the inputs. This way, the app acts as if the old component was removed and a new component matching the new input specifications was created in its place.
- It’s Plugins All The Way Down: plugins are insanely flexible, giving you the power to build custom computer vision applications for almost any workflow. This same flexibility makes it possible to build a plugin-builder plugin. But by the same token, it is impossible to bottle up the full richness of the plugin system and provide this via a crisp UI.
For more details, check out the blog post, Plugin for Building and Managing Plugins.
🔊 Audio Retrieval with ImageBind (✨ Bonus)
GitHub Repo: https://github.com/jacobmarks/audio-retrieval-plugin
If querying your image datasets with other images or natural language isn’t your cup of tea, this plugin allows you to search for images that most closely match input audio clips! Have a .wav file with a train chugging along? Now you can use that to find images of trains. This is made possible by Meta AI’s ImageBind model, which embeds data of six modalities in the same space, including images and audio. The plugin also uses a custom Qdrant collection (not the native integration) to enable fast vector search.
- Non-Native Vector Database Integrations: while FiftyOne does have native integrations with vector databases (Qdrant, Pinecone, Milvus, LanceDB), sometimes it is easier to create a custom vector db wrapper for a specific use case. In this case, because ImageBind is accessed via Replicate, I found it easier to write the search queries explicitly than try to fit the form factor of FiftyOne’s similarity index.
🔎 Semantic Document Search (✨ Bonus)
If you liked the OCR and Keyword Search plugins from week five, you’re going to love this one, which allows you to semantically search through the text blocks generated by the OCR engine. With this plugin, you can embed the text blocks from PyTesseract using the GTE-base embedding model from Hugging Face’s Sentence Transformers library, and generate a vector index out of these embeddings. You can then find all text blocks across all pages in your dataset containing the most similar text to your queries.
Plugging in Your Plugins Redux: with OCR and Keyword Search, we saw how composing plugins could create more value. Semantic Document Search doubles down on this. You can use the PDF loader plugin to read pdf pages in as image samples, run the OCR plugin to extract text from the generated pages, and then use the Semantic Document Search plugin to find similar text blocks throughout the pages of a PDF!
But this is just the beginning. I barely scratched the surface of FiftyOne’s plugin framework. And these were just the plugins that I wanted to build. Without exception, everyone has their own set of essential workflows, and their own vision for novel computer vision applications. I hope these plugins serve as a helpful starting point as you begin to explore the endless possibilities of FiftyOne plugins!
If you’ve made it this far, register for our FiftyOne Plugins Workshop on Authoring Data-Centric AI Applications, taking place on November 15th at 9am PT. In just 90 minutes, you’ll learn how to build your own FiftyOne Plugin!
And while you’re at it:
⭐ Star the FiftyOne GitHub repo (5,300+ stars and counting)
⭐ Star the FiftyOne Plugins repo on GitHub
📚Check out the FiftyOne Plugin docs
🎉 Join the 2,200-strong Slack community of ML engineers and data scientists, and check out the #plugins channel!
🎉 Join 5,000+ in the Computer Vision Meetup network — and stay tuned for our upcoming workshop on plugins!