We recently released two new products: FiftyOne 0.21, which is packed with exciting new features, and VoxelGPT, a revolutionary new plugin that turns natural language queries into actual Python code that can filter, sort, semantically slice and reveal insights about the images and videos in your dataset, all without having to write a single line of code.
Voxel51 Co-Founder and CTO Brian Moore walked us through the new capabilities in a live webinar, with plenty of live demos and code examples, so that you can see all the awesomeness in action.
Specifically, the new goodies that Brian demonstrated in the webinar include:
- VoxelGPT: Your AI assistant for computer vision, powered by GPT-3.5 and FiftyOne’s flexible computer vision query language
- Operators & Plugins: New capabilities in FiftyOne 0.21 that make VoxelGPT and any custom features you can dream up to add to your own FiftyOne App
Watch the video playback on YouTube, take a look at the slides, read the transcript, and read the recap below for the highlights. Enjoy!
First, Thanks for Voting for Your Favorite Charity!
In lieu of swag, we gave attendees the opportunity to help guide our monthly donation to charitable causes. This time, the votes were split evenly between two charities: Global Empowerment Mission and Wildlife AI. This means we’ll be sending $100 to each of these charities on behalf of the live audience and the FiftyOne community!
What Is FiftyOne?
Brian starts with a quick overview of what FiftyOne is for those who might be new to it. FiftyOne is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.
Here’s a snapshot of some of the computer vision workflows made possible by FiftyOne:
- Visualize, query, and analyze computer vision datasets
- Streamline data annotation workflows
- Identify and correct labeling mistakes
- Analyze model performance, both visually and programmatically
- And dozens more workflows!
VoxelGPT: Your AI Assistant for Computer Vision
From ~2:31 to 13:44 in the webinar, Brian walks through what VoxelGPT is and gives a demo on how to use it.
Brian explains that “VoxelGPT is a chat-based interface to allow you to ask questions about your datasets, the FiftyOne documentation, and even general computer vision knowledge directly in the FiftyOne App.”
To demonstrate how to use VoxelGPT, Brian uses a dataset from COCO with ground truth and object detections, as well as model predictions from YOLO-NAS. Then Brian asks VoxelGPT these questions and shows the results:
- Can I export this data set in coco format? (Under the hood, VoxelGPT understands to look into the FiftyOne documentation.)
- Show me the samples with high confidence predictions that were evaluated as false positives. (Under the hood, VoxelGPT understands to look into the dataset.)
- Explain computer vision in one sentence. (Under the hood, VoxelGPT understands that you’re asking a general knowledge question and puts together an answer for that.)
For those of you who are curious how we built VoxelGPT, Brian shares this information starting at ~8:29 in the video. Here’s the high level diagram:
You can also learn more about how VoxelGPT was built in these two articles on Towards Data Science:
- How I Turned ChatGPT into an SQL-Like Translator for Image and Video Datasets
- What I Learned Pushing Prompt Engineering to the Limit
Plugins & Operators
At ~13:44 in the webinar replay, Brian describes the upgrades FiftyOne 0.21 brings to the Plugin framework, as well as a new capability introduced in 0.21 – operators.
FiftyOne Plugins provide a flexible mechanism for extending the functionality of the FiftyOne App:
- Add new panels, visualizers, and other types
- Trigger custom Python workflows
- On-startup hooks
- Custom icon placement
- All components are optional
- May contain 100% JavaScript (JS), 100% Python, or a mix
FiftyOne Operators allow you to trigger custom workflows directly from the App via custom forms
- Rich type system to accept user inputs
- Dynamic forms
- Asynchronous streaming output
- Can be implemented in 100% Python!
Bringing It All Together: Wait, VoxelGPT Is a Plugin?!
Yep, you read that right. At ~21:58 in the webinar replay, Brian reveals: VoxelGPT is a plugin!
Brian shows how it’s possible to build a full-fledged application like VoxelGPT using FiftyOne’s plugin architecture.
And he walks through the code used to build VoxelGPT at ~25:12 in the playback video, explaining which pieces of code perform which tasks to bring VoxelGPT to life as a plugin. You can see VoxelGPT instantly in your browser at gpt.fiftyone.ai! And because VoxelGPT is 100% open source, you can find everything you need to use it locally on GitHub.
Live Demo Time!
Watch the live demo starting at ~29:58 in the playback video. In this section, Brian first points our attention to a curated collection of FiftyOne Plugins on GitHub that you can add to your local FiftyOne install (or your FiftyOne Teams deployment).
From this list, Brian demonstrates two ways to use the I/O Plugin, showcasing these two operators:
- add_samples: Use this operator to add samples to an existing dataset by specifying a directory or glob pattern of media paths for which you want to create new samples.
- export_samples: You can use this operator to export your current dataset or view to disk in any supported format.
Along the way, you’ll see examples of using the builtin radio group selector, the progress bar shown to the user, the builtin reload dataset operator, and more!
Other Notes
Open source software like FiftyOne doesn’t happen without an amazing community supporting it. Before closing the presentation, Brian explains that we’re always open to new contributions and contributors! Check out the good first issue label on GitHub. If you are interested in working on one of those, feel free to leave a comment there and we will be happy to help you out in any way.
