Embed Custom UIs Directly into Data-Centric Workflows
Welcome to week three of Ten Weeks of Plugins. During these ten weeks, we will be building a FiftyOne plugin (or multiple!) each week and sharing the lessons learned!
If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:
Here’s what we’ve built so far:
Ok, let’s dive into this week’s FiftyOne Plugin!
Custom YouTube Player Panel 📺🔧🔘
Over the first two weeks of this 10 Weeks of Plugins journey, all three of the plugins that I built were pure Python plugins plugins that allow you to create operators, which execute custom code behind the scenes. These plugins enable a plethora of workflows, and as primarily a Python programmer myself, I find the process for creating Python plugins to be quite intuitive, especially after creating a few.
In FiftyOne, these “canvases” are called
For the third week of 10 Weeks of Plugins, I built a YouTube Player Panel Plugin. For this plugin, I built a relatively simple UI consisting of a YouTube video player and a few selectors to make it interactive. While this plugin itself is not terribly complex, and my UI is not terribly artistic, it showcases the basic elements required to build a panel to your liking.
Plugin Overview & Functionality
YouTubePlayerPanel. You can choose to play a video from a preset list of YouTube videos from the Voxel51 YouTube channel, or you can play a YouTube video from URL.
There are three ways to execute the
open_youtube_player_panel operator and open the panel:
- Press the YouTube player button in the Sample Actions Menu:
- Click on the
+icon next to the
Samplestab and select
YouTube Playerfrom the dropdown menu:
- Press “
`” to pull up your list of operators, and select
The Python portion of the plugin is exceedingly simple:
- Storing an SVG icon for YouTube in an
assetsfolder within the plugin’s directory, I specified this as the icon to use to represent the operator in two ways: first, in the operator’s
config()method, so that this would be the icon in the operators list, and second, in its
resolve_placement()method, so that the button in the action menu is also the YouTube icon.
- In the
execute()method, the operator triggers an
open_panelaction, and the params passed into
trigger()instruct that the panel to be opened is named
YouTubeIcon: the SVG icon that appears next to the label
YouTube Playeron the tab in the panel.
YoutubeEmbed: an iframe component (basically an embedded live view of HTML loaded on another page — in this case YouTube). This component takes as an argument an
embedId, which is the unique identifier for YouTube videos.
YouTubePlayerPanel: the custom panel containing the YouTube player iframe, a list of preset videos, a text field for inputting a different video link, and the logic for updating the display based on values of the associated variables.
registerComponent: this registers the
YouTubePlayerPanelwe defined as a
Panelwith the name “YouTubePlayerPanel”, so that we can reference it (and open it) from our Python operator.
Installing the Plugin
If you haven’t already done so, install FiftyOne:
pip install fiftyone
Then you can download this plugin from the command line with:
fiftyone plugins download https://github.com/jacobmarks/vqa-plugin
Refresh the FiftyOne App, and you should see the YouTube button show up in the actions menu.
FiftyOne plugins are very flexible when it comes to icon rendering. If you desire, you can have distinct icons for an operator in the action menu, the operator list, and at the top of its panel. Even though this particular plugin only uses one icon, I try to stick to the convention of putting all icons in an
assets folder, and then referencing the svg file via absolute path within the plugin’s directory. For example:
viewBox coordinates to make all of the icons line up nicely in their respective contexts.
Trigger Composes Operators
In the AI Art Gallery and Twilio Automation plugins, I had used the
ctx.trigger() method to perform operations like reloading samples (
ctx.trigger(“reload_samples”)), and reloading the dataset (
ctx.trigger(“reload_dataset”)). I was even aware from VoxelGPT that you could use
ctx.trigger() to set the session’s view.
But it wasn’t until this YouTube Player Plugin that it clicked for me that
In this plugin the Python operator’s
execute() method looks like:
def execute(self, ctx): ctx.trigger( "open_panel", params=dict( name="YouTubePlayerPanel", isActive=True, layout="horizontal" ), )
What this means is that when executed, the
open_youtube_player_panel operator is triggering another operator,
open_panel, with the specific input parameters. In this case, the
open_panel operator is opening the panel named
YouTubePlayerPanel, which we defined in our
YouTubePlayerPlugin.tsx file, and is splitting the canvas of the FiftyOne App in two horizontally (as opposed to vertically).
You can do so much with these operator compositions. I’m excited to explore this further in coming weeks!
Because FiftyOne plugin components map fairly directly onto React Material UI components, I found that the FiftyOne Plugin docs plus the Material UI docs plus ChatGPT was more than enough to turn my vision into reality. If you’d like to dive deeper into this, let me know and I’ll put together some resources 🙂
FiftyOne plugins are incredibly flexible. They give you the blank canvas you need to create tailor-made data-centric machine learning applications. Playing YouTube videos in the FiftyOne App is cool, but the real value is the ability to educate, to showcase workflows, and to create a compelling story — all with data at the forefront!
Stay tuned for the remainder of these ten weeks as we continue to pump out a killer lineup of plugins! You can track our journey in our ten-weeks-of-plugins repo — and I encourage you to fork the repo and join me on this journey!
Join the thousands of engineers and data scientists already using FiftyOne to solve some of the most challenging problems in computer vision today!
- ⭐ Star the FiftyOne GitHub repo (4,000+ stars and counting)
- ⭐ Star the FiftyOne Plugins repo on GitHub
- 📚Check out the FiftyOne plugin docs
- 🎉 Join the 2,000-strong Slack community of ML engineers and data scientists, and check out the #plugins channel!
- 🎉 Join 5,000+ in the Computer Vision meetup network — and stay tuned for our upcoming workshop on plugins!