Embed Custom UIs Directly into Data-Centric Workflows
Welcome to week three of Ten Weeks of Plugins. During these ten weeks, we will be building a FiftyOne plugin (or multiple!) each week and sharing the lessons learned! If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:
Here’s what we’ve built so far:
Ok, let’s dive into this week’s FiftyOne Plugin!
Custom YouTube Player Panel 📺🔧🔘
Over the first two weeks of this 10 Weeks of Plugins journey, all three of the plugins that I built were pure
Python plugins plugins that allow you to create
operators, which execute custom code behind the scenes. These plugins enable a plethora of workflows, and as primarily a Python programmer myself, I find the process for creating Python plugins to be quite intuitive, especially after creating a few. Sometimes, however, it’s nice to be able to create a custom user interface. Take
VoxelGPT for instance: the chatbot-like interface (plus easy statefulness!) is really only possible when given its own devoted space within the FiftyOne App. In FiftyOne, creating custom interfaces like this is possible via JavaScript Plugins, which give you blank canvases on which to design workflows and experiences. In FiftyOne, these “canvases” are called
Panels
, and when you write a JavaScript plugin, you can build your own panels from scratch. Using JavaScript (and React Material UI components) you can essentially design a custom UI for yourself and/or others, which you can pull up at the click of a button. You’ll see what I mean shortly! For the third week of
10 Weeks of Plugins, I built a YouTube Player Panel Plugin. For this plugin, I built a relatively simple UI consisting of a YouTube video player and a few selectors to make it interactive. While this plugin itself is not terribly complex, and my UI is not terribly artistic, it showcases the basic elements required to build a panel to your liking. The best part: I used ChatGPT to write the JavaScript for me!
Plugin Overview & Functionality
The YouTube Player Panel Plugin is a joint Python/JavaScript plugin with a single operator:
open_youtube_player_panel
, which opens a custom JavaScript component named
YouTubePlayerPanel
. You can choose to play a video from a preset list of YouTube videos from the
Voxel51 YouTube channel, or you can play a YouTube video from URL. There are three ways to execute the
open_youtube_player_panel
operator and open the panel:
- Press the YouTube player button in the Sample Actions Menu:
[@portabletext/react] Unknown block type "externalImage", specify a component for it in the `components.types` prop
- Click on the
+
icon next to the Samples
tab and select YouTube Player
from the dropdown menu:
[@portabletext/react] Unknown block type "externalImage", specify a component for it in the `components.types` prop
- Press “
`
” to pull up your list of operators, and select open_youtube_player_panel
:
[@portabletext/react] Unknown block type "externalImage", specify a component for it in the `components.types` prop
The Python portion of the plugin is exceedingly simple:
- Storing an SVG icon for YouTube in an
assets
folder within the plugin’s directory, I specified this as the icon to use to represent the operator in two ways: first, in the operator’s config()
method, so that this would be the icon in the operators list, and second, in its resolve_placement()
method, so that the button in the action menu is also the YouTube icon. - In the
execute()
method, the operator triggers an open_panel
action, and the params passed into trigger()
instruct that the panel to be opened is named YouTubePlayerPanel
.
The JavaScript portion is a bit more involved. It consists of:
YouTubeIcon
: the SVG icon that appears next to the label YouTube Player
on the tab in the panel.YoutubeEmbed
: an iframe component (basically an embedded live view of HTML loaded on another page — in this case YouTube). This component takes as an argument an embedId
, which is the unique identifier for YouTube videos.YouTubePlayerPanel
: the custom panel containing the YouTube player iframe, a list of preset videos, a text field for inputting a different video link, and the logic for updating the display based on values of the associated variables.registerComponent
: this registers the YouTubePlayerPanel
we defined as a Panel
with the name “YouTubePlayerPanel”, so that we can reference it (and open it) from our Python operator.
Installing the Plugin
If you haven’t already done so, install FiftyOne:
Then you can download this plugin from the command line with:
Refresh the FiftyOne App, and you should see the YouTube button show up in the actions menu.
Lessons Learned
Icon Rendering
FiftyOne plugins are very flexible when it comes to icon rendering. If you desire, you can have distinct icons for an operator in the action menu, the operator list, and at the top of its panel. Even though this particular plugin only uses one icon, I try to stick to the convention of putting all icons in an assets
folder, and then referencing the svg file via absolute path within the plugin’s directory. For example:
For the icon in the panel itself, I had to create a separate icon component in JavaScript. The paths in the svg were the same, but I had to use different viewBox
coordinates to make all of the icons line up nicely in their respective contexts.
Trigger Composes Operators
In the
AI Art Gallery and
Twilio Automation plugins, I had used the
ctx.trigger()
method to perform operations like reloading samples (
ctx.trigger(“reload_samples”)
), and reloading the dataset (
ctx.trigger(“reload_dataset”)
). I was even aware from
VoxelGPT that you could use
ctx.trigger()
to set the session’s view. But it wasn’t until this YouTube Player Plugin that it clicked for me that
ctx.trigger()
allows you to compose operators. You can trigger a Python operator from JavaScript, or a JavaScript operator from Python. You can even trigger operators defined in other plugins! In this plugin the Python operator’s
execute()
method looks like:
What this means is that when executed, the open_youtube_player_panel
operator is triggering another operator, open_panel
, with the specific input parameters. In this case, the open_panel
operator is opening the panel named YouTubePlayerPanel
, which we defined in our YouTubePlayerPlugin.tsx
file, and is splitting the canvas of the FiftyOne App in two horizontally (as opposed to vertically). You can do so much with these operator compositions. I’m excited to explore this further in coming weeks!
No JavaScript?; No Problem!
My biggest takeaway from building this plugin is that you don’t actually need to know any JavaScript, React, or Typescript to build a “JavaScript” plugin in FiftyOne. The greatest strength of FiftyOne’s JavaScript plugins is their straightforward interface. Because FiftyOne plugin components map fairly directly onto React Material UI components, I found that the
FiftyOne Plugin docs plus the
Material UI docs plus ChatGPT was more than enough to turn my vision into reality. If you’d like to dive deeper into this, let me know and I’ll put together some resources :)
Conclusion
FiftyOne plugins are incredibly flexible. They give you the blank canvas you need to create tailor-made data-centric machine learning applications. Playing YouTube videos in the FiftyOne App is cool, but the real value is the ability to educate, to showcase workflows, and to create a compelling story — all with data at the forefront! Stay tuned for the remainder of these ten weeks as we continue to pump out a killer lineup of plugins! You can track our journey in our
ten-weeks-of-plugins repo — and I encourage you to fork the repo and join me on this journey!