Want to surface interesting insights about your image and video datasets without writing code? Now you can with VoxelGPT!

https://www.youtube.com/watch?v=MDeG6lM7hUg

VoxelGPT combines the power of large language models (LLMs) with FiftyOne’s flexible computer vision query language, making it easier than ever to semantically slice computer vision datasets and build better machine learning models.

This breakthrough gives you unprecedented control over your data via natural language. You can filter your image and video datasets, uncover new insights, and become a better computer vision engineer, all without writing a single line of code!

Even better, VoxelGPT is completely open source and free to use. The code is located at https://github.com/voxel51/voxelgpt and you can try it out live at gpt.fiftyone.ai. Want to use VoxelGPT locally on your own data? You can install the plugin locally (read on for instructions).

VoxelGPT is an LLM-powered application built with FiftyOne, GPT-3.5, and LangChain. VoxelGPT provides a chat-like interface for interacting with your computer vision dataset by translating natural language queries into FiftyOne Python syntax and constructing the resulting DatasetView.

This is a game-changer because mastering the FiftyOne query language, with all of its flexibility, can have a steep learning curve. With VoxelGPT, you can immediately wield the full power of FiftyOne to semantically slice your data without any prior knowledge of the query language.

What’s more, VoxelGPT can also answer specific FiftyOne usage questions and machine learning questions more broadly.

You can use VoxelGPT via a simple Python API, or you can use VoxelGPT natively inside the FiftyOne App by installing it as a FiftyOne Plugin!

VoxelGPT won’t do your computer vision work for you, but it can certainly increase your speed and efficiency. It is a pair programmer, a translator, and an educational tool all rolled into one.

VoxelGPT is capable of handling any of the following types of queries:

When you ask VoxelGPT a question, it will interpret your intent, and determine which type of query you are asking. If VoxelGPT is unsure, it will ask you to clarify.

https://www.youtube.com/watch?v=rSJAAy5uCg0

How does it work? VoxelGPT interprets your query, translates it into FiftyOne query language Python code, and displays the resulting view. It knows how to work with datasets where the samples are images or videos, and has full support for one and two-stage queries based on the set of examples on which it was developed.

When interpreting your query, VoxelGPT will do the following:

Here are some examples of dataset queries you can ask VoxelGPT—try them out live at gpt.fiftyone.ai:

https://www.youtube.com/watch?v=_CfBYMMdTHk

VoxelGPT is not just a pair programmer; it is also an educational tool. The model has access to the entire FiftyOne docs - including tutorials, user guide, and the API reference—and can use this information to answer your questions.

Here are some examples of documentation queries you can ask VoxelGPT:

By effortlessly switching between dataset queries and docs queries, you can use VoxelGPT to better understand how the FiftyOne query language works.

https://www.youtube.com/watch?v=0g_PVdCSOPU

VoxelGPT can also answer general questions in computer vision, machine learning, and data science. It can help you to understand basic concepts and overcome data quality issues.

Here are some examples of computer vision queries you can ask VoxelGPT:

While VoxelGPT is powerful, we have purposely limited its scope to provide a focused user experience while also leaving the door open for high value upgrades in the future.

VoxelGPT is currently not able to:

VoxelGPT’s current implementation is based on a limited set of examples, so it may not generalize well to all data. The more specific your query, the better the results will be.If you have a more involved task, see if you can split it up into multiple natural language queries and combine VoxelGPT’s results.

We’d love to hear from you about how VoxelGPT performs on your use cases. Bugs, feature requests, and surprising findings that VoxelGPT uncovered are all welcome. Please drop us a line!

If you want to experience VoxelGPT and see for yourself how the model turns natural language into computer vision insights, check out the live demo at gpt.fiftyone.ai, where you can use VoxelGPT natively in the FiftyOne App on a few example datasets.

You can also interact with VoxelGPT programmatically in Python and/or work with your own datasets by installing it locally by following the instructions below.

If you haven't already, install FiftyOne:

Next provide an OpenAI API key:

If you do not have an OpenAI key, you will need to create one.

Now if you only want to use VoxelGPT in the FiftyOne App, you can install it as a plugin by running:

Alternatively, if you want to programmatically interact with VoxelGPT - or you want to contribute to the project - then clone voxelgpt the repository:

and install the requirements:

To make the plugin available within the FiftyOne App as well, you can symlink it into your FiftyOne plugins directory:

With VoxelGPT installed, you can use it natively within the FiftyOne App with any dataset:

In the FiftyOne App, simply:

Pro tip: use the now keyword to incorporate your previous prompts as context for your next query!

If you opted for a Python/development install, you can also interact with VoxelGPT programmatically in Python.

You can use VoxelGPT in interactive mode via ask_voxelgpt_interactive(). In interactive mode, you can converse with VoxelGPT via input() prompts.

To get started, just pass in a dataset and an App session:

As usual, you can prompt VoxelGPT with any combination of dataset, documentation, and general computer vision queries. For example, a conversation might look like:

You:

    what is the difference between precision and recall?

VoxelGPT:

    Precision and recall are two important metrics used to evaluate the

    performance of a machine learning model in classification tasks. Precision

    measures the proportion of true positive predictions among all positive

    predictions made by the model. Recall, on the other hand, measures the

    proportion of true positive predictions among all actual positive instances

    in the dataset.

    In other words, precision is the ability of the model to correctly identify

    positive instances, while recall is the ability of the model to identify

    all positive instances. A high precision means that the model is making

    very few false positive predictions, while a high recall means that the

    model is able to identify most of the positive instances in the dataset.

    It is important to balance precision and recall when evaluating a model's

    performance. A model with high precision but low recall may be missing

    important positive instances, while a model with high recall but low

    precision may be making too many false positive predictions.

    In the context of the FiftyOne library, precision and recall can be

    computed using the `fiftyone.utils.metrics` module, which provides

    functions for computing various classification metrics. For example, the

    `fiftyone.utils.metrics.binary_classification` function can be used to

    compute precision and recall for binary classification tasks.

You:

    does FiftyOne integrate with Label Studio?

VoxelGPT:

    Yes, FiftyOne integrates with Label Studio, which is an open-source data

    labeling tool with a user-friendly UI. The integration allows you to easily

    upload your data directly from FiftyOne to Label Studio for labeling. You

    can get started with Label Studio through a simple pip install to get a

    local server up and running. FiftyOne provides simple setup instructions

    that you can use to specify the necessary account credentials and server

    endpoint to use. FiftyOne also provides an API to create projects, upload

    data, define label schemas, and download annotations using Label Studio,

    all programmatically in Python.

You:

    show me predicted airplanes

VoxelGPT:

    Identified potential algorithms: evaluation

    No evaluation runs found.

    Identified potential fields: predictions

    Identified potential label classes: {'predictions': ['airplane']}

    Identified potential view stages: ['match', 'filter_labels', 'match_labels', 'exclude_labels', 'filter_field']

    Okay, I'm going to load dataset.filter_labels("predictions",F("label") == "airplane")

You:

    now only show me the first 10 samples

VoxelGPT:

    Identified potential algorithms: evaluation

    No evaluation runs found.

    Identified potential fields: predictions

    Identified potential label classes: {'predictions': ['airplane']}

    Identified potential view stages: ['match', 'limit', 'limit_labels', 'skip', 'sort_by']

    Okay, I'm going to load dataset.match(F("predictions.detections.label").contains("airplane")).limit(10)

You: exit

In interactive mode, VoxelGPT automatically loads any views it creates in the App, and you can access them via your session object:

If you just want to run a single query, you can use ask_voxelgpt():

Yes, FiftyOne integrates with CVAT, which is an open-source image and video

annotation tool. You can upload your data directly from FiftyOne to CVAT to

add or edit labels. FiftyOne provides simple setup instructions that you can

use to specify the necessary account credentials and server endpoint to use.

CVAT provides three levels of abstraction for annotation workflows: projects,

tasks, and jobs.

If you pass a dataset along with your query and VoxelGPT interprets your prompt as a request to load a view into your dataset, it will be returned to you:

From there you can then interact with or further refine the view as you would any other DatasetView in FiftyOne.

VoxelGPT brings the power of large language models to your computer vision datasets. It is entirely open source and you can try it out today at gpt.fiftyone.ai!

If you like VoxelGPT, consider showing your support by giving voxelgpt a star on GitHub. And while you’re at it, give fiftyone a star too; it’s all open source! We appreciate your support 🙂

Join the thousands of engineers and data scientists already using FiftyOne to solve some of the most challenging problems in computer vision today!