FiftyOne Environments¶
This guide describes best practices for using FiftyOne with data stored in various environments, including local machines, remote servers, and cloud storage.
Terminology¶
Local machine: Data is stored on the same computer that will be used to launch the App
Remote machine: Data is stored on disk on a separate machine (typically a remote server) from the one that will be used to launch the App
Cloud storage: Data is stored in a cloud bucket (e.g., S3, GCS, or Azure)
Local data¶
When working with data that is stored on disk on a machine with a display, you can directly load a dataset and then launch the App:
1 2 3 4 5 6 | # On local machine
import fiftyone as fo
dataset = fo.Dataset(name="my_dataset")
session = fo.launch_app(dataset) # (optional) port=XXXX
|
From here, you can explore the dataset interactively from the App and from your
Python shell by manipulating the
session object.
Note
You can use custom ports when launching the App in order to operate multiple App instances simultaneously on your machine.
Remote data¶
FiftyOne supports working with data that is stored on a remote machine that you
have ssh access to. The basic workflow is to load a dataset on the remote
machine via the FiftyOne Python library, launch a
remote session, and connect to the session on your
local machine where you can then interact with the App.
First, ssh into your remote machine and
install FiftyOne if necessary.
Then load a dataset using Python on the remote machine and launch a remote session:
1 2 3 4 5 6 | # On remote machine
import fiftyone as fo
dataset = fo.Dataset(name="my_dataset")
session = fo.launch_app(dataset, remote=True) # (optional) port=XXXX
|
Leave this session running, and note that instructions for connecting to this remote session were printed to your terminal (these are described below).
On the local machine, you can use the CLI to automatically configure port forwarding and open the App.
In a local terminal, run the command:
# On local machine
fiftyone app connect --destination <user>@<remote-ip-address> --port 5151 # (Optional) --ssh-key /path/to/key
Note
If you are using ssh keys instead of a password to login then you
can use the kwarg --ssh-key. Though if you are using this key
more often, it is recommended to add it to your ~/.ssh/config as
the default IdentityFile.
The above instructions assume that you used the default port 5151 when
launching the remote session on the remote machine. If you used a custom port,
then substitute the appropriate value in the local commands too.
Note
You can use custom ports when launching remote sessions in order to serve multiple remote sessions simultaneously.
Cloud storage¶
FiftyOne does not yet support accessing data directly in a cloud bucket. Instead, the best practice that we recommend is to mount the cloud bucket as a local drive on a cloud compute instance.
The following sections describe how to do this in the AWS, Google Cloud, and Miscrosoft Azure cloud environments.
Amazon Web Services¶
If your data is stored in an AWS S3 bucket, we recommend mounting the bucket as a local drive on an EC2 instance and then accessing the data using the standard workflow for remote data.
The steps below outline the process.
Step 1
Create an EC2 instance. We recommend a Linux instance.
Step 2
Now ssh into the instance and install FiftyOne if necessary.
# On remote machine
pip install --index https://pypi.voxel51.com fiftyone
Note
You may need to install some system packages on your compute instance instance in order to run FiftyOne.
Step 3
Mount the S3 bucket as a local drive.
We recommend using s3fs-fuse for
this. You will need to make a .passwd-s3fs file that contains your AWS
credentials as outlined in the
s3fs-fuse README.
# On remote machine
s3fs <bucket-name> /path/to/mount/point \
-o passwd_file=.passwd-s3fs \
-o umask=0007,uid=<your-user-id>
Step 4
Now that you can access your data from the compute instance, start up Python and create a FiftyOne dataset whose filepaths are in the mount point you specified above. Then launch the App as a remote session:
1 2 3 4 5 6 | # On remote machine
import fiftyone as fo
dataset = fo.Dataset(name="my_dataset")
session = fo.launch_app(dataset, remote=True) # (optional) port=XXXX
|
Step 5
Finally, on your local machine, connect to the remote session that you started on the cloud instance.
# On local machine
fiftyone app connect --destination <user>@<remote-ip-address> --port 5151 # (Optional) --ssh-key /path/to/key
The above instructions assume that you used the default port 5151 when
launching the remote session on the remote machine. If you used a custom port,
then substitute the appropriate value in the local commands too.
Note
If you are using ssh keys instead of a password to login then you
can use the kwarg --ssh-key. Though if you are using this key
more often, it is recommended to add it to your ~/.ssh/config as
the default IdentityFile.
Google Cloud¶
If your data is stored in a Google Cloud storage bucket, we recommend mounting the bucket as a local drive on a GC compute instance and then accessing the data using the standard workflow for remote data.
The steps below outline the process.
Step 1
Create a GC compute instance. We recommend a Linux instance.
Step 2
Now ssh into the instance and install FiftyOne if necessary.
# On remote machine
pip install --index https://pypi.voxel51.com fiftyone
Note
You may need to install some system packages on your compute instance instance in order to run FiftyOne.
Step 3
Mount the GCS bucket as a local drive.
We recommend using gcsfuse to do this:
# On remote machine
gcsfuse my-bucket /path/to/mount --implicit-dirs
Step 4
Now that you can access your data from the compute instance, start up Python and create a FiftyOne dataset whose filepaths are in the mount point you specified above. Then launch the App as a remote session:
1 2 3 4 5 6 | # On remote machine
import fiftyone as fo
dataset = fo.Dataset(name="my_dataset")
session = fo.launch_app(dataset, remote=True) # (optional) port=XXXX
|
Step 5
Finally, on your local machine, connect to the remote session that you started on the cloud instance.
# On local machine
fiftyone app connect --destination <user>@<remote-ip-address> --port 5151 # (Optional) --ssh-key /path/to/key
The above instructions assume that you used the default port 5151 when
launching the remote session on the remote machine. If you used a custom port,
then substitute the appropriate value in the local commands too.
Note
If you are using ssh keys instead of a password to login then you
can use the kwarg --ssh-key. Though if you are using this key
more often, it is recommended to add it to your ~/.ssh/config as
the default IdentityFile.
Microsoft Azure¶
If your data is stored in an Azure storage bucket, we recommend mounting the bucket as a local drive on an Azure compute instance and then accessing the data using the standard workflow for remote data.
The steps below outline the process.
Step 1
Create an Azure compute instance. We recommend a Linux instance.
Step 2
Now ssh into the instance and install FiftyOne if necessary.
# On remote machine
pip install --index https://pypi.voxel51.com fiftyone
Note
You may need to install some system packages on your compute instance instance in order to run FiftyOne.
Step 3
Mount the Azure storage container in the instance.
This is fairly straight forward if your data is stored in a blob container. We recommend using blobfuse for this.
Step 4
Now that you can access your data from the compute instance, start up Python and create a FiftyOne dataset whose filepaths are in the mount point you specified above. Then launch the App as a remote session:
1 2 3 4 5 6 | # On remote machine
import fiftyone as fo
dataset = fo.Dataset(name="my_dataset")
session = fo.launch_app(dataset, remote=True) # (optional) port=XXXX
|
Step 5
Finally, on your local machine, connect to the remote session that you started on the cloud instance.
# On local machine
fiftyone app connect --destination <user>@<remote-ip-address> --port 5151 # (Optional) --ssh-key /path/to/key
The above instructions assume that you used the default port 5151 when
launching the remote session on the remote machine. If you used a custom port,
then substitute the appropriate value in the local commands too.
Note
If you are using ssh keys instead of a password to login then you
can use the kwarg --ssh-key. Though if you are using this key
more often, it is recommended to add it to your ~/.ssh/config as
the default IdentityFile.
Setting up a cloud instance¶
When you create a fresh cloud compute instance, you may need to install some system packages in order to install and use FiftyOne.
For example, the script below shows a set of commands that may be used to configure a Debian-like Linux instance, after which you should be able to successfully install FiftyOne.
# Example setup script for a Debian-like virtual machine
# System packages
sudo apt update
sudo apt -y upgrade
sudo apt install -y build-essential
sudo apt install -y unzip
sudo apt install -y cmake
sudo apt install -y cmake-data
sudo apt install -y pkg-config
sudo apt install -y libsm6
sudo apt install -y libxext6
sudo apt install -y libssl-dev
sudo apt install -y libffi-dev
sudo apt install -y libxml2-dev
sudo apt install -y libxslt1-dev
sudo apt install -y zlib1g-dev
sudo apt install -y python3
sudo apt install -y python-dev
sudo apt install -y python3-dev
sudo apt install -y python3-pip
sudo apt install -y python3-venv
sudo apt install -y ffmpeg # if working with video
# (Recommended) Create a virtual environment
python3 -m venv fiftyone-env
. fiftyone-env/bin/activate
# Python packages
pip install --upgrade pip setuptools wheel
pip install ipython