Anomaly detection (AD) is crucial in mission-critical applications such as fraud detection, network security, and medical diagnosis. Anomaly detection in visual data like images, videos, and satellite imagery is particularly challenging due to the high dimensionality of the data and the complexity of the underlying patterns. Yet visual anomaly detection is essential for detecting defects in manufacturing, identifying suspicious activity in surveillance footage, and detecting abnormalities in medical images. In this post, you'll learn how to perform anomaly detection on visual data using
FiftyOne and
Anomalib from the
OpenVINO toolkit. For demonstration, we'll use the
MVTec AD anomaly dataset, which contains visual anomaly data including images of various objects with anomalies like scratches, dents, and holes. This anomalib tutorial covers the following:
- Loading the MVTec AD dataset in FiftyOne
- Training an anomaly detection model with Anomalib
- Evaluating anomaly detection models in FiftyOne
Anamalib Setup | Anomalib Tutorial
Installing dependencies
Make sure you are running this in a virtual environment with python=3.10. Anomalib requires Python 3.10, so make sure you have the correct version installed.
After this, install Anomalib from source, per the instructions in the
Anomalib README, and its dependencies. These might take a moment on Google Colab but should be quick for local installs:
We're ready to go with a few more packages to install. Now you can see why we recommend using a virtual environment for this project!
huggingface_hub for loading the MVTec AD datasetclip for computing image embeddingsumap-learn for dimensionality reduction
The MVTec AD Anomaly Dataset | Anomalib Tutorial
Now, let's import all of the relevant anomaly dataset modules we will need from FiftyOne:
Before moving on, let's take a look at the anomaly dataset in the
FiftyOne App:
Visualizing the Anomaly Dataset | Anomalib Tutorial
The anomaly detection dataset has 5354 images across 12 object categories. Each category has "good" and "anomaly" images with defects like scratches, dents, and holes. Each anomaly picture also has a mask which localizes the defective regions of the image. The defect labels differ across categories, which is typical in real-world anomaly detection scenarios. In these scenarios, you train a different anomaly detection model for each category. Here, we'll go through the process for one category, and you can apply the same steps to other categories. One more thing to note is that the dataset is split into anomaly datasets for training and test. The training set contains only "good" images, while the test set contains both "good" and "anomalous" images. Before we train a model, let's dig into this particular anomaly detection dataset more. We can get a feel for the structure and patterns hidden in our data by computing image embeddings and visualizing them in a lower-dimensional space. First, we'll compute embeddings for all the images in the dataset using the
CLIP model:
Refresh the FiftyOne App, click the "+" tab, and select "Embeddings". Choose "all_clip_vis" from the dropdown menu. You'll see a scatter plot of the image embeddings in a 2D space, where each point corresponds to a sample in the dataset.
Using the color-by dropdown, notice how the embeddings cluster based on the object category. This is because CLIP encodes semantic information about the images. Also, CLIP embeddings don't cluster within a category based on the defect type.
Training an Anomaly Detection Model | Anomalib Tutorial
Now that we have a sense of the anomaly dataset, we're ready to train an anomaly detection model using Anomalib.
Task: Anomalib supports classification, detection, and segmentation tasks for images. We'll focus on segmentation, where the model predicts whether each pixel in the image is anomalous, creating a mask that localizes the defect.
Model: Anomalib supports a
variety of anomaly detection algorithms. For this Anomalib tutorial, we'll use two algorithms:
Preprocessing: We will resize the images to 256x256 pixels for this walkthrough before training the model. Adding this as a transform via Torchvision's Resize class lets us resize the images on the fly during training and inference. Import the necessary modules from Anomalib and helper modules for processing images and paths:
Now, define some constants to use throughout the notebook.
OBJECT: The object category we'll focus on. For this walkthrough, we'll use "bottle". If you want to loop over categories, you can get the list of categories from the dataset with the dataset.distinct("category.label").ROOT_DIR: The root directory where Anomalib will look for images and masks. Our data is already stored on disk, so we will just symlink files to the directory Anomalib expects.TASK: The task we're performing. We'll use "segmentation" for this walkthrough.IMAGE_SIZE: The size to resize images to before training the anomaly detection model. We'll use 256x 256 pixels.
For a given object type (category), the create_datamodule() function below creates an Anomalib DataModule object. This will be passed into our engine's fit()method to train the model and used to instantiate dataloaders for training and validation. The code might look complex, so let's break down what's going on:
- We create subsets of our data containing only the "good" training images and "anomalous" images or anomaly picture sfor validation.
- We symlink the images and masks to the directory Anomalib expects.
- We instantiate and set up a datamodule from Anomalib's
Folder, which is the general-purpose class for custom datasets.
💡 It is also possible to create a torch DataLoader from scratch and pass it to the engine's fit() method. This gives you more control over the data loading process. This is left as an exercise for the reader 😉.
Now, we can put it all together. The train_and_export_model() function below trains an anomaly detection model using Anomalib's Engine class, exports the model to OpenVINO, and returns the model "inferencer" object. The inferencer object is used to make predictions on new images.
Let's try this with PaDiMe first. The training process should take less than a minute:
And just like that, we have an anomaly detection model trained on the "bottle" category. Let's run our inferencer on a single image and inspect the results:
The output contains a scalar anomaly score pred_score, a pred_mask denoting the predicted anomalous regions, and a heatmap anomaly_map showing the anomaly scores for each pixel. This is all valuable information for understanding the model's predictions. The run_inference() function below will take a FiftyOne sample collection (e.g. our test set) as input, along with the inferencer object and a key for storing the results in the samples. It will run the model on each sample in the collection and store the results. The threshold argument acts as a cutoff for the anomaly score. If the score is above the threshold, the sample is considered anomalous. In this example, we'll use a threshold of 0.5, but you can experiment with different values.
Let's run inference on our test split and visualize the results in the FiftyOne App:
Evaluating Anomaly Detection Models | Anomalib Tutorial
We have an anomaly detection model, but how do we know if it's good? For one, we can evaluate the model using precision, recall, and F1 score metrics. FiftyOne's
Evaluation API makes this easy. We are going to evaluate the full-image classification performance of the anomaly detection model, as well as the segmentation performance. We need to prepare our data for evaluation. First, we need to add null masks for the "normal" images to ensure the evaluation is fair:
We also need to ensure consistency in naming/labels between ground truth and predictions. We'll rename all of our "good" images to "normal" and every type of anomaly to "anomaly":
For classification, we'll use binary evaluation, with "normal" as the negative class and "anomaly" as the positive class:
The anomaly detection model we've trained performs quite well on the classification task. If we go back to the app and sort by anomaly score, we can see that certain anomalies tend to have higher scores than others. In this example, contamination instances tend to have very high or low scores relative to broken_small and broken_large. When we put this model in production, we might be more likely to miss certain anomalies. Other anomaly detection models, or ensembles of models, might be more robust to this! For segmentation evaluation, we will only be interested in pixel values of 0 (normal) and 255 (anomaly), so we will filter our report for these "classes":
Comparing Anomaly Detection Models | Anomalib Tutorial
Just because anomaly detection is unsupervised doesn't mean we can't compare models and choose the best one for our use case. We can train multiple models on the same data and compare their performance using metrics like F1 score, precision, and recall. We can also compare the models visually by inspecting the masks and heatmaps they generate. Let's repeat the training process for the PatchCore model and compare the two models:
The metrics back up what we see in the app: PatchCore has much higher recall for the "anomaly" class but lower precision. This means it's more likely to catch anomalies but also more likely to make false positive predictions. After all, PatchCore is designed for "total recall" in industrial anomaly detection.
Looking at the heatmaps, we can also see what types of anomalies each anomaly detection model is better at detecting. An ensemble of the two models might be more robust to different types of anomalies.
Creating Your Own Anomaly Detection Models
In this walkthrough, we learned how to perform anomaly detection on visual data using FiftyOne and Anomalib. While we trained two anomaly detection models, we only scratched the surface of what is possible with visual anomaly detection. If you want to improve performance, there are many other knobs you can turn:
- Algorithm: We only used PaDiM and PatchCore. Anomalib currently supports 13 algorithms!
- Backbone: The architecture of the model used for feature extraction
- Hyperparameters: Parameters specific to the anomaly detection algorithm. For PatchCore, this includes
coreset_sampling_ratio and num_neighbors. - Data augmentation: Techniques to artificially increase the size of the training set and improve the model's generalization.
- Custom Solutions: It’s never too late to introduce a new algorithm/technique!
If you want to give any of these a go, the
Visual Anomaly Detection (VAND) 2.0 Challenge is currently underway! This challenge, connected to the upcoming CVPR 2024 conference, is hosted by Intel and leverages both the MVTec AD dataset and Anomalib. Submit a model and you could win a brand new
Intel AI PC. If your solution also leverages FiftyOne, our team at Voxel51 will send you some additional swag 😉
