Skip to content
Close Modal

VAND 3.0 – Challenge at CVPR 2025

Latest News

⚠️  This Challenge page is still under construction. A lot of details haven’t been defined yet!

⚠️  Register to the Challenge by March 14th and stay tuned for updates

⚠️  Any questions – visit our Discord Channel #cvpr-challenge-vand3-0

 

About the Challenge

Welcome to the Visual Anomaly and Novelty Detection 2025 Challenge, VAND 3rd Edition workshop at CVPR 2025! Our workshop challenge aims to showcase current progress in anomaly detection across different practical settings whilst addressing critical issues in the field.

Despite promising results from previous years, there remains significant room for improvement in developing robust and generalizable anomaly detection models for industrial use cases. Our challenge aims to improve previous submissions by addressing industry-relevant issues in practical anomaly detection.

We warmly invite participants from both academia and industry to collaborate and innovate. Voxel51, Intel, and MVTec proudly sponsor this challenge and aim to encourage solutions that demonstrate robustness across varying conditions and adaptability to real-world variability.

 

Challenge Objective and Categories

Focusing on real-world visual anomaly detection applications, particularly in industrial visual inspection, this year’s challenge is to advance robust and generalizable anomaly detection methods.

These challenges address critical industrial needs for reliable anomaly detection under varying conditions and with limited data. we aim to bridge academic research with industrial requirements to develop solutions directly applicable to manufacturing, healthcare, and beyond. Participants can enter one or both categories.

  • Category 1 — Adapt & Detect: Robust Anomaly Detection in Real-World Applications
  • Category 2 — VLM Anomaly Challenge: Few-Shot Learning for Logical and Structural Detection

Participants can choose a category or enter both in two separate submissions. These challenge categories aim to advance existing anomaly detection literature and increase its adaptation in real-world settings. We invite the global community of innovators, researchers, and technology enthusiasts. Engage with these challenges and contribute towards advancing anomaly detection technologies in real-world scenarios.

From April 7th to May 26th, 2025, this global community will showcase its ideas on how to solve these challenges in visual anomaly detection.

For more information about the submission and the challenge, please visit the workshop web page or the Discord Channel of this Challenge.

 

Category 1 — Adapt & Detect: Robust Anomaly Detection in Real-World Applications

Participants will develop anomaly detection models that demonstrate robustness against external factors and adaptability to real-world variability. Many existing anomaly detection models, trained on normal images and validated against normal and abnormal images, often struggle with robustness in real-world scenarios due to data drift caused by external changes such as camera angles, lighting conditions, and noise.

Dataset

Participants will use the novel MVTec Anomaly Detection 2 (MVTec AD 2) dataset, which will be released shortly before the start of the challenge. MVTec AD 2 is a public anomaly detection benchmark dataset that follows the design of previous popular anomaly detection datasets like MVTec AD or VisA. In particular, it contains anomaly-free images for training and validation and both anomaly-free and anomalous images for testing.

However, MVTec AD 2 aims to bridge academic research with industrial requirements in two ways. First, it contains 8 new challenging real-world scenarios captured under varying lighting conditions to reflect real-world distribution shifts. Second, the ground truth of the official test set is non-public to emphasize the unsupervised nature of industrial anomaly detection, i.e. not knowing which defects to expect at inference time. For development purposes, a small set of normal and anomalous test images with public ground truth is included in the dataset download.

For more information on MVTec AD 2 please refer to the arXiv preprint: Available on April 1st.

Model

Participants are encouraged to develop models based on the one-class training paradigm, which is training exclusively on normal images. These models are then validated and tested on a mix of normal and abnormal images to assess their anomaly detection capabilities. The focus is on enabling these models to effectively identify deviations from normality, emphasizing the real-world applicability of the techniques developed.

Evaluation

Evaluation happens on pixel level F1 scores (SegF1). This approach ensures a balanced consideration of precision and recall in the models’ anomaly detection performance. Besides, it requires to select a single threshold for the usually continuous anomaly maps – a challenge often not yet considered within the scientific community but indispensable for deployment in real-world applications.

The final metric to assess model performance in Category 1 of the VAND 2025 Challenge considers the overall performance as well as robustness against real-world distribution shifts. It is computed as the average rank of a model on the private and private_mixed test set in terms of the average SegF1 over all 8 object categories of MVTec AD 2:

final_model_rank = (rank(SegF1[‘private’]_average) + rank(SegF1[‘private_mixed’]_average)) / 2

where

SegF1[‘private’] is the model performance on the test set that contains normal and anomalous images captured under the same lighting conditions as the training images (private test set)

SegF1[‘private_mixed’] is the model performance on the test set that contains normal and anomalous images captured under a variety of lighting conditions both seen and unseen in the training images (private_mixed test set)

The final_model_rank will be determined at the end of the challenge for all valid submissions to the MVTec Benchmark Server (see ‘’Submission Platform”)

Submission Platform: MVTec Benchmark Server

https://benchmark.mvtec.com/

(Online at the start of the challenge on April 7th)

The MVTec Benchmark Server serves as the official leaderboard for the MVTec AD 2 dataset and as the submission platform for Category 1 of the Visual Anomaly and Novelty Detection 2025 Challenge.

To submit to Category 1, you need to upload your model predictions (anomaly images + thresholded anomaly images) in the following way

  • Navigate to the ‘SUBMIT’ section of the MVTec Benchmark Server (https://benchmark.mvtec.com/submit).
  • Click ‘LOGIN’ to login to the submission system/create an account.
  • Enter your university or organization email.
  • Login to an existing account /create a new account.
  • Follow the detailed submission instructions now visible to you.
  • Create a submission that meets all the requirements to participate in Category 1 of the VAND 2025 challenge (editing an evaluated submission is possible except for the uploaded data).
  • As soon as the evaluation has finished, your submission will be visible in the leaderboard (you can track the state of your submission under ‘my submissions’). Please note that the final_model_rank for the VAND2025 challenge is not displayed directly and will be computed after the end of the challenge for all valid submissions (see SUBMISSION REQUIREMENTS below)

More information can be found in the Frequently-Asked-Question (FAQ) section of the MVTec Benchmark.

Submission Requirements

To participate in the VAND 2025 challenge via the MVTec Benchmark Server:

  • [R1] Begin your method name by “VAND2025” followed by your personal method name, e.g., “VAND2025 MyMethodName”.
  • [R2] Include the continuous AND the thresholded anomaly images in your submission.
  • [R3] Open-source your code via specifying a ‘Project link’.
  • [R4] Provide a link to a technical report describing your method that is publicly accessible (e.g. arXiv) via ‘Pub link’. Specify the title and the authors of your publication.

Please note that only submissions meeting all four criteria will be considered as valid submissions for Category 1 of the VAND 2025 challenge. It is possible to edit the method name [R1] and to add/edit the link to the project [R3] and to the technical report [R4] after the successful evaluation of a submission.

Evaluation Budget

The evaluation budget per account is limited. This means that one participant is only allowed to make a certain number of submissions (= uploads) within a specific time. Currently, this limit is set to 2 submissions per 30 days.

This setting avoids extensive hyperparameter tuning on the official test data of MVTec AD 2 (private and private_mixed test set). It highlights the concept of unsupervised anomaly detection, i.e., not knowing which defects and test data to expect. A small set of standard and anomalous test images with public ground truth is included in the dataset download (public test set) for development purposes.

We will freeze the MVTec benchmark leaderboard at the end of the challenge (May 26th, 11:59 pm AOE) and disable new submissions and editing submissions until the evaluation of Category 1 is completed. We will then filter for valid submissions according to the submission requirements, identify the best-performing methods, and notify the winners via email. Submissions successfully uploaded by the end of the challenge will still be evaluated and considered for the final evaluation.

Category 2 — VLM Anomaly Challenge: Few-Shot Learning for Logical and Structural Detection

Participants will create models using few-shot learning and VLMs to find and localize structural and logical anomalies in the MVTec LOCO AD dataset, which contains images of different industrial products showing both defects. This indicates that the models can handle structural defect detection and logical reasoning.

With the development of vision language models (VLMs), finding anomalies could reach an exciting new level, such as detecting logical anomalies that require more than identifying structural defects.

Model

Participants can pre-train their models on any public dataset except the MVTec LOCO dataset, ensuring the challenge focuses on few-shot learning capability.

Dataset

This challenge uses the MVTec LOCO AD dataset. This dataset contains images of different industrial products, showing structural and logical anomalies.

For each few-shot learning scenario, k normal images are sampled randomly from the train set of the MVTec LOCO dataset. We will explore scenarios where k = 1, 2, 4, and 8 with the randomly selected samples provided by the organizing committee.

Additionally, if participants use text prompts within the model, they can include the name of the dataset category in their prompts.

Evaluation

We will follow last year’s evaluation criteria, outlined here:

The evaluation metric for each k-shot setup in the MVTec LOCO subset will be the F1-max score for the anomaly classification task.

We will perform three random runs using the pre-selected samples for each k-shot scenario in a particular subset. These runs will be averaged and assessed.

The arithmetic mean of the averaged metrics is the evaluation measure for a k-normal-shot setup across each category.

We will evaluate the effectiveness of few-shot learning algorithms by plotting the F1-max curve. This shows the F1-max scores in relation to the k-shot number. The ultimate evaluation metric will be the area under the F1-max curve (AUFC).

Submission Platform and Requirements

⚠️  TBD

General Aspects

Model Development

Participants are encouraged to explore and leverage state-of-the-art anomaly detection models without limitations. Creativity and originality in model architecture and training methodology are strongly encouraged.

Description coming from Anomalib

Datasets

For Category 1 (Adapt & Detect), the novel MVTec AD 2 dataset will be used. Its design allows for evaluating models under real-world distribution shifts induced by changes in lighting conditions. For further information, please refer to the detailed description of Category 1.

For Category 2 (VLM Anomaly Challenge), the MVTec LOCO AD dataset will be used. Its design allows for evaluating models not only on structural but also on logical defects, i.e., violating logical constraints. For further information, please refer to the detailed description of Category 2.

Resources

  • Challenge Registration Page – TBD
  • Discord Channel #cvpr-challenge-vand3-0
  • Support materials – Notebooks and blog to navigate this challenge
  • MVTec Datasets
  • FiftyOne Documentation
  • Anomalib GitHub

Participation Guidelines

  • Eligibility: This challenge is open to individuals, teams, and academic and corporate entities worldwide.
  • Registration: Contestants interested int any time between March 14th and May 26th, 2025. We encourage participants to register early to allow more preparation time, and participants will receive important information via email. Voxel51 page TBD
  • General Submission Requirements: Details on the submission format are outlined in every single track above in this page

Evaluation Criteria

  • Reproducibility: If judges cannot reproduce the submission, the submission is disqualified.
  • Evaluation metrics (100): Each category has separate evaluation metrics, which are the unique criteria that judges will follow to evaluate.
  • For specific details, please refer to the evaluation section of the corresponding category.

Important Dates

  • Challenge time: April 7th – May 26th
  • Registration Opens: March 14th
  • Dataset Release: April 1st
  • Submission Deadline: May 26th
  • Results Announcement: June 3rd

Prizes

Participate for a chance to win prizes! Prizes range from monetary presentations at the CVPR workshop and opportunities for collaboration.

Grand Prize for Each Category

  • TBD
  • An invitation to present their solutions during our workshop
  • The winner will have the chance to publish a blog detailing their results and approach on the Voxel51 webpage. Be part of the Voxel51 meet-up as a speaker.

Second Place Prize for Each Category

  • TBD
  • The winner will have the chance to publish a blog detailing their results and approach on the Voxel51 webpage. Be part of the Voxel51 meet-up as a speaker.

Sponsors

If you want to be part of the sponsors, please send me a LinkedIn Message: https://www.linkedin.com/in/paula-ramos-phd/