Near-duplicate detection
Near-duplicate detection finds samples in a dataset that are identical or nearly identical, such as the same image resaved or lightly edited. Removing them cuts redundancy, saves labeling effort, and prevents leakage between training and test sets.
In this article
Building visual or physical AI?
Let's talk.
What is near-duplicate detection?
Near-duplicate detection identifies samples that are the same or almost the same, going beyond exact byte-for-byte copies to catch resized, recompressed, cropped, or slightly edited versions. In large real-world datasets these duplicates accumulate quietly, and they distort both training and evaluation.
It is usually done by comparing embeddings or perceptual hashes, so that visually similar items land close together even when their files differ.
Key takeaways
- It finds identical and near-identical samples, not just exact copies.
- It typically compares embeddings or perceptual hashes.
- Removing duplicates prevents train-test leakage and wasted labeling.
How it works
Each sample is turned into a compact representation, such as an embedding or a hash, and pairs that fall within a small distance are flagged as near-duplicates. A threshold controls how aggressive the matching is. Flagged groups can then be reviewed and reduced to a single representative.
Before you split a dataset, find the images that are almost the same
Near-duplicates are images that aren't identical files but show the same thing: burst shots, resized copies, or one scene with a different crop. They make a dataset look bigger without adding information, and they leak between training and test splits. The quiet problems:
- a test score that's high because the test set holds copies of training images
- one scene repeated so often that the model learns its background
- a file-hash check that misses copies that were resized or cropped
In Voxel51, Open Images 10K comes with embeddings, so you can run Find similar on any image and see its nearest neighbors.
Open Images 10K
10,000 images
Open Images V7 images balanced across 20 everyday classes, with boxes, instance masks, image labels, points, and relationships.
Try this: Pick an image, run Find similar, and check whether the closest results are different images or near-copies of the same scene.
Open the images in Voxel51
Free with a work or .edu email.
Why it matters
Duplicates inflate dataset size without adding information, waste money when they are labeled repeatedly, and, most dangerously, cause leakage when the same item appears in both training and test sets, which quietly inflates reported performance. Catching them is a foundational data-quality step.
For how near-duplicates leak between train and test splits, see data leakage and near-duplicate splits.
Frequently asked questions
Why are near-duplicates a problem?
They waste labeling budget and, when they span training and test sets, leak information that inflates evaluation scores.
How is near-duplicate detection different from deduplication?
Detection finds the near-identical groups, and deduplication is the follow-on step of removing the redundant copies.
Related terms
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 11, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved