What is near-duplicate detection?
Near-duplicate detection identifies samples that are the same or almost the same, going beyond exact byte-for-byte copies to catch resized, recompressed, cropped, or slightly edited versions. In large real-world datasets these duplicates accumulate quietly, and they distort both training and evaluation.
It is usually done by comparing embeddings or perceptual hashes, so that visually similar items land close together even when their files differ.
Key takeaways
- It finds identical and near-identical samples, not just exact copies.
- It typically compares embeddings or perceptual hashes.
- Removing duplicates prevents train-test leakage and wasted labeling.
How it works
Each sample is turned into a compact representation, such as an embedding or a hash, and pairs that fall within a small distance are flagged as near-duplicates. A threshold controls how aggressive the matching is. Flagged groups can then be reviewed and reduced to a single representative.
Why it matters
Duplicates inflate dataset size without adding information, waste money when they are labeled repeatedly, and, most dangerously, cause leakage when the same item appears in both training and test sets, which quietly inflates reported performance. Catching them is a foundational data-quality step.
Frequently asked questions
Why are near-duplicates a problem?
They waste labeling budget and, when they span training and test sets, leak information that inflates evaluation scores.
How is near-duplicate detection different from deduplication?
Detection finds the near-identical groups, and deduplication is the follow-on step of removing the redundant copies.
Related terms