A synthetic benchmark built to break foundation models, and a four-stage FiftyOne workflow — curate, annotate, generate, evaluate — for catching the failure before you deploy it.
You’re about to hand a robot a natural-language instruction: “pick the small bolt, not the big one.” Not “pick a bolt.” Not “pick the bolt.” The small one, specifically, is sitting next to a bin full of the wrong one.
That instruction only works if the model underneath it can ground it — lock onto the exact object you meant, not the object that happens to be biggest, or brightest, or most common in the bin. Robotics is currently making two big, correlated bets that this will just work out: that synthetic data can replace the teleop and real-world collection bottleneck, and that foundation models can do open-world perception well enough to follow instructions like that one. Both bets fail quietly. Synthetic pipelines ship artifacts nobody authored. Foundation models report a confident, wrong number. You usually find out after the GPU hours are already burned — or after the robot has grabbed the wrong block.
A new benchmark called KubriCount — introduced in Count Anything at Any Granularity (Liu, Wu & Xie, SJTU) — turns that fuzzy fear into something you can measure. It reframes counting as a “verifiable prompt-following problem”: the question isn’t whether a model can output a plausible number; it’s whether it can count the correct set under an explicit, fine-grained prompt. The paper’s headline finding is blunt — models systematically “default to visually dominant or repetitive groups” when a prompt gets specific. That’s the robot grabbing the wrong block, expressed as a benchmark score.
Counting is just the vehicle here. It’s the most projector-legible, plot-legible version of open-world grounding. The real subject of this post is the workflow: how you use FiftyOne to walk a synthetic dataset and a foundation model’s predictions through one continuous loop — curate → annotate → generate → evaluate — and catch the failure at each stage instead of finding out about it on a leaderboard.
Everything below runs against a 6,736-sample FiftyOne-ready subset of KubriCount, Voxel51/KubriCount on Hugging Face, with predictions from NVIDIA’s open-world grounding model LocateAnything-3B already loaded in. Every number quoted is pulled directly from that dataset — not estimated.
Key Takeaways
KubriCount’s full benchmark is 110,507 synthetic images, ~7.3 million annotated objects, 157 categories across 16 super-categories, built by a fully automatic four-stage pipeline: 3D asset curation (ShapeNetCore-v2 + TRELLIS-generated assets, ~58K assets), Kubric+Blender rendering, Nano-Banana-Pro image editing for sim-to-real realism, and Gemini-3-Pro VLM filtering for quality control.
The dataset defines five explicit counting granularity levels (L1 identity → L5 concept), each pairing a target set with a controlled distractor set that differs by exactly one semantic factor — this is what makes “count the correct set” a verifiable, gradeable task instead of a vibe.
Every ground-truth claim in this post is verified against the live dataset, not assumed: len(sample.example_boxes.detections) == sample.count holds for all 6,736 samples, zero violations — the synthetic labels really are self-consistent.
NVIDIA LocateAnything-3B, evaluated via dataset.evaluate_detections() (COCO protocol, IoU 0.5) against example_boxes, scores precision 0.33 / recall 0.28 on the Level-5 test splits, with 18 of 154 test-split categories (including teddy_bear and saw) sitting at exactly 0.00 recall.
Filtering the evaluated view down to predicted_count ≈ negative_count (and far from the true count) surfaces 94 scenes where the model counted the distractor instead of the target it was prompted for — the exact “wrong block” failure mode, reproducible with a four-line ViewField query.
Coloring precomputed SigLIP embeddings (siglip_viz) by split makes the generalization gap visible without a single line of clustering code: train and testA overlap, testB (entirely novel categories) sits off on its own.
What KubriCount actually is
Each scene in KubriCount is a 1024×1024 synthetic image. Every image is produced by a four-stage automatic pipeline, and it’s worth naming the stages once because they map directly onto the curate → annotate → generate → evaluate loop this post walks through:
3D asset curation. ~58K 3D assets across 157 categories, sourced from ShapeNetCore-v2 (53K assets, 55 categories with clean taxonomic labels) and supplemented with TRELLIS-family controllable 3D generation (~5K more assets, 102 new categories) to fill in the long tail. ~5K HDRI environment maps (Poly Haven + Text2Light) provide lighting diversity.
Prototype synthesis. Kubric + PyBullet + Blender render multi-object scenes with exact instance-level metadata — RGB, instance masks, 2D/3D boxes, center points — all authored by the simulator, not a human annotator. Object counts run 1 → 250 per scene (capped by Kubric’s 256-instance ID limit).
Consistent image editing. Nano-Banana-Pro refines textures and harmonizes lighting to narrow the sim-to-real gap, under a hard constraint: no instance may be added, removed, merged, or split. Whatever the simulator counted remains in the frame after editing.
Automatic quality filtering. Gemini-3-Pro inspects each edited image against the original prototype and masks, issuing a PASS/FAIL verdict. A single pass rejects ~20% of edits; three rounds of iterative re-editing bring the final rejection rate down to ~5%.
Good to know. What are the five granularity levels?
KubriCount’s core idea is to make “what counts as the target” explicit instead of implicit. Each level pairs a target set with an optional distractor set that differs by exactly one semantic factor:
KubriCount's five counting granularity levels. Each pairs a target prompt with a distractor set that differs by exactly one semantic factor.
KubriCount's five counting granularity levels. Each pairs a target prompt with a distractor set that differs by exactly one semantic factor.
Level
Granularity
Prompt example
Distractor
L1
Identity
“Count all the dogs.”
None
L2
Attribute (size/color)
“Count the large cherries.”
Small cherries
L3
Category
“Count the cans.”
Bags
L4
Instance type
“Count backpack A.”
Backpack B
L5
Concept
“Count the lobsters.”
Octopuses
Keep it honest. The full KubriCount benchmark spans all five levels across train (99,639 scenes), testA (5,462 scenes, unseen assets from seen categories), and testB (5,406 scenes, entirely unseen categories) — roughly 110K images and 198K counting queries in total. The 6,736-sample subset this post uses is narrower by design: train here is entirely Level 1 (“count everything,” no distractor), and both testA (924 samples) and testB (812 samples) are entirely Level 5 (the hardest disambiguation level). That’s the deliberate easy-to-hardest arc the rest of this post walks — it is not a representative sample of all five levels, and any number below that says “test” means “Level 5 test,” not “the full evaluation protocol from the paper.”
Get the subset and open it:
The field names you’ll see throughout — verified against the live dataset’s own schema, not guessed:
Fields in the FiftyOne-ready KubriCount subset, verified against the live dataset's own schema.
Fields in the FiftyOne-ready KubriCount subset, verified against the live dataset's own schema.
Field
Type
What it holds
split
StringField
train, testA (novel assets, seen categories), or testB (entirely novel categories)
level
IntField
Granularity level, 1–5 (this subset: 1 and 5 only)
category
StringField
The target prompt — what the model was asked to count
count
IntField
Ground-truth number of target objects
target_points
Keypoints
One center point per target object, len == count by construction
example_boxes
Detections
Ground-truth target bounding boxes, one per target instance
segmentation
Segmentation
Per-instance Kubric-rendered mask
negative_category
StringField
The distractor prompt (empty for L1)
negative_count
IntField
Ground-truth distractor object count
negative_example_boxes
Detections
Ground-truth distractor bounding boxes
locate_anything_pred
Detections
Predictions from NVIDIA LocateAnything-3B, prompted with category
siglip_viz
Embedding coords
2D UMAP projection of a SigLIP image embedding (brain key siglip_viz)
predicted_count
IntField
len(locate_anything_pred.detections)
abs_error
FloatField
\|predicted_count - count\|
Two of those are load-bearing distinctions to keep straight for the rest of this post: example_boxes is ground truth, authored by the simulator. locate_anything_pred is model output — everything that follows about precision, recall, and “the model got it wrong” refers to that field, never to the labels themselves.
Step 1: Curate — open it and look before you burn compute
The first move with any new dataset, synthetic or real, is to open it and look — not read a README, not trust a data card. FiftyOne’s App gives you three views into the data’s shape with zero configuration.
The Histograms panel auto-plots the distribution of whatever field you pick and updates live to the current view — no dashboard code required to get the shape of the data:
count: object density per scene runs from 1 to 248 — these are dense, cluttered scenes by design.
category: 157 distinct target categories in this subset.
level / split: this is the structural fact most people skim past. train (5,000 samples) is entirely Level 1 — “count everything, no distractor.” Both testA (924) and testB (812) are entirely Level 5 — the hardest disambiguation level — with two categories, each containing multiple instance types. The data itself ramps from the easiest possible task to the hardest.
For the visual structure, color the precomputed embeddings — no clustering code needed:
Color the Embeddings panel by category, and the 157 classes are separated by visual appearance. Color it by level, and the L1/L5 split falls out as two visually distinct regions. On a real synthetic-data pipeline, this is the exact panel where you’d notice a category your generator over- or under-produced — before you’ve spent a single GPU-hour training on it.
Step 2: Annotate — trust no label, verify it
Synthetic labels have a reputation for being “free and perfect” — the simulator knows exactly what it rendered, so why would you check? Because “supposed to be perfect” and “verified perfect” are different claims, only one of them is defensible on stage or in a paper.
KubriCount’s design gives you a free invariant to check: every target object gets exactly one ground-truth box, so len(example_boxes.detections) must equal count for every sample, with no exceptions.
Zero violations across all 6,736 samples. That’s not an assumption carried over from a data card — it’s a five-second query against the live dataset. Save it as a view so it’s one click away, not a query you retype live:
That second toggle is the point of Act 2 for a robotics audience: example_boxes is the target you were prompted for, negative_example_boxes is the look-alike you have to ignore. On real robot data, this exact pattern — verify an invariant, then eyeball the ambiguous cases — is how you’d hunt the bad 2% of a labeling run and route the genuinely unclear scenes to a human annotator instead of shipping them straight into training.
Step 3: Generate — find the gap you’d be generating blind into
KubriCount’s test splits are built for exactly this: testA holds out unseen assets from categories the model has already seen in train; testB holds out entire categories the model has never seen at all. Recolor the same embeddings panel by split:
train and testA overlap — same categories, new object instances, no real surprise for the model. testB sits off on its own, visually distinct. That separation is the generalization gap, made visible by a single recolor with no dimensionality-reduction code written live. For a robotics dataset, this is exactly how you’d sanity-check whether your “held-out test set” is actually testing generalization or just re-sampling the training distribution with new random seeds.
Make it concrete with similarity search instead of a 2D scatterplot alone:
That’s the honest version of “how far are you asking the model to extrapolate”: not a number, but the actual nearest neighbor sitting a visible distance away.
Step 4: Evaluate — aggregate metrics lie, so look at the boxes
This is where the foundation-model bet gets tested. The model under test is NVIDIA LocateAnything-3B, an off-the-shelf open-world grounding detector, applied to every sample with category as the text prompt:
Summed across the whole subset, the model returned 233,729 predicted boxes against 334,983 ground-truth boxes — undercounting on average, but that single aggregate hides everything interesting. Run a proper COCO-style detection evaluation against the target boxes instead of eyeballing totals:
Restricted to the Level-5 test splits (testA + testB, 1,736 samples — where the target/distractor disambiguation actually gets hard), that comes out to:
The model finds fewer than a third of the objects it’s supposed to find, and two-thirds of what it does return is wrong. Open the per-class breakdown and it gets worse: 18 of 154 test-split categories sit at exactly 0.00 recall — teddy_bear and saw among them. It’s not that the model is bad at counting in general (Level 1, “count everything,” is comparatively fine). It’s bad at counting the thing you specifically asked for the moment a look-alike distractor is in frame.
The money shot: it counted the distractor
Aggregate precision/recall still doesn’t show you why it fails. The specific failure the paper predicts — models defaulting to the visually dominant or repetitive group — has a signature you can query directly: scenes where predicted_count lands close to negative_count (the distractor’s true count) but far from count (the target’s true count).
94 scenes where the model grounded onto the wrong object entirely. Open one and it’s not subtle. One sample prompts category = "boat" — 6 boats are actually in the scene — while 33 vessel objects (the distractor class) sit in the same frame. The model returned 34 boxes. It didn’t miscount the boats; it counted the vessels and reported that number back as if the prompt had been “count the vessels.” That’s “pick the large red block, not the small one” failing, live, on a current open-world model — and the only reason it’s diagnosable at all is that someone filtered to the actual scenes instead of reading a leaderboard row.
A second, related failure mode is worth a look too: scenes where the model counted target and distractor together, with no disambiguation at all —
The loop doesn’t end at evaluation
Four stages, but really one loop: curate, annotate, generate, evaluate, and back to curate again with what the evaluation just told you. The 94 scenes in counted_the_distractor aren’t just a punchline — they’re a work order. They tell you exactly which category pairs (boat/vessel, pillow/clock, spoon/candle) need more or harder synthetic examples next, which labels need a second look, and which embedding regions to inspect for more of the same failure.
FiftyOne doesn’t collect your data, train your policy, or close your robot’s control loop. It’s the layer in between — for synthetic data, for real data, and for foundation-model failures alike, whether the failure mode is “counted the wrong thing” or “picked the wrong block.” You can’t fix what you can’t see. So look first.
Not in this subset. The general KubriCount project describes example_boxes upstream as a small set of few-shot exemplar crops (2–8 per scene). In this FiftyOne-ready subset, example_boxes has been verified directly against the live data to be exhaustive, one-per-instance ground truth (len(example_boxes.detections) == count for all 6,736 samples) — that’s what makes it usable as detection-evaluation ground truth in the steps above. If you’re working from a different KubriCount release, check the field’s description in the FiftyOne App sidebar before assuming either behavior.
That’s a deliberate choice for this particular subset, not a property of the full benchmark. The full KubriCount spans all five levels in every split (~100K train images, ~5K each in testA/testB). This subset selects the two extremes — the easiest task for training exposure and the hardest task for evaluation — to make the easy-to-hard arc legible in a small sample. If you need the middle levels (L2–L4), pull the full dataset from liuchang666/KubriCount.
No — those are computed in this post via dataset.evaluate_detections() (COCO protocol, IoU 0.5) comparing NVIDIA LocateAnything-3B’s locate_anything_pred field against the example_boxes ground truth, on this 1,736-sample Level-5 test slice. The paper’s own benchmark table evaluates a different, larger set of MLLMs and specialist counting models (MAE/RMSE protocol) on the full benchmark and does not include LocateAnything-3B. Don’t cite 0.33/0.28 as a paper result — it’s this post’s own model-in-FiftyOne evaluation.
abs_error is a scalar, |predicted_count - count| — how far off the total object count was, with no notion of which boxes were right. eval_boxes_tp/eval_boxes_fp/eval_boxes_fn come from the box-level COCO evaluation and tell you whether the individual predicted boxes actually matched ground-truth boxes (by IoU) — which is what lets you compute precision/recall and catch the “right count, wrong objects” failure mode that a bare count-error metric would miss entirely.
Yes — the App’s grid and panels paginate server-side, so millions of samples behave the same way interactively. This post uses the 6,736-sample subset specifically because it’s the one with precomputed embeddings, predictions, and evaluation results ready to go; the full benchmark is ~110,507 images and ~7.3 million annotated objects.
