Data curation is often described as drudgery, but an agent can turn it into an interactive, puzzle-solving process. This talk walks through a data-centric computer vision workflow built on the FiftyOne platform and the FiftyOne Agent, which uses skills (step-by-step instructions for tasks) and the Model Context Protocol (MCP) to handle dataset management, model inference, and brain computations through natural language — querying metadata, visualizing class distributions, and running complex operations without writing manual code.
The demo applies the workflow to a crop and weed dataset of 8,034 images and more than 112,000 labels (bounding boxes, keypoints, and segmentation masks). The agent diagnoses severe class imbalance and duplicate frames, builds interactive histograms and dashboards to surface domain shifts like lighting variation, finds invalid annotations so they can be corrected directly in the FiftyOne App, then automates exporting the curated dataset, fine-tuning a YOLO model, and evaluating its performance. The result is a repeatable pattern — ingest, explore, improve, fine-tune, evaluate, iterate — with a human in the loop through an intuitive, agentic interface.
Resources