Automotive AI teams are under relentless pressure to get new and improved capabilities on the road, and the clock keeps ticking like a stuck blinker. Maybe the press release announcing self-parking has already gone out, maybe the drivetrain team is blocked, maybe leadership keeps swinging by “to see how things are going.” Or maybe something scary or unfortunate has just happened on the road….
Whether teams are working on adaptive cruise control, collision avoidance, sensor obstruction detection, in-cabin distracted driver monitoring or any of the other AI-powered systems that are critical to the modern and future automobile, projects are at risk of drifting off the road to success because of blindspots in their AI development.
An AI Horror Story: Ghost Potholes and the Mystery Model Failure
To illustrate, consider this scenario from conversations we’ve had with people on the front lines.
One day you wake up to an alert: the lane-following software that your team is responsible for is suddenly struggling – it’s almost like there are “ghost” potholes in the street shrinking the polyline detections used in your driveable area segmentation model, making your lane-following performance way below standard.
The team is scrambling to understand what happened. Experiment tracking tools aren’t helping – they can find out that models are failing but not where or why. The model weights haven’t changed, everything looks like it did yesterday. So it’s time to turn attention to the data.
First, the team needs to get their hands on the data, so they email the data team with their data access request. And they wait. And wait.
Finally… the data reaches their inboxes–all 3 million images of it. Unfortunately, the data team can only send a maximum of 100k images at a time via zip file. So, your team has to open 30 emails and download each locally–they don’t know yet what they’re looking for, so they will need to go through all the images as they work to figure that out.
Now the team is stuck writing scripts and clicking through the local view, trying to remember naming conventions and find out what’s going on.
Even more unfortunately, there isn’t enough room locally to store all 3 million images… so they’ll have to do everything at least twice (and that’s assuming everyone is upgraded to 1TB local memory).
Everyone is freaking out about how complicated it is to figure out what’s the problem, while reports from the service team keep piling up. It’s like there’s a blind spot in the middle of the AI process.
What’s going on?? Your team is struggling with blindspots, unable to clearly see what’s missing in their data and models that is leading to failures in the real world. Is the data the problem? Are the model or the model weights the problem? Is it something else?
Blindspot Warning Indicators
Unfortunately, scenarios like this are all too common. How do you know that you’re missing something because of those blindspots? Here are some common signals:
-
- Models are struggling to handle the long tail of the huge variety of scenarios in the real world. Not only does it take forever to discover what scenarios your model is missing, but it’s also hard to figure out how to break through the resulting performance ceiling in a world where the difference between 95% and 99.99999% can be life or death.
-
- There’s so much visual data – but it’s a mess. The team can’t access it, is not sure what’s in it, and doesn’t know how it’s impacting model performance. No one really knows if the model needs more data, different data, better data, or … something else.
-
- Everything seems stuck in the slow lane. The team is frustrated that the work takes forever, and it’s hard to understand what’s happening across the ML process because the pipelines and codebases are a mess of patchwork tools and band-aids hacked together.
-
- Plus, even though new models and frameworks keep cropping up, it’s hard to test them out, let alone determine if and where they can help.
- Plus, even though new models and frameworks keep cropping up, it’s hard to test them out, let alone determine if and where they can help.
Stymied by those challenges, far too many AI projects stay stuck in neutral for months and months before eventually being shut down. The grim reality is that most AI projects fail – as many as 80% by some estimates.
But it doesn’t have to be that way. With a different approach and the right tools, stories like this play out much differently – and aren’t scary at all.
Seeing What’s in the Blindspots
What does it take to see what’s hiding in your blindspots? It starts with changing the way that you and your team work with data.
All too often, data and models are handled as separate domains – separate teams, tools and processes are responsible for each. Taking that separation even further, core processes such as labeling, annotation and data generation are often handed off to outside vendors and partners. However, the development of AI is a highly iterative process–choose a model, fine tune the model with data, evaluate model performance and failures, gather additional datasets to improve the model, and so on. Separating data and models adds overhead and complexity that create roadblocks that get in the way of success, not to mention leading to cobbled together patches trying to bridge the divide but instead creating even more inefficiency and complexity.
If data and models aren’t connected in the AI process, getting model performance where it needs to be – and getting AI systems on the road – is extraordinarily difficult.
The solution? Tear down the divide, making it possible to refine your models and applications together with your data. When you give your team the ability to manage, examine and curate datasets alongside the work they do to evaluate and iterate their models, you enable them to cut through inefficiency and blindspots. Instead of waiting for datasets to be created and annotated, your team can easily and iteratively understand what’s in their data, determine what data models need to increase accuracy, and evaluate changes to models. That makes your team more productive, accelerating time to delivery for new capabilities and improvements.
At Voxel51 we’ve built a solution designed to make that possible. We created open source FiftyOne and its complementary commercial offering, FiftyOne Teams, to simplify and automate how AI builders explore, manage, visualize and curate visual data and models. FiftyOne Teams is built for how ML teams work – it’s designed to plug into your team’s stack no matter what your toolkit looks like, so that your team is able to connect their experiment tracking directly to their data. Tens of thousands of users rely on Voxel51 solutions to build, troubleshoot and iterate visual AI faster and more easily.
What’s Possible: Taking the Fast Lane to Success
Going back to our earlier scenario, what would it look like if your team could refine their data with their models?
The alert goes off: model drift. Ghost potholes. But instead of scrambling, the team opens up FiftyOne Teams. Using FiftyOne, the team is able to connect their experiment tracking directly to their data and find the exact model failure modes in context with the data.
Since FiftyOne Teams is securely integrated with your data sources, teams have direct access to data – no more waiting, no more zip files, no more disorganized e-mails or file shares! The team can access – based on their permissions – data wherever it lives (on-prem, the cloud, multiple cloud providers, etc.).
The tool’s extensibility in Python allows the team to build custom queries, bringing in the newest approaches and models to understand what’s happening. To find outliers, they use FiftyOne Teams’s uniqueness detection to sort through edge cases and instantly find the culprit: tree branches! Massive storms across the South and Midwest knocked over tons of trees recently, and the new data was creating ghost shadows in the model. The team is able to respond quickly, cleaning up the data inputs and getting model performance back on track.
Get on the Road
If your team is working on automotive visual AI challenges, we’d love to help you join other leading OEMs and Tier 1 suppliers that are already using Voxel51 solutions to connect data and models with the way that teams work to build successful visual AI applications.
Schedule a demo and we’ll show you how.
