Automating the Data Pipeline, Enhancing Model Performance, and Saving Valuable Resources
Success story at a glance
Company:
- Name: Secury360
- Industry: Commercial & Home Security
- Data type and sources: Video footage from cameras deployed for perimeter detection in commercial and residential customer environments
Challenge:
- Secury360’s security solutions must be fast and accurate while eliminating false positives
- While automating their data pipeline, Secury360’s ML team searched for a dataset management solution to easily filter, query, and visualize datasets and false positives, and evaluate model performance
- To free up valuable engineering time, the dataset management solution must be able to seamlessly integrate with Azure, CVAT, and vector databases
Solution:
- Secury360 deployed FiftyOne Teams as the main access point for their video dataset of 1M samples (and growing)
- From the seamless connection to Azure cloud storage, FiftyOne Teams automatically creates a data subset of new false detections using detailed filtering and sends it off for annotation in CVAT
- FiftyOne Teams enables easy evaluation and continuous improvement of model performance
- FiftyOne Teams saves Secury360 significant time, effort, and money
Introduction
Secury360 makes it possible to transform any security camera into an active burglar detector without complicated configurations. The Secury360 box transforms CCTV setups into a proactive detection solution that uses AI to eliminate false alarms and guarantee only human detection. The box features edge AI to gradually learn the terrain through deep learning to recognize behavior and discern if someone has bad intentions. Because only human detections get through the filter, operators have more time to respond to and prevent real threats.
Searching for a dataset management solution
Secury360’s AI-powered perimeter intrusion detection solution must proactively identify and stop burglary attempts—before any theft or damage has been done. This means the solution needs to be fast and accurate while eliminating false positives that lead to unnecessary alerts and wasted resources as security teams investigate non-existent threats.
When Secury360’s ML engineering team set out to automate their data pipeline, they needed a dataset management solution that could serve as the main access point to their data. This would allow them to efficiently filter, query, and visualize their datasets and false positives, as well as evaluate model performance.
Secury360 also needed a dataset management solution that could seamlessly integrate with Azure, CVAT, and vector databases, freeing up valuable engineering time to focus on delivering even more value to customers.
Finding FiftyOne Teams
After evaluating several solutions, Secury360 found open source FiftyOne and used it to prove that it met their needs. Then, Secury360 upgraded to FiftyOne Teams to get out-of-the-box support for connecting to Azure cloud storage.
“One of my first projects at Secury360 was to set up a robust data pipeline for visualization, filtering, and searching. At that time, all our data came into Azure, and that was it—we had just one heap of data. While searching for a dataset management solution, I came across open source FiftyOne, set it up, and started testing it. I asked some questions in the FiftyOne Community Slack and noticed how active and how good the help was,” said Joy Timmermans, Machine Learning Engineer at Secury360.
Joy continued, “From the beginning, FiftyOne Teams had several features that were very interesting to us. After validating the open source version of FiftyOne, we upgraded to FiftyOne Teams for a smooth integration with our Azure cloud and to fully automate our data pipeline. Now, FiftyOne Teams is the main access point for our data.”
Not yet familiar with FiftyOne Teams? It’s where real AI work happens. FiftyOne Teams helps you visualize, augment, manage, and QA data; it also helps you streamline the workflows that make enterprise machine learning possible. FiftyOne Teams extends open source FiftyOne with added features enterprises need, including dataset permissions, versioning, sharing, support, and more.
Automating the data pipeline
Secury360 relies on FiftyOne Teams daily for data filtering, visualization, and evaluating model performance. FiftyOne Teams manages Secury360’s expanding dataset, which currently comprises one million videos, and facilitates rapid querying. This allows for swift identification of intriguing scenarios, including false positives like a tree branch obscuring a camera’s view, which are then used for model training.
“We set up our data pipeline to use FiftyOne Teams as the backend so we can ingest our data from Azure and automate everything across the board. For example, with FiftyOne Teams we can now automatically create a data subset of new false detections using very detailed filtering, and then automatically send it off to our annotators. Then, after training our models on that new data, we can very easily compare results to previous and future training runs because all our data is managed by FiftyOne Teams.”
Improving model performance
Secury360’s security solution is composed of two main components, each with its own model, to ensure a robust and efficient security solution:
- The S360 box: an edge device equipped with a detection model that analyzes video images on the spot and performs initial filtering of potential reports
- The Cloud AI: a bigger model in the cloud that performs a more detailed analysis of the detections, enabling swift verification and allowing control rooms to concentrate on genuine reports rather than false alarms
In this system, detections from the S360 box are transmitted in 10-second video clips to the cloud AI for further scrutiny. Upon confirmation of the detection by the cloud model, the information is relayed back to the device, which subsequently triggers the alarm.
With the help of FiftyOne Teams, Secury360’s ML team can track and increase the performance of their models over time.
“With FiftyOne Teams, we are able to create datasets with even distributions and evaluate models much more easily. For example, we continuously track our number of false positives, and one time, we performed a model rollback when we were seeing false positives at a higher rate than normal. We caught this before there was any impact on our customers,” explained Joy.
Saving significant time, effort, and money
One of the biggest selling points of FiftyOne Teams was that Secury360 could seamlessly connect to their Azure cloud-backed media. Secury360 briefly considered a “DIY approach” but quickly realized they would have to invest substantial time and money to buy and maintain a server to host their data, make it available to multiple users, and take on the risk and burden of keeping the data secure and always available. But with FiftyOne Teams, this is all handled effortlessly.
“FiftyOne Teams saves us significant time, effort, and money. We don’t have to maintain a local server and store all of those video clips. We don’t have to worry about losing our data if the internet goes down or having to re-ingest all the data and metadata after a server crash. We can simply connect FiftyOne Teams to our cloud storage and then spend our energy on our core business of building the best security solutions for our customers,” Joy described.
