Using Computer Vision to Enhance Customer Experience in Retail

Although the retail industry remains a pillar of the global economy, with the global retail sector reaching $22 trillion in sales in 2023, it continues to face transformative disruptions. From the continuing growth and evolution of ecommerce to the long-term impacts of the COVID pandemic, the influence of social media and more, retailers need to adapt and evolve. Throughout any challenge or disruption, the ability to deliver superior customer experiences is critical to success–according to Zendesk’s Customer Experience Trends Report 2024, 57% of consumers say they would switch to a competitor following a single bad experience.

As retailers look for new ways to innovate the customer experience, computer vision and artificial intelligence (AI) are emerging as transformative technologies with the potential to enhance customer experience to the benefit of both retailers and their customers. Across both physical stores and online shopping, leading retailers are turning to solutions powered by computer vision to deliver more personalized, engaging, and efficient experiences for their customers. 

The Power and Potential of Computer Vision in Retail

Computer vision is a specialized field within AI that analyzes and interprets visual data with machine learning algorithms. Using computer vision, AI applications interpret and analyze visual information to provide insights and help make decisions based on that data.

Computer vision includes core functionality essential to building applications for retail, such as:

• • Object detection: computer vision algorithms can analyze picture, video, and imaging data collected from security cameras and sensors to identify objects, whether people or carts or items on store shelves.

• • Object classification: further analysis of the objects detected in visual data can be used to identify which people are shoppers vs. employees, which items were taken from a shelf, or damaged items.

• • Object tracking: computer vision algorithms can also track the movement of individual objects, from tracking where people walk in a store to what a shopper does with an item after picking it up from the shelf.

• • Behavior analysis: computer vision also makes it possible to analyze customer behavior and paths through a store, to distinguish dwell times assessing a product from time spent checking one’s phone, and to assess wait times at checkout.

Retail is ideally suited for applications that utilize computer vision and AI (also referred to as “visual AI”). Visual information plays a critical role in consumer behavior and experience, from how shoppers navigate stores to how they assess the attractiveness and value of a product. There is a range of camera and sensor solutions designed for retail, increasingly built for use with computer vision, that capture and process picture, video, and imaging data from stores, warehouses, and delivery vehicles.

The recent surge of investment in AI has led to rapid advances in computer vision, and has made those advances available to a broader range of organizations and applications than ever before. Not only is computer vision enabling innovations in customer experience, but there are also transformative applications in other areas of retail, from inventory management to supply chain management, loss prevention, and more. 

Transforming Customer Experience with Computer Vision: Real-World Applications

Let’s take a closer look at a few examples of ways that retailers can use computer vision and visual AI to improve customer experience, both in-store and online. 

Just-in-Time Customer Assistance

We’ve all had the experience of wandering up and down aisles of a store, frustrated by being unable to find a specific item or unable to find someone who can help us with an item that’s hard to reach or locked away. Computer vision technology and analysis can help, processing data from video cameras to identify shoppers who appear to be circling looking for something or struggling to reach an item on a high shelf, and notify staff so that an associate can be routed to offer the customer assistance. Particularly in large footprint stores, that can dramatically reduce customer frustration while also ensuring that staff are able to provide more timely and efficient service.

Store Layout 

Computer vision technology also helps retailers organize their stores and aisles in ways that improve the customer experience. By analyzing customer movements throughout the store and correlating that information with customer shopping baskets, retailers can improve store layout to reduce the amount of time that customers spend walking to different parts of the store, making customers’ shopping more efficient. They can also use computer vision algorithms for object and pose detection to identify items that are most frequently difficult for customers to reach so that those items can be placed on more accessible shelves.

Virtual Try-Ons

It’s not only physical retail stores where computer vision can give customers a better experience. One of the challenges of online retail is not having the ability to try out a product–to see how that jacket looks on you, or if that pair of glasses works well on your face, or if that coffee table would complement your sofa. 

Using computer vision, retailers can now create a virtual try-on experience that online shoppers can use anywhere, including the comfort of their own homes. These augmented reality (AR) applications use computer vision algorithms such as pose estimation, image segmentation, and 3D modeling to analyze images and video from the user’s phone or computer to provide a realistic virtual representation of how a product would look in real life. For example, L’Oreal provides a virtual makeup try-on so that online customers can see how different lipstick shades and brands might look on their face.

Offering virtual try-ons provides an improved customer experience for online shoppers. That increases online engagement, reduces returns, and offers a competitive edge in the ecommerce landscape.

Streamlined and Automated Checkout

Extended time waiting in the checkout line or dealing with errors during checkout has a significant impact on the customer experience–according to one study, 77% of customers said that they would avoid returning to a store where they previously had a long wait at checkout. 

Rather than relying on staff to notice long lines and request help, retailers can use computer vision-powered cameras to continuously monitor checkout lines and automatically request additional staff and checkout lanes. In addition, real-time analysis of in-store foot traffic can be used to predict when checkout is going to become busy so that staffing levels at checkout can be proactively adjusted.

Retailers are also beginning to deploy solutions that streamline and automate self-checkout with computer vision technology. We’ve all experienced the frustration of extended waits while other shoppers struggle with self-checkout systems. Self-checkout areas using computer vision cameras can help by monitoring and guiding shoppers through the process. More advanced systems can instantly recognize and tally products, eliminating the need to individually scan items. Shoppers can simply place their items in a designated area and the system automatically calculates the total. Not only do these systems facilitate a seamless and rapid checkout experience, they also help prevent accidental or intentional failure to scan all items that leave the store. For retailers, streamlined self-checkout increases accuracy and efficiency while reducing labor costs. For shoppers, contactless checkout systems offer a smooth grab-and-go shopping experience, reducing the time spent at checkout counters and enhancing overall convenience.

Challenges and Considerations for Implementing Computer Vision in Retail

Computer vision solutions can be transformative for the retail customer experience, but that doesn’t mean that there are not challenges and considerations that need to be taken into account when evaluating and adopting them. 

Data privacy and security are important to both consumers and regulators, with implications for computer vision systems and for how data captured and used by those systems is stored and managed. Privacy regulations, including the California Consumer Privacy Act (CCPA), require explicit consent from the consumer to allow use of facial recognition, and personal data protection and privacy regulations, including the General Data Protection Regulation (GDPR), also apply to image and video data. When planning to deploy computer vision systems, retailers need to understand these requirements and chose vendors and partners who can help meet them.

Integration with other systems is also crucial to success. Although some vendors offer fully-integrated solutions spanning from hardware to software and services, in many cases retailers can benefit from combining pieces from multiple vendors. When selecting tools for a computer vision stack, look for compatible data formats, storage options, and security measures to simplify deployment and make scaling across different projects and stores easier.  

Data quality is critical to the success of computer vision solutions. Without quality data, computer vision models and algorithms can fail to correctly identify and analyze objects and patterns. Video and image data collected in retail operations can be particularly challenging due to variable lighting conditions, dusty or smeared camera lenses, and placements that result in high and wide camera angles. It’s not just a question of having enough quantity and variety of real-world data to train your computer vision models, it’s also critically important to have tools that allow you to understand and manage that data to understand and address the edge cases, outliers, gaps, and scenarios that are challenging for your computer vision models. For example, if training data doesn’t include video samples from dirty cameras in poor lighting, computer vision models will struggle to accurately identify shoppers and their activity in similar scenarios. 

Building Customer Experience Solutions for Retail Using FiftyOne

When it comes to building computer vision solutions tailored to deliver enhanced customer experiences in the retail sector, being able to understand and intelligently use visual data at every step of development is critical to success. The explosion in quantity and variety of visual data has created both opportunities and challenges for builders creating retail solutions. More data helps train algorithms to understand a greater variety of scenarios, but also exponentially increases the complexity of managing data and putting it to use effectively and efficiently, not to mention addressing privacy and security considerations. Those challenges are only magnified if approached with handbuilt patchworks of fragmented tools.

FiftyOne from Voxel51 is a solution designed from the start to make it easy for computer vision developers to understand, refine, and use data throughout development. Powered by open source, FiftyOne helps builders:

• • Explore, visualize, and understand visual data and find valuable insights in it

• • Improve data quality and make it possible to curate optimal datasets for development use

• • Use data to evaluate and refine computer vision models to improve capability and accuracy

Whether used by retailers building computer vision solutions customized for their specific needs or by suppliers creating solutions for retailers, FiftyOne’s flexibility and extensibility enable builders to simplify and accelerate development of robust and reliable solutions. FiftyOne makes it easy for builders to leverage the latest advances in algorithms and models with their data, and integrates easily with other critical solutions including those for labeling and annotation, experiment tracking, and more.

FiftyOne also helps ensure that data security and privacy requirements are met. Unlike handbuilt solutions, FiftyOne provides built-in access control and auditing, whether deployed on-premises or in the cloud, so that you can control how data is accessed and used.

The Future of Computer Vision in the Retail Customer Experience

New examples of ways to use computer vision to enhance the retail customer experience continue to emerge as adoption grows. Integration with other data sources creates possibilities for even more personalized customer experiences, for example enhancing virtual try-on experiences with visual suggestions for complementary accessories. It also creates new opportunities for targeted promotions, such as noticing increased traffic for certain products and delivering real-time promotions via digital signage to convert a surge in interest to shopping baskets and sales. 

As illustrated by these examples, computer vision is a transformative technology for the retail customer experience. By enabling personalized, efficient, and engaging experiences for shoppers across brick and mortar stores and digital retail, computer vision contributes to critical customer satisfaction and retention priorities. Powered by FiftyOne, AI builders are creating new and innovative solutions that computer vision makes possible.