Welcome to the fourth installment of Voxel51’s computer vision industry spotlight blog series. In this series, we highlight how different industries — from construction to climate tech, from retail to robotics, and more — are using computer vision, machine learning, and artificial intelligence to drive innovation. We’ll dive deep into the main computer vision tasks being put to use, current and future challenges, and companies at the forefront.
In this edition, we’ll focus on sports! Read on to learn about computer vision in the sports industry.
Industry Overview
Sports brings people around the globe together in fitness, fun, and the spirit of competition. In addition to being an entertainment source, the sports industry boosts economies by standing behind new technical innovations and opening up job opportunities.
Key facts and figures:
- The global sports market is growing, increasing from $486.61 billion in 2022 to $512.14 billion in 2023
- Millions of people worldwide are employed in the sports sector, including more than 456,000 people in the United States with a projected 7 percent job growth over the next decade
- Narrowing in on technology, the global sports technology market was valued at USD 13.14 billion in 2022 and is expected to grow at a CAGR of 20.8% from 2023 to 2030
- The global AI in sports market is projected to reach $19.2 billion by 2030, growing at a CAGR of 30.3% from 2021 to 2030
Applying computer vision and artificial intelligence (AI) to sports opens up a multitude of helpful new tools for teams, coaches, sports analytics professionals, players, scouts, and fans, while also presenting a multi-billion dollar opportunity for tech companies. Computer vision enables organizations to build capabilities like real-time video analysis, fitness and health tracking, sports predictions, and improving the overall fan experience. These tech advancements can help improve how things like player performance analysis, fan engagement, and marketing strategies are handled.
Before we dive into various popular applications of computer vision-based AI technologies in sports, it’s important to highlight the key challenges facing the industry, presenting areas of opportunity for AI innovations.
Key Industry Challenges in Sports
- Player health and performance: Ensuring player well-being and implementing strategies to optimize player availability is crucial for sustained performance in elite team sports.
- Fan engagement in the digital age: While digital platforms have expanded reach and revenue, striking the right balance between meaningful interaction and oversaturation is crucial. The top 25 leagues in the world had a combined audience of over four billion and generated more than €2.8 billion and 676 billion impressions through their digital inventory. Keeping fans engaged and fostering a sense of community requires innovative strategies to ensure that the essence of sports isn’t lost in the digital noise.
- Venue evolution: According to industry experts, work must be done to make the venues themselves (stadiums, arena, and ballparks) more attractive, affordable, comfortable, safe, and technology-equipped to satisfy the needs of today’s fans and as they evolve over time.
Continue reading to learn about several exciting and useful ways in which computer vision applications are helping organizations in the sports industry.
Computer Vision Applications in Sports
Sports Analytics & Strategy
Sports are exhilarating for all involved — teams, players, coaches, and fans. Whether it’s a come-from-behind victory or a record-breaking play, the thrill of the game keeps people coming back for more. And when one match ends, sports pros on the field and in the back office are already thinking of ways to continue to improve performance and safety, and make the game even more exciting for fans.
Data is the key to unlocking winning strategies. The use of cameras, equipment sensors, wearables, and even radar and LiDAR scans like in the MLB, makes a variety of visual information available. Now, every jump, sprint, shot, throw, and maneuver can be captured so that the information can be organized and analyzed.
Tracking players’ movements, positions, speeds, and trajectories offers a rich data source. This wealth of information enables rich analytics for coaches, athletes, and sports professionals to gain valuable insights into performance — not only to advance individual and team performance at game time, but also to refine training plans, scout new talent, and for competitive analysis and strategies.
Pose estimation and object tracking are on the forefront of computer vision in sports. For example, coaches and analysts use pose estimation to determine ideal swing and pitch patterns in MLB to squeeze every ounce of performance out of their players. Soccer is also quick to adopt both of these technologies, tracking players on the pitch and seeing how they react to sudden changes in the ball’s positions. Through thorough analysis of penalty kick pose estimations, goalies could get a leg up on the competition and look for tells on where the shot is likely to go, using insights only possible through computer vision.
For further reading, check out these articles on computer vision and AI in popular sports franchises:
- A Deeper Look at MLB’s New Statcast
- Fueling the future of sports: How the NFL is using data to change the game–on the field, in the stands, and in your home
- The NBA’s Game-Changing Approach to Data
- NFL’s Next Gen Stats
Also check out these papers related to using computer vision for athletic motion tracking:
- Motion Capture for Sporting Events Based on Graph Convolutional Neural Networks and Single Target Pose Estimation Algorithms
- A survey on location and motion tracking technologies, methodologies, and applications in precision sports
- All Keypoints You Need: Detecting Arbitrary Keypoints on the Body of Triple, High, and Long Jump Athletes
- VIRD: Immersive Match Video Analysis for High-Performance Badminton Coaching
- A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions
Injury Prevention and Rehabilitation
Using computer vision in sports has led to the development of cutting-edge ways to prevent injuries and heal from them. By analyzing athletes’ movements during exercises and game-time matches, computer vision techniques like feature extraction, pose estimation and motion detection can detect improper actions that might lead to injuries. Analyzed data comes in handy for coaches and medical teams when they are putting together personalized training and conditioning programs to prevent potential injuries. This information is also important for improving the design and construction of protective gear and equipment.
When it comes to rehabilitation, computer vision is an essential asset. The healing journey of athletes can be monitored to make sure that rehabilitation exercises are done correctly to lower the risk of reinjury. Techniques such as marker-less human pose estimation are particularly promising, allowing for cost-effective and reliable telerehabilitation services without additional equipment. Digitizing rehabilitation not only improves accuracy but also holds the potential to speed up recovery, making the process more organized and data-driven.
Here is an example to check out on vision-based technologies being used in a popular sports franchise:
Here are some papers related to using computer vision for injury prevention and rehabilitation:
- Hybridized Hierarchical Deep Convolutional Neural Network for Sports Rehabilitation Exercises
- A Beta Version of an Application Based on Computer Vision for the Assessment of Knee Valgus Angle: A Validity and Reliability Study
AI Referee Assistance
AI referee assistance aids human referees in overseeing sports games. Through computer vision and machine learning, AI referees can make precise calls in real time, reducing human errors. They can detect goals, misconduct between players, and other rule infringements swiftly and accurately. This not only enhances the credibility of the game but also alleviates the pressure on human referees.
In soccer for example, an AI refereeing system can use multiple cameras to capture the field from different angles. These images are then analyzed in realtime to detect events like fouls or offside situations. For example, in an offside scenario the system can calculate the positions of players, the ball, and the last defender at the moment the ball is played. If a player is found in an offside position, the system instantly alerts the human referee and can provide a visual representation of the scene on a sideline monitor for verification, ensuring that the call is accurate and fair.
Here are a few resources on vision-based referee assistance technologies in use in popular sports franchises:
Here are a couple of papers related to using computer vision for refereeing:
- VARS: Video Assistant Referee System for Automated Soccer Decision-Making from Multiple Views
- Vision Based Dynamic Offside Line Marker for Soccer Games
Fan Experience Enhancement
Enhancing fan experiences and fueling new ones are exciting applications of computer vision in sports. While there are more ways unfolding to inform and engage fans further than ever before, in this article we’ll focus on two prominent use cases: augmenting the broadcast experience and new ways to engage via Augmented Reality (AR) and Virtual Reality (VR).
Augmenting the Broadcast Experience
Computer vision is paving the way for enhancements to sports broadcasting to create a richer viewing experience. It’s now possible to analyze live and recorded content in real time in order to extract insights to pair with the broadcast. For example, vision-based AI can provide real-time statistics and player information, real-time ball tracking, the ability to generate on-screen graphics, as well as identify key moments in matches. All of these new possibilities help fans develop deeper connections to the game.
Check out this resource on AI-powered sports features:
Immersive AR and VR Experiences
With the combination of AR and VR, the traditional stadium experience is evolving, making sports events more engaging and interactive for fans. Fans can virtually explore stadiums, get the latest player profiles, enjoy 3D game rewinds, and live chat with other fans. Computer vision technology captures and analyzes the game as it happens, turning complex on-field actions into digital data. This data then powers the AR and VR applications, making spectators feel like they’re part of the game.
Moreover, renowned football teams and leagues have already begun experimenting with AR/VR technologies. Through interactive apps and other digital platforms, fans can virtually immerse themselves in live games, accessing features like panoramic camera angles, real-time stats, and on-demand replays. With the continued advancement in AR, VR, and computer vision technologies, the boundaries of how fans experience sports are set to expand further, potentially leading to features like holographic player projections and live streams of remote audiences, offering a new level of engagement and excitement for sports enthusiasts worldwide.
Here are some resources related to enhancing the fan experience using AR and VR:
- The impact of virtual reality (VR) technology on sport spectators’ flow experience and satisfaction
- The Metaverse and Sports: How VR and AR Could Transform the Fan Experience
Personalized Fitness & Training
Computer vision is revolutionizing the realm of personalized fitness and training. Advancements in AI and ML make it possible to analyze human movement with remarkable accuracy, providing real-time feedback and tailored guidance to people pursuing their fitness goals. Popular use cases include:
- Posture and form tracking and correction: By monitoring body positions, angles, and movements, computer vision systems can detect deviations from proper form, providing immediate feedback to help people correct their technique and prevent injuries.
- Personalized workout recommendations: Data-driven approaches enable users to receive workout recommendations tailored to their specific needs and abilities, helping them achieve their fitness goals.
- Virtual coaching: With the ability to remotely monitor and assess user movements, fitness professionals and coaches can provide personalized guidance and support in virtual settings.
Check out this paper on using computer vision for fitness and training:
Organizations at the Cutting Edge of Computer Vision in Sports
The United States Tennis Association (USTA)
The USTA is using AI to level up player performance. With an increasing amount of data available from court cameras, video recordings, and wearables during practice, AI plays a significant role in bringing performance insights to the forefront for the entire performance team of athletes, coaches, mental skills staff, and strength and conditioning professionals.
At the heart of USTA’s AI-driven performance management system is data: the x-y coordinate of the player on the court, every shot of the rally, the speed of the shots, the spin, the number of changes in direction, and more. Being able to access and analyze critical data is important in evolving player strategies and winning more matches.
AI-generated insights enable athletes to compare their technique with top players, for example those with outstanding backhands, to understand where exactly to tune up their stroke to improve performance. Athletes can also analyze their performance over time, and narrow in on perfecting the one or two techniques that will make a sizable impact on match performance.
Tennis tournaments generate a wealth of data, while highlighting the importance of AI-powered systems for the sport: athletes can analyze the performance of players they will compete against, as well as analyze their own matches once they’re over, so they can swiftly and continuously tune and improve.
AiSport
AiSport is building an AI fitness platform to give users real-time feedback on their workout techniques, all from the convenience of their smartphones. AiSport’s platform uses AI and computer vision to analyze and correct people’s posture while they are exercising to maximize effectiveness and prevent injuries. With AiSport, fitness clubs can provide their members with not only equipment and a location, but also with a personal AI fitness trainer. The AiSport’s tech analyzes and provides real-time recommendations to maximize performance while keeping workouts injury-free. Computer vision techniques at the center of AiSport’s platform include: 3D body pose and shape recognition, biomechanical analysis, pattern matching, and deviation estimation.
AiSport was co-founded by two Ukrainian women and long-time sports enthusiasts, Anna Stepura and Dariia Hordiiuk. Development of the AiSports platform continues today in Silicon Valley.
Hawk-Eye Innovations
Hawk-Eye Innovations, a pioneering UK-based company and part of the Sony group, focuses on applying computer vision to sports. With a team of dedicated professionals, Hawk-Eye has become a household name in the sporting world, delivering precise and real-time tracking, analytics, and officiating assistance in a wide variety of sports, including tennis, cricket, and soccer. Their key technologies include the Synchronized Multi-Angle Replay Technology (SMART) for enhanced video capture, review, clipping, and distribution, the TRACK systems for Performance Tracking, Ball Tracking, and Object Tracking, and the INSIGHT suite, which provides data collation, storage, aggregation, delivery, and visualization capabilities.
Founded in 2001, Hawk-Eye has grown into a thriving company with a global presence. Not only have they partnered with Major League Baseball (MLB) for optical tracking and vision-processing technology and the National Basketball Association (NBA) for deploying 3D optical tracking technology, they have achieved international recognition and are an integral part of sports events in more than 90 countries worldwide.
Sportlogiq
Based in Montreal, Quebec, Sportlogiq initially focused on professional hockey and then expanded its reach to collaborate with major sports teams and data providers around the world. Numerous NHL clubs, more than 150 professional and amateur hockey teams worldwide, media outlets, content producers, top performance research firms for soccer and football, as well as amateur sports and video companies, all rely on their data and insights.
The company is supported by prominent investor Mark Cuban and the TandemLaunch incubator. Sportlogiq is made up of a team of 13 AI researchers. They have been granted 180 patents and publications and have 75 full-time professionals onboard. Sportlogiq is helping to shape the future of AI in sports by providing creative solutions for improving athletic performance and training.
Ludimos
Ludimos, a smartphone-based cricket training app, is transforming the world of cricket coaching. Founded by Madan Rajagopal, an Indian cricket enthusiast living in the Netherlands, Ludimos was born out of his frustration with inconsistent coaching advice and a lack of tools to track players’ progress.
As a data scientist and AI engineer, Rajagopal developed Ludimos to address these challenges. The app has gained widespread popularity, with over 19,000 users across 15 countries, including national cricket associations and teams like Royal Challengers Bangalore.
What sets Ludimos apart are its specialized features aimed at cricket training. It offers multi-angle video analysis, allowing a thorough look at player techniques from different viewpoints. The app excels in ball and bat tracking, giving a data-driven insight into player performance. Additionally, it provides a communication platform for coaches to assign drills and give feedback, making the coaching process more interactive and efficient. While ball tracking is its strong suit for now, Ludimos has plans to expand into bat tracking and biomechanics analysis, showing a promising trajectory for evolving cricket coaching and player analysis.
Track160
Track160 is changing the way soccer is coached and analyzed. Founded and chaired by Miky Tamir, a pioneer in sports computer vision, Track160 combines cutting-edge technology with a deep understanding of the game to provide valuable insights to soccer clubs and academies.
At the core of Track160’s offerings is an AI-based solution that utilizes multiple cameras tethered to a single base. These cameras, equipped with computer vision and deep learning algorithms, capture a wealth of data related to player performance and team tactics. What sets Track160 apart is its commitment to data accuracy, earning FIFA certification for its data quality from a single installation point.
Tonal
Tonal is an AI-powered home gym system that combines strength training equipment, personalized fitness coaching, and live and on-demand classes. It was founded by Aly Orady, a supercomputer engineer who wanted to create a more effective and convenient way to strength train at home.
Tonal’s main piece of equipment is a wall-mounted device with two electromagnetic pulleys that can provide up to 200 pounds of resistance. The device also has a touchscreen display that shows you how to perform each exercise and tracks your progress. Tonal uses AI to dynamically adjust the resistance for each exercise based on your individual strength and fitness level in order to provide you with your most effective workout. Tonal is trusted by an impressive number of world-class athletes.
Sports Datasets
If you are interested in exploring applications of computer vision in sports, check out these datasets:
- SoccerNet-V3: A fantastic collection of annotated soccer games capturing the key moments of the league. Spanning 3 years, 33 teams, and almost 1800 matches, the dataset features both bounding box and polyline detection. This dataset combines the best of both worlds allowing for high level sports analysis with high grain filters, as well as annotation to empower even the most advanced computer vision techniques.
- NFL-Impact-Detection: Helping the NFL usher in a new age for analytics and player safety, the NFL Impact Detection dataset serves as an open community challenge to help detect concussions in videos from past NFL games. In a different approach than most datasets, the dataset has annotated bounding boxes on only the helmets of the players. This dataset is just one of the challenges the NFL has issued to help advance safety in football by using machine learning.
- Football Player Segmentation: This dataset is specifically designed for computer vision tasks related to player detection and segmentation in soccer (football) matches. The dataset contains images of players in different playing positions, such as goalkeepers, defenders, midfielders, and forwards, captured from various angles and distances. The images are annotated with pixel-level masks that indicate the player’s location and segmentation boundaries, making it ideal for training deep learning models for player segmentation. Explore this dataset with FiftyOne in your browser.
- SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes: A large-scale multi-object tracking dataset consisting of 240 video clips from 3 categories (i.e., basketball, football, and volleyball). The objective is to only track players on the playground (i.e., except for a number of spectators, referees and coaches) in various sports scenes. Explore this dataset with FiftyOne in your browser.
- Sports Videos in the Wild (SVW): A Video Dataset for Sports Analysis: This dataset comprises 4,200 videos captured via smartphones, covering 30 categories of sports and 44 different actions. The dataset can be used for genre categorization, action recognition, and other computer vision applications.
- DeepSportRadar-v1: This dataset can be used to solve four challenging tasks related to basketball—ball 3D localization, camera calibration, player instance segmentation, and player re-identification.
- UCF Sports Action Data Set: This dataset consists of actions collected from various sports typically featured on broadcast television channels such as the BBC and ESPN. The video sequences were obtained from a wide range of stock footage websites, including BBC Motion Gallery and GettyImages. The dataset includes a total of 150 sequences.
- Olympic Sports Dataset: This dataset consists of videos of athletes practicing 16 different sports.
- Sports-1M: This dataset consists of over a million videos with 487 sports-related categories, with 1,000 to 3,000 videos per category.
If you would like to see any of these or other computer vision sports datasets added to the FiftyOne Dataset Zoo, get in touch, and we can work together to make this happen!
