As broadcast delivery becomes more varied and agile, viewers continue to expect more dynamic viewing experiences, whether they are watching a game live or recording it to view on-demand. They want to see distinctive elements like creative graphics and visualised data, fast turnaround replays and highlights.
Ahead of its US broadcast coverage of the FIFA World Cup Qatar 2022 held in November and December, FOX built and deployed a new AI-powered feature, Catch Up With Highlights.
Developing Catch Up with Highlights
Using Amazon’s open source Media Replay Engine (MRE) framework for automating video clipping and replay generation, alongside the Amazon Web Services (AWS) infrastructure, Catch Up With Highlights can ingest broadcast streams, identify valuable match moments through AI-trained computer vision, and then encode delivery streams with metadata across user devices. Because Fox Corporation’s media workflows were already based on AWS, the development team was able to launch this feature in about eight weeks, despite its complexity.
FOX’s Vice President of Data Products and Machine Learning (ML) Strategy, Phil Martin, who led the Catch Up With Highlights development and first deployment, said, “Over the last five years our team has successfully used data to build a number of intelligent platforms and products. We have also been making the most of fans and fandom. We believe that sports should be enjoyable and accessible for everyone, and designed our platform to make it easy to engage with and enjoy.
“Our Catch Up With Highlights feature is an example of this philosophy, and we wanted it to reflect the exceptional standard of the athletes’ performance at events like the World Cup. To achieve this, we developed the feature as a highlights-specific AI engine that can dynamically retell the game’s story in real-time, highlighting key moments and events as they unfold. This keeps our users up-to-date with the action and equipped to experience the excitement of the game, no matter where they are. AWS has been a constant, valuable partner of ours, before and throughout development.”
Above is an example of an MRE training dataset for tennis, containing frame images that are separated into image classes and then labelled.
Automation
At the launch, Catch Up With Highlights ran automatically with very little human input. Phil and his team built the feature to continually optimise the highlight recap video as the game progressed, retelling the game story 20 times or more per match with event-driven updates. As a result, the viewers would be able to access a comprehensive recap of the game to date, available via the FOX Sports app on connected TVs and mobile devices.
Incremental improvements were rolled out based on performance, such as adding a thumbs-up or down, along with event enhancements like shots saved and penalty kicks. Ultimately, 731 recap videos were published autonomously across 61 matches.
The Catch Up With Highlights workflow includes the AWS Lambda event-driven serverless compute service, relying on Amazon Rekognition as the computer vision. Lamda runs code in response to events, and then automatically manages the computing resources that the code requires. Sports data is organised into an Amazon DynamoDB NoSQL database, then piped into Media Replay Engine. Live video is quickly delivered to viewers via AWS Elemental MediaConnect. The live clips are cached in Amazon ElastiCache, a managed in-memory data store and cache service for faster performance, and the videos are stored in Amazon Simple Storage Service (Amazon S3).
Audio Cues and Camera Angles
“Soccer is often referred to as ‘the beautiful game’ because of the flowing and artistic style of play. Although for viewers it’s a simple sport – score more goals than the opposition, training models and developing logic around a game that lacks set patterns is anything but simple,” Phil commented. “Furthermore, this was the first platform feature we have released that ingests multiple concurrent streams, truncates clips for inference, contextualises and optimises each segment, and transcodes and delivers directly into our CMS within moments of an event occurring.”
AWS Lamda event-driven compute service
As the team had limited reference information from the event organisers, they relied on audio cues and camera angles instead to help train the computer vision model. Clips could also be assembled automatically with polished elements, like transitions and a fade to black, to present a pleasing, relaxed viewing experience to fans. Now that they have a proven approach, made scalable through AWS cloud, Martin looks forward to replicating the Catch Up With Highlights delivery for other sports broadcasts.
Phil said, “We feel very confident now about working in this space and are already expanding to other sports properties. Fans can use this feature today during broadcasts of Major League Baseball (MLB), the United States Football League (USFL) and the FIFA Women’s World Cup. It will also be available on football broadcasts in the autumn and more personalised content is on the horizon, with many other possibilities.” aws.amazon.com
