At IBC 2023, b<>com showed a new AI-driven application for broadcasters called *Wisdom* that performs intelligent frame rate optimisation. At the Show, Digital Media World met Tania Pouli, researcher and deputy director from the Image, Vision & Immersion Laboratory at b<>com, to learn about the software.

b<>com *Wisdom* is tool that adapts frame rate to content. High Frame Rate formats of over 100 fps may improve the visual quality of video, especially for sports, but it challenges the broadcast transmission chain by increasing the processing complexity and bit-rate. b<>com’s technique for *Wisdom* selectively removes frames from video to save storage without losing the quality of the original content. Relying on AI decision-making, it only removes images where the deletion is imperceptible to the viewer.

Immersive Goal

When the new UHD formats were defined, b<>com produced content for their own research work in their Image Vision & Immersion lab. The goal of b<>com’s lab is to take advantage of the modern audio-visual formats to immerse viewers in the heart of the action or information using computer vision techniques such as AR, VR and volumetric video enhanced by artificial intelligence.

On the b<>com website, Tania described how the lab team worked on motion blur, and filmed simultaneously with two cameras – one at 60 fps as the reference value, and the other at 120 fps, a higher frame rate for greater fluidity. "This enabled us to compare the resulting videos, and to highlight the benefits of high frame rates for certain types of content," she said. "But it also showed that for certain scenes, increasing the frame rate was of no help at all, which gave us the idea of varying it dynamically according to the content.”

Small Groups of Images – Greater Accuracy

*Wisdom* is based on an AI engine that quickly selects the ideal frame rate for each group of four images. “Working in very small groups of images means we can be highly responsive to changes in content. AI gives us the ability to know precisely when it's possible to delete images without it being noticeable, thanks to its engine trained on thousands of images, and takes into account multiple characteristics extracted from the content,” said Tania.

b<>com is considering both content creators like studios and TV channels, and distribution platforms such as Netflix, Disney and Orange, in their development on *Wisdom*. The process can be applied at different points during content creation, as well.

It can be used in post-production, for instance, to avoid manipulating excessive amounts of data. As it achieves file size reductions of up to 40%, it could also serve as a way to zip and unzip files when storing videos in mezzanine format. Tania said, “Another application is content distribution. Because you are dealing with broadcast formats such as MPEG-AVC or HEVC, the reduction in bitrate is more moderate, about 5 to 15%, because the images removed are not intended to be used as references, and are therefore B images of reduced size.”

When compressing video, B frames hold and use only the differences between the current frame and both the preceding and following frames to describe their content, thereby saving space.

The Option to Choose

The main advantage of *Wisdom* is that it varies the frame rate according to the content itself, whereas other tools work according to external criteria such as bandwidth or compute power. This gives it a special ability to preserve the quality of experience, since processing is imperceptible. Furthermore, it complements rather than competes with other approaches.

“For example, if we take the adaptive bitrate (ABR) commonly used for video streaming, each video segment, which is encoded according to different formats, could then be processed with our software, improving the overall video quality distributed to subscribers for a given distribution capacity,” said Tania.

Jean-Yves Aubié (left), Head of the Image Vision & Immersion Laboratory, and Tania Pouli, researcher and deputy director at the lab.

“Although some video encoders already perform similar tasks and significantly reduce the bitrate of certain frames – particularly B frames – *Wisdom* opens the possibility of choosing which frames the user can afford to remove – and that makes all the difference to the final render. Our approach is the only one that both respects the initial quality of the content and optimises image savings.”

Saving Space

Ultimately, it will save storage capacity on servers, with no loss of content quality. As mentioned, when *Wisdom* is applied before compression, the gain on the size of files to be stored or distributed to providers is around 35%, and when applied after encoding, the advantage is less – from 5% to 15% depending on the video. “But there's also an impact on the receiver side that's difficult to quantify because it depends strongly on the hardware implementation. Nevertheless, it seems natural to assume that removing images that contribute nothing can only be a benefit,” Tania said.

She also notes that b<>com would like to integrate the software into a CDN so that they can measure the effects in real-life conditions. She said, "By showing how a range of users can be served with the same approach and infrastructure, then we'll be able to quantify its benefits even more precisely. At the same time, we're continuing to think about ways of developing the algorithm that could have an impact on other video parameters.”   b-com.com