Rising Sun Pictures (RSP) has appointed two post-doctoral researchers from the Australian Institute for Machine Learning (AIML) to lead a new initiative centred on the application of Artificial Intelligence (AI) to its visual effects pipeline. Dr John Bastian and Dr Ben Ward will develop pipeline tools based on machine learning and other AI applications as a means of speeding up visual effects production and creating highly photorealistic visuals.
Believing in the potential of AI to substantially change current VFX production practices, RSP is committed to working at the forefront of new AI-based techniques and systems. “AI holds great promise for visual effects applications, especially in terms of accelerating labour intensive tasks and augmenting human creativity,” said RSP’s Managing Director Tony Clark. “Our early work with AI has produced spectacular results and we are eager to push development further.”
Working through AIML, a part of the University of Adelaide, John and Ben recently collaborated with RSP on visual effects sequences for Marvel Studios’ film ‘Shang-Chi and the Legend of the Ten Rings’. They developed a novel technique for facial replacement that was used in high-intensity martial arts combat sequences. Machine learning tools were employed to train neural network models of principal actors and apply them to the faces of stunt doubles. The process proved significantly faster than traditional means of facial replacement and produced more lifelike results.
At AIML, Ben and John have been working over the past three years as post-doctoral researchers involved in researching and developing machine learning and computer vision. They have contributed to projects related to medicine, defense, agriculture, manufacturing, environmental sustainability and general science. Both men earned their PhDs in computer science from the University of Adelaide.
In their new roles, John and Ben will work with RSP’s creative team on the application of AI tools to specific production challenges. They are currently involved in three film and series projects involving facial replacement applications like those used in ‘Shang-Chi’, when the tools were used for high-intensity combat scenes.
For that project, neural network models of principal actors were trained and applied to the faces of stunt doubles engaged in martial arts fighting. An 'identikit', or set of characteristics, of both the stunt double's and actor's faces was learned against a shared dictionary of corresponding facial features. The faces are then detected in the plate, and the machine rebuilds the image using the identikit of the intended actor. They used about 30,000 face images across five characters, training five principal models that combined to over 4 million training iterations. The team’s work totalled 51 face replacements over six scenes.
John and Ben will also seek to develop general workflow tools for routine 2D and 3D visual effects. AI can be a means of relieving artists of tedious, labour intensive tasks, speeding up the delivery of visual effects and controlling costs. “Our focus is on building AI-tools that can be integrated into RSP’s existing workflow to overcome the limitations of current technology,” said Ben. “It’s about helping artists improve results, achieve outputs faster and push beyond what was previously possible.”
John noted that AI should be viewed as a means of empowering artists. “AI can help artists accomplish incredible artistic effects without the tedium of executing every frame in a long sequence,” he said. “Facial replacement was a good place to start because faces are largely rigid, but we’ll soon be able to do the same with the human body. We’ll be able to make an actor taller, shorter, skinnier or more muscular.” www.rsp.com.au
