Impossible Objects Visualises Mexico’s Sport Heritage for UFC at Sphere Las Vegas
VFX Supervisor Morgan McDermott at Impossible Objects talks about making their immersive film for UFC’s debut at the Sphere, combining heros and symbols of Mexican independence with UFC legends.
Chaos puts Project Arena’s Virtual Production tools to the test in a new short film, achieving accurate ICVFX with real-time raytracing and compositing. Christopher Nichols shares insights.
Moving Picture Company (MPC)has appointed Lucinda Keeler as Head of Production for its London studio, bringing over 20 years of experience and leadership in the VFX industry.
REALTIME studio has launched a Virtual Production division, following its grant from Media City Immersive Technologies Innovation Hub to develop a proprietary Virtual Production tool.
ZibraVDB plugin for Virtual Production and CG studios delivers high compression rates and fast render times, making it possible to work with very large volumetric effects in real-time.
Maxon One 2025 updates Cinema 4D, Redshift, Red Giant and Cineware, and releases ZBrush for iPad, putting ZBrush sculpting tools into a mobile device with a new UI and touch controls.
Das Element asset library software version 2.1 has new video playback controls, hierarchy tree customisation for libraries, faster set-up processes and simpler element migration.
Autodesk returned to SIGGRAPH 2024 to show software updates that include generative AI and cloud workflows for 3D animation in Maya, production scheduling and clip retiming in Flame.
Shutterstock launched a cloud-based generative 3D API, built on NVIDIA Edify AI architecture, trained on licensed Shutterstock content, as a fast way to produce realistic 3D models with AI.
Freefolk has promoted Rob Sheridan to VFX Supervisor in their Film and Episodic division and Paul Wight is now the company’s first Chief Operating Officer.
Golaem 9 includes a new animation engine, helps create new shots without simulations, and has better crowd control, helping artist improve the quality of their animated hero characters.
Adobe’s Generative Remove feature, based on the Firefly AI model, is now part of Lightroom, and the AI-powered Lens Blur tool has now become generally available with new presets.
VFX company Pixomondo and Canada’s TV & film equipment rental business, William F. White International, a Sunbelt Rentals company, announced a collaboration to build LED volumes across Canada together with a combined initial capital commitment of more than $30 million.
Under the new arrangement, William F. White and Pixomondo will co-rent LED volume equipment, with two new stages already under joint construction in Vancouver and Toronto. Several other new sites across Canada will follow in 2022.
The high demand for virtual production shoots, coupled with strong bookings at Toronto’s first LED Volume in the city’s West End, prompted the decision to formalise the working arrangement over the long term, for the whole of Canada.
The second Toronto stage will be completed in July 2021 and is designed for shorter-term bookings and projects with quick turnaround times. It will be based at Studio City’s recently completed development at the east end of the city's downtown core, minutes away from Pixomondo’s Toronto office at Revival Studios and close to William F. White’s Toronto office and Pinewood Studios.
The new Vancouver stage, which will be the largest build of the three sites to date, is under construction at an undisclosed location and is scheduled for completion in September 2021.
As with the first Toronto stage, the two new volumes will be operated by on-set Pixomondo personnel for all productions, including those using the company’s 3D environment creation services. Pixomondo and William F. White have designed and will build the volumes in close collaboration with their technology partners Brompton Technology, Epic Games, Lenovo, NVIDIA and ROE Visual.
Jonny Slow, Pixomondo CEO, said, “We are delighted to be able to formalise our collaboration with William F. White and their excellent track record in film and TV equipment rental and studio property management. Combined with Pixomondo’s award winning VFX offering, and a game changing capital commitment from both parties, we are excited to confirm additional virtual production investment into Canada and into these state-of-the-art physical production techniques.”
“As leaders in our respective industries, our two companies are such a natural fit when it comes to providing a full-service client experience that is truly unmatched in the world of virtual production,” says Garin Josey, Executive Vice President/Chief Operating Officer, William F. White. “This next phase we’re embarking on together has the potential to radically change the production process. These LED volumes in Vancouver and Toronto are only a preview of what’s to come.”
“Over the past few months, we’ve been building a fantastic relationship with Jonny and the team at Pixomondo and we’re eagerly anticipating a bright future together,” said Rick Perotto, Vice President, Business Development, William F. White. “Our local offices in Vancouver and Toronto will provide virtual production clients with access to a world-class selection of production equipment to support their needs.” www.pixomondo.comwww.whites.com
Visual effects and animation company DNEG has announced plans to expand its India footprint into Bangalore / Bengaluru. The ‘Silicon Valley’ of India, Bangalore, is a global hub and centre of excellence for technology and innovation and DNEG is in the process of onboarding a talented new team of artists and technicians to work alongside its established teams in Mumbai, Chennai and Chandigarh, and across the world. DNEG was recently recognised with both Oscar and BAFTA award wins for its visual effect (VFX) work on ‘TENET’.
The recent explosion in global demand for premium content, driven in large part by the incredible success of over the top (OTT) media / streaming companies, has resulted in increased demand from content creators for high-quality VFX and animation services. DNEG’s new Bangalore team will have the opportunity to contribute to a variety of high-profile international projects across feature film, OTT and feature animation.
Commenting on the expansion, Merzin Tavaria, President Global Production and Operations, DNEG said, “Two of the cornerstones of our success at DNEG are providing a platform and exciting opportunities for our industry’s brightest and best talent, and strategic expansion and integration of new sites and services at just the right time. We’re extending our footprint into Bangalore to provide both further capacity and support for our global teams, and to explore the potential of the incredible talent in the area while providing opportunities to work on the biggest international projects. Our new team in Bangalore will be working alongside the best in the business, and we look forward to welcoming them onboard.”
DNEG is currently in production on international blockbusters such as ‘The Matrix 4’, ‘Venom: Let There Be Carnage’ and ‘Uncharted’ and OTT shows like ‘The Wheel of Time’ and ‘Doctor Who’ Season 13, as well as feature animation shows such as ‘Under The Boardwalk’ for Paramount and three other yet to be announced projects. www.dneg.com
Luxion KeyShot 10.2 includes major improvements for caustics, a redesigned Materials Library, a Configurator UI style for touch screens and a new Mesh Simplification Tool.
View of Sydney Opera House from the Sirius Building.
Architects traditionally use highly detailed, accurate visualisations to bring their blueprints, sketches and concepts into the real world, and sell their designs to clients. Production company Binyan Studios specialises in creating such visualisations, applying their own distinctive approach to depicting how concepts and designs will look once they are built. Their work ranges from 3D renders, film and animation and broadcast-quality commercials to immersive VR tours.
Under the direction of CEO and Founder Andrei Dolnikov, Binyan’s teams work from facilities in Sydney, Melbourne, Brisbane, London, New York and Los Angeles, and service clients including property developers and architects who need to sell, lease or promote new developments. Sir David Adjaye, Frank Gehry, B.I.G. and Zaha Hadid are among the architects they have worked with.
The studio has developed an Autodesk 3ds Max pipeline they use to transform concepts from into narrative visualisations. These visualisations are then used as imaginative marketing tools to tell stories and create experiences that take clients on a journey through their architectural designs, ultimately selling them on the concept.
Trend Setting
Binyan recently completed renderings that transform the Brutalist-style Sirius Building in Sydney, originally built in 1980 to house working class residents, into a new development that features spectacular views of the Sydney Opera House and Harbor Bridge. Andrei said, “In Brisbane, we worked on a vertical garden that was designed for Aria Property by architect Koichi Takada. We’ve also recently worked on a photoreal animated film for the Rivière by Aria in Brisbane, as well as renderings for a resort-style development for the Howard Hughes Company in Honolulu.”
{media id=166,layout=solo}
Binyan has initiated trends that are seen everywhere in today’s design visualisation industry. For example, Andrei and his team focus on a more artistic approach when developing visualisations, using photography as inspiration to transform architectural sketches into full 3D renderings that evoke emotion and memories in people, and convey a narrative story behind the work.
“The images we create may be commercial art, but still need to follow the principles of composition, lighting hierarchy and narrative,” Andrei said. “I’m always looking outside our immediate industry for inspiration, viewing things like new product launches, art installations and the entertainment industry to bring new depth into our work to really capture clients’ attention.”
Photo Inspiration
“Photography is all-important in our work, used as reference to achieve the photoreal look we need,” he said. “We look at great photographers' work for inspiration as well to develop a specific look for each project. Our projects are so diverse in terms of type, location and demographic that it’s vital we capture the essence of each venue we are trying to bring to life. A San Francisco sunset looks different from a Dubai or Melbourne sunset, and the harbour lighting in Sydney looks different from a Brooklyn morning sunrise.
“We capture the photography for backplates, plus foreground details, and use this material for photomontages. For preference, we’ll shoot our own images wherever possible or brief a photographer. Then we matte paint or composite the images in Photoshop to create still images, and for our animations and films, we use video compositing software.”
Riviere, Brisbane QLD
Design Workflow
The typical design workflow begins when Binyan receives a client brief, detailing the project, brand guidelines, overall look and target demographics. Once the tone, artistic direction and project style are approved, Binyan begins the 3D modelling phase in 3ds Max – the studio’s main content creation tool – to model each of the distinct areas that will be illustrated in the final visualisation.
“We typically receive 3D DWG files or Revit files. The more precise they are the better, but even when we get a client model, we do a lot of our own modelling to be able to inject the extra level of detail our clients expect. We are geekily passionate about detail ourselves – it makes all the difference.
“3ds Max works especially well for us because of its complexity – it has tools we use to approach projects in a granular way, taking an up-close look at every minor detail. Everything is controllable, everything is possible. Just about whatever a client asks for, we are able to deliver,” said Andrei.
Real-time Iteration
One 3ds Max feature that stands out for Binyan’s style of work is the ability to view iterations in real-time. The ActiveShade feature starts an interactive rendering session in a viewport that allows you to see your scene in near-final render quality as you work. Whenever you adjust vertices and apply transforms to geometry, or adjust lights, camera or materials, directly in the viewport, the results will be automatically updated to show the final render. The rendering quality and interactivity depends on the renderer you are using.
Sirius Building, Sydney
3ds Max also has robust retopology tools that automatically optimise the geometry of high-resolution models to create clean, quad-based meshes. For example, the tools remove artifacts and other mesh issues ahead of animation and rigging, or when applying textures and manipulating objects. When the model's vertices, edges and faces are well- organised, animations are more fluid and rendering needs less memory – holes, spikes and isolated elements take extra processing time.
Since retopology works best on objects with uniform, equally-distributed faces, modifiers are available to optimise the mesh first. The Subdivide modifier creates smaller, more even triangles, and Smooth applies smoothing groups to the mesh. Mesh Cleaner will check and fix errors in data imported from other applications.
Rendering and Post
During composition, the Binyan team explores the best time of day, lighting and angle to portray the architectural model, later managing colour production, texturing, lighting and landscaping.
Once the client has signed off on a near-final draft, the studio renders all files and completes post-production, using tools like Adobe Photoshop for stills, Houdini and Phoenix for animation, and Fusion for compositing, among others. For digital experiences or VR projects, the team takes over the installation of the physical hardware and custom software development to help the project come to life exactly as required.
Ward Village pool house, Honolulu
Story at the Core
As storytelling hasn’t traditionally been associated with architectural visualisation, Binyan has worked out their own processes and made it a signature in their work. Andrei said that storytelling is in fact at the core of what they do and the foundation of every image, not to mention their films, animations and immersive experiences.
“Even for a humble bathroom CG still image, we ask ourselves – who is the audience, how do we want them to feel when they look at this image, and how will we achieve that through composition, lighting and detail?”
To answer those questions, the story remains front and centre. While it’s always a collaborative process, Binyan typically originates the key concept, based on a thorough brief from the client. “This brief includes the brand identity and graphic design elements for the overall campaign from the design agency and the architectural vision from the architects. Our in-house Directors take these ingredients and propose two to three ideas or treatments for the client group to review and from there, the chosen direction is fleshed out into a storyboard and animatic.
“Then we coordinate the production of each scene, which can range from purely CG/3D scenes and VFX scenes, to live action scenes involving talent, hair and make-up artists and a full cast-and-crew shoot on location. The shoots usually last two or three days and frequently include green screen shoots that our VFX supervisors lead.”
Architecture in Motion
Ward Village pool at sunrise, Honolulu
Binyan’s use of motion and animation makes their projects varied and interesting, incorporating camera moves, FX and procedural effects, lights, motion graphics. Decisions about the approaches are all driven by the idea as much as the budget, beginning with concept development and becoming concrete at the storyboarding stage.
“As the budget is a known factor from the outset, we’ll pitch an idea based on this – the greater the scope, the more complex and memorable the FX scenes will be. Motion graphics are determined by the kind of production we are working on. If the objective is to explain a complex scheme and to 'introduce' the audience to a project, then we will propose more motion graphics components.
“On the other hand, more ‘product-based’ films – where the audience is already informed about the developer and architect and simply wants to fall in love with the architecture, views and so on – we will go for a more pure 3D approach like the Riviere animated film.”
Time Factor
Because projects often take years to develop, time is another work factor. Helping clients use visualisation to get started on major projects while they are still in an early design phase adds to the challenge. “For example, we are currently working on a large, complex project in California that will completely transform its precinct. Still in its early days, many elements are at a more advanced stage than others.
Burleigh Heads, Gold Coast, QLD
“The project will be developed over several years, but meanwhile they need us to create accurate visualisations of how the final product will look. This process is both exciting and challenging, and it takes all of our experience to bring it to life. In the end, though, it’s very gratifying.”
In the midst of a global pandemic, the company has witnessed increasing demand for visualisations, which can be easily accessed by remote clients and shared online. He describes the work environment as demanding and high-performance, calling on them to handle a diverse body of work – “everything from a small bathroom to a massive masterplan on top of a mountain in Saudi Arabia,” he said.
It’s a super exciting time for us as well. Animations and digital experiences like immersion rooms, sales gallery activations, media tables and so on, are becoming a component of many projects we work on. We are really investing in our talent, hardware and software to be able to grow in this space. Real time rendering is a big part of this too, and is finally becoming a realistic component of a photo-real workflow. We see very, very fun times ahead.” www.autodesk.com.au
In March 2021, Foundry introduced machine learning into Nuke 13.0 with the addition of the CopyCat node, a tool that takes the process of building custom tools beyond scripting, Python extensions and SDKs. CopyCat hands users a process whereby they can show a network exactly what needs to be done to complete a task, how it should be done and to what standard.
Starting with reference frames from a sequence, artists can use the CopyCat node to train neural networks to automatically complete tasks ranging from rotoscoping, clean-up and beauty work, to correcting focus and other camera effects. The artist inputs a selection of the original frames, each matched to a frame showing what he or she wants them to look like, called the Ground Truth, and effectively trains a network to replicate the desired effect across all frames in a shot.
Training neural networks to work on video is an application of machine learning (ML) that the software developers at Foundry have been interested in for some time. Digital Media World had a chance to speak to Foundry’s Head of Research Dan Ring about the history of ML at Foundry, what they are working on within this field now, and what directions they plan to take it in the future.
From Research to Studios – Bridging the Gap
Foundry’s research team launched a project in 2019 called ML Server – an open source academic exercise for users to test. One of its goals was to determine the level of interest that exists in their market for machine learning. Dan said, “ML isn’t a new development, and many people are still thinking in terms of ‘deep learning’, layers of operations stacked on top of each other to mimic human neurons. So far, the way human neurons work has proven incredibly complex to recreate, but the research being devoted to tasks that rely on image recognition has been more promising.”
“That research led to thinking about image-based tools for compositing. Images are a dense source of precise information, and image recognition is a common element of VFX work and editorial. Furthermore, creating masks base on images is a major element of compositing. However, bringing the findings of academic research into practical applications has been difficult, and bridging that gap was a further motivation behind ML server.”
For instance, the computation and code for academic applications and experimentation don’t have to be extremely robust, whereas to be useful in VFX applications, coding has to perform consistently in virtually any situation. ML Server helped define practical constraints as well, such as graphics hardware requirements, which operating systems were relevant and what interests artists the most about ML for their work today.
CopyCat can be used to create a precise garbage matte for the woman in this video. (See the image below.)
Supervised Learning
As a consequence of that project, Foundry felt compelled to put ML into a set of tools that artists could use in their own way to solve their own problems. “Foundry did not want to be sitting in the middle of, or interfere with, that problem-solving process. Instead, the purpose of CopyCat is to allow artists to train a neural network to create their own sequence-specific effect in Nuke.”
At the simplest level, ML is supervised learning. You have some data you want to use to develop a model to achieve a certain task. So you go through a process of training, assessing the results and refinement. For the model, learning at this level happens largely through making mistakes and being corrected.
To address the challenge of transferring that process to VFX production, we can use the model’s ability to improve itself and, more specifically, to improve by doing the task your way. “Here is the data I have, HERE is what I want you to pull from it.” An example of a practical application would be - “I give you an input image, you give me a useful mask based on that. I’ll show you some examples of what I mean by a mask, and then you create a network that can generate a mask from any of the other frames.”
Shot-Specific
This was the premise behind Foundry’s ML Server. In fact, the pool of artists who were testing the program loved using it. Even though it wasn’t quite ready for production yet, they started trying to use it in production anyway. Once they understood that users do want access to ML in their software, Foundry wanted to find out exactly what they were using ML Server for. That was where CopyCat node came from – the result of transferring those capabilities into Nuke as native functionality.
“The core principle of the node is based on take an image from your source media, and then giving it to the system with a specially created frame – for example, a keyframe – as an ideal outcome,” said Dan. “Once you have given it several more examples – in the tens, not hundreds – the network will then try to copy your work across all frames as it continues to develop, but in a manner generalised only as far as that shot extends as defined by your samples. In other words, the system will be shot-specific.
“That is a key difference between CopyCat and a traditional ML model. No generalising is done for the wider world. You are training this limited-application model for just what you are doing right now. This approach allows artists to work at scale, accelerating their work with the option to use the same trained network on multiple sequences.”
Inference
The CopyCat tool consists of two nodes. Training networks is done with the CopyCat node – you give this node your example pairs and generate a .cat file. But while this node has intimate knowledge of the training pairs and the relationship between the two images in each one, that is all that the .cat file knows about. To do actual work, it has to be passed on to the Inference node, which applies the .cat file’s knowledge across the rest of the frames to create the effect modelled by the network.
Foundry split these functions for two reasons. One is hardware efficiency. The learning process that builds the knowledge into the .cat file takes more compute power than applying the effect. After that huge learning effort, the Inference node can in fact run on a laptop.
Dan said, “Another reason for separating the CopyCat node from Inference is to make the .cat file more flexible and recognise its value as a piece of IP in itself. Nuke has native support for your .cat file, and many existing models trained in other software can be saved as .cat files as well by adding certain types of metadata. Both can be used in Nuke with the Inference node.
The Value of an Artist
“Instead of working as a predefined tool that does only one thing, CopyCat puts machine learning within reach of an artist’s imagination and can therefore do lots of things, depending on that artist.” This process empowers artists, but can’t replace them because the trainer is the artist, not the model. The artist is isolating the tedious, repetitive portions of the work and teaching the model how to complete those portions to the necessary standard.”
The idea that artists will somehow lose out through the adoption of ML tools is obviously of primary concern. A roto artist working manually will normally finish 60 to100 frames day, and thus a 200 frame sequence might take two the three days. Using ML can bring that down to two hours.
However, Dan also said, “One thing that our rotoscoping example highlighted to us was the value of the artists we were working with. A good roto artist can elevate a shot to something beautiful and totally believable. It takes a long time and enormous skill to be really good at the work and, therefore, to be able to train a model effectively. As the network can only imitate the style of its trainer – precise and conscientious, or haphazard and inconsistent – the work needs a good artist as well as someone who can select the most relevant series of reference pairs.”
Beauty repairs achieved with CopyCat.
Applications
Another application Foundry has tested with CopyCat is beauty repairs, which can be labour-intensive due to subtle changes in lighting that may occur across frames, or to shifting camera angles. Artists can remove beards, tattoos, bruises and scars this way.
An even more valuable application is correcting focus issues in video, for example, bringing a blurred shot into focus. The artist searches through the video to find the frames, or even just parts of frames, that are focussed correctly. As before, show the network before/after pairs, as in the roto mask and beauty examples, and ask it to discover relationships between the source material and the desired outcome.
“If the network turns out a bad frame or two during training, you don’t have to re-start the process, said Dan. “The system is cumulative, incrementally building up expertise. You can just correct the bad frames in the same way as the others, and add them to the training file – that is, the .cat file – of before/after pairs. The system will continue from there.”
Creativity
The other, critical side of artists’ contribution to projects is creativity. Networks and models are capable of extreme consistency – that is what they are good at and what CopyCat takes advantage of. Creativity comes from elsewhere.
Masking, beauty work and re-focussing have traditionally been hugely time-consuming, costly issues for compositing. The cost comes not only from having to pay for hours of manual work, but also from missing out on creative ideas that never had a chance to be developed ahead of deadlines.
“That potential was evident when Foundry found that the artists testing CopyCat were using it for a much wider variety of interesting, specific tasks – de-aging, animation corrections and anything that benefits from a repetitive, automated approach and absolute consistency. Training a tool to detect and flag animation errors, or even go on to fix them, at the QC stage is a use case with a lot of scope that just needs creativity to expand,” Dan noted.
Extensibility and Data Quality
Foundry’s developers want to develop scope for extensibility in CopyCat for artists who train networks, whether these be .cat files originally built in CopyCat or others. They especially want to encourage artists to bring their own models to the pipeline, not just bring data to Nuke and wait for an algorithm to be developed for them.
Training a model to de-blur video.
“But it has to be done properly, just as developers have done with their C++ SDKs for Nuke, scripting and Python Nuke extensions,” Dan said. “These now have conventions for use as part of Nuke. We’re aiming to do something similar for users’ ML models as part of our effort to bridging the gap between academic research and production."
The high quality of the data that VFX artists give their networks also gives them a head start in terms of training time and use-specific tools. Video from media & entertainment projects usually has a high dynamic range and resolution, accurate lighting properties and approval from a director and DP. In the past, neural networks were set up to chew through large volumes of relatively low quality, available images, and could take a long time to train into a workable model. When it comes to creating useful tools, however, starting with a good artist and good data will shorten training time.
VFX Wish-list
“VFX studios have fantastic data sets for training networks. Foundry wants them to know that they already have the means to train models for many applications. We don’t have that data ourselves, but have developed CopyCat to help them take advantage of those data sets. The quality of their data, combined with their artists’ skills, can make them more competitive," said Dan.
Ultimately, tools like CopyCat will not remove artists’ jobs – it will make VFX work more creative. Today’s short timeframes mean the wish-lists associated with jobs may only ever be 40 percent completed. With ML, teams may be able to get to 60 percent of items on those lists. ML gives a studio an opportunity to move past the busy work and solve the harder problems, squash repetitive tasks and move on to the creative faster. It may also mean VFX artists have a new skill to acquire – quickly and efficiently training neural networks.
Smarter Roto
Dan also talked about how Foundry will continue to progress the use of ML in its software. An on-going project with University of Bath and DNEG, called SmartROTO, addresses issues of correcting models in the course of training. They wanted to figure out how best to correct the model by telling the system, in a meaningful way, that it’s not working so that it understands its work has fallen short of the goal, appreciates the problem and tries to improve its work.
Training networks to recognise arbitrary shapes that change over time requires interactivity.
He said, “It called for something more interactive than preparing a new ‘after’ frame and making sure that all feedback is detected, interpreted and applied correctly. We needed to give the network a region of ‘correctness’ and developed two models, Appearance and Shape, both of which test whether a given image is visually similar to the desired outcome. But they test for that in different ways.
“The Appearance model determines whether a given arrangement of pixels looks like something we've seen before, while the Shape model constrains the set of possible matches, saying 'force the current silhouette to match a silhouette I've seen before'. Typically, ML models for images only use Appearance models, that is, they're given an image patch and asked to classify or transform it – for example, determining whether or not an image contains a dog.“
Appearance models work well in most cases, but have no meaningful concept of the shape or structure of the element being worked on, which is much more important for roto. The Shape model encodes this idea of structure, and validates what the Appearance model has found.
“If we are rotoscoping an arm, it will only encourage arm-like shapes and reject others," Dan said. "In SmartROTO, we show what an arm looks like by giving the system a handful of keyframes to use as its base model. Every additional keyframe from the artist updates the model further. By the end, it has a solid understanding of the object and all the possible ways it deforms in the shot.” www.foundry.com