HARRY POTTER - A MAGICAL ENDING

When getting started on the movie, each vendor’s brief required updating their own effects from a previous film, updating another vendor’s effects or, best of all, creating something totally new. But what was essential was not to change the looks of any single asset, creature or character enough to distance the audience from the story they love.

As Sean Mathiesen from Rising Sun Pictures remarked, “Most of the Potter world is a familiar place to viewers, with a known visual language. A VFX team has to figure out how to make it instantly recognizable and look exactly as it did before - and then slightly adapt and improve on it to continue to impress a more sophisticated audience.” While artists want to take advantage of techniques and tools that may have improved since the previous film, the goal is not to change things but only make them better.

Smoke and Cloth
Sean Mathiesen, VFX Supervisor at Rising Sun Pictures, first got involved with the movie, his first in the franchise, in May 2009. His team completed 165 shots in all. He travelled to meet VFX Producer Emma Norton and production VFX Supervisor at Leavesden Studios in London just as the shoot was starting and a few sets had been built. Tim and Emma’s team were responsible for on set supervision for all vendors’ effects teams and supplied comprehensive set information via ftp, as a means of controlling and monitoring franchise material for consistency.

RSP had an interesting brief for their contributions to the opening sequence. For the first shot, the camera descends through clouds to reveal the title card by Double Negative and a helicopter shot over the forest but without either Malfoy Manor or, more significantly, a mysterious flying smoke-and-cloth creature, who comes and lands at the front door of the Manor. It was RSP’s job to add these as 3D CG elements.

“We built the Manor house as a 3D model, tracked all the plates, converting them day to night, put the 3D building in, and did some Nuke projections, patches and matte paintings to enhance the environment and integrate the building. This part of the brief was pretty straightforward if not easy,” said Sean.

Timed Blend
But for the creature, they had only a description of “a creature creating itself from smoke and cloth referencing a bat” and a green screen shot with a set of hedgerows and the actor walking through it as the camera moves around him. Because the production had never shot a corresponding clean plate, the team had to build a digital environment as their own ‘clean plate’. Then they needed to create the smoky creature effect and gradually replace it with the full live character in a digital handover using a match move.

This character is, of course, Severus Snape, revealed as a Death Eater in the previous film but with special characteristics. “We created an effect inspired by the Death Eaters, but unique and different because it was Snape. David Yates described it as a bat-like creature and we wondered for a long time if he meant it was actually supposed to be a bat -should we have a bat inside smoke? We did tests and tried to emulate Double Negative’s work on the Deatheaters.”

They rigged a cyberscan of the actor to move in the same space on their plate as the character, and used this to generate the smoke passes. The cloth team then animated their cloth accordingly around the character. Thus, they had Houdini smoke effects passes moving up from the ground based on the match move character’s geometry, and the cloth simulation around that, while making sure they overlapped correctly in time. Sean said, “It was a timed blend, all arriving to coalesce in the character as he walks forward up to the Manor door.”

Opportunity
Although the brief was a bit vague, it gave the team a fantastic opportunity to contribute new ideas. “Somehow, you start out in the sky where he is an undefined bat-inspired creature and end up with Snape, clothed and walking into Malfoy Manor. Figuring out what kinds of effects would help engage the audience and draw them into the story took a fair amount of development. For me, the challenge was to determine what it was that we were trying to do and how to it. But I worked out a concept for the director and Tim, and with our crew figured out what was physically possible to execute. In the end, I believe it opens the movie well.”

He started visualizing the idea in Cinema 4D. “I brought match moves, published geometry and digital doubles that we had created at RSP, into Cinema 4D to experiment with ideas on separating smoke, building it from the ground, then bringing in cloth to look at how cloth and robes would merge into the same character throughout the shot. I exported the tests using FBX. The Maya cloth team and 3D lead responsible for the Dementors’ robes established a set of falling natural, realistic robes using nCloth that would feel and would merge into the handover to the real character Snape.
“Our Houdini team grabbed the same scene files and the same motion and were able to build a wave of smoke on the ground that built up into the solid parts of flesh of Snape. The cloth team were able to drape robes over the top of the growing Houdini smoke creature, until you eventually end up with a real live actor. The entire effect was composited with a few real elements to tone down the CG edge, using Nuke. We also used Nuke for the day for night conversion by changing settings for less contrast, directional light and so on, and replacing the sky with a matte painting for a stylized night look."

Fluid Drapery
The dark skeletal creatures called the Dementors, shrouded in black cloth, are well known to Potter fans and are first seen in this film looming above Lord Voldemort’s courtroom. They were first created by ILM, and RSP are the third company to handle them, taking over the original design and incorporating it into their own pipeline. They began by testing the cloth effect to get it working properly and give it the looks David Yates was after.

He wanted it to appear to be moving underwater, which meant developing its look in Maya nCloth to both resemble the look of past films and take on this new quality. “To make the Dementors as terrifying as they need to be, they must look like real characters, present and well integrated into the plate,” said Sean. “Our compositing techniques have improved since the previous film, shifting from a Shake to a Nuke pipeline and we wanted to take use this new capability. But it was important to keep the overall look the same.”

They didn’t use any fluid software for the underwater look, focusing instead on the parameters in nCloth. They looked at early experimental videos of puppets draped in cloth shot underwater as reference for the Dementors, very helpful for achieving the elegant, scary look they wanted.

Cavernous Interior
The Ministry of Magic is another familiar location to fans, who have already visited parts of its cavernous interior in other Potter films. Sean said, “Recent tools, especially Nuke, gave us a chance to improve set extensions, projections and handling 2D and 3D data to make it feel like an even more real place, with no clues that any of it was digital. We used small matte paintings to cover objects, integrating live action elements into massive set extensions to make it feel populated everywhere, not just at the location of the lead performances.”

“We would be handed a plate of the hero shot with an area of green screen where we needed to extend in the distance or add a required section. But as soon as it’s complete, you notice that it is empty of people. So as well as providing the plates, the production VFX crew shot elements of people on green screen for us, with appropriate lenses at the right distances.”

Here, we’d put them into Nuke so we could track them with the cameras to fit the shot and integrate them. Lighting references were also supplied to recreate the scenes as on set, but the sets, built on a sound stage, only went so far, while the Art Department’s concept for shots may contain places ten times as large. Even without specific lighting detail for the extensions, just references as start points, it was still our job to make them look good

Initiative
RSP were assigned to create the Horcrux through their own initiative. After receiving their initial award, the company didn’t feel their full potential as a facility had been tapped. They had more seats, more computing capacity and were prepared to hire more artists. They discussed the artistic and technical demands of the movie’s remaining unassigned shots and sequences with VFX Producer Emma Norton. One of these reveals one of the Horcruxes, part of the soul of evil and powerful Lord Voldemort which Harry, Ron and Hermione have been searching for, emerging from a locket.

While Sean was with the production in London, RSP’s Art Director Nick Pill worked up ideas on looks for the effect, researching and reading the Harry Potter books, and discussed different interpretations with Sean by telephone. The script provided little detail, so Nick presented lots of options, one of which Tim Burke and David Yates found had exactly the right tone – tortured, frightening, creepy, contorted and full of tension – but everyone agreed it had to be bigger, as much as 60 feet tall.

Elegant Tempest
Once Nick and Sean had taken all these comments on board and won approval on their final concept, it took another 12 months to refine while they experimented first with designs for different types of monsters, then for something more like an event or tempest, reflecting on Voldemort’s visual language from the past films. Then they noticed from the books that the Horcruxes could create themselves from their environment.

Because this sequence takes place on the surface of a pond, they wanted to show that it escapes from the locket not only as a part of Voldemort’s soul but also of its surroundings, twigs and leaves, snow, slime, pond scum and algae – all resulting in an entity capable of terrifying and confounding Harry’s friend, Ron. Sequence Supervisor Dennis Jones worked with a team of Houdini artists to construct a CG event from the concept art with a sense of chaotic motion, able to project terror.

Among many elements to incorporate into the effect, the team was given elements of Harry and Hermione as lovers, shot very late in production. When they arrived at RSP, integrating them into the rest of the Horcrux vision presented an unexpected challenge. While the elements up until then had been horrifying and violent, these lovers were insidious, slow moving and tranquil, preying on Ron’s fear of losing Hermione to his friend. They were to be seductive and elegant, but disgusting because of their slimy teachery – ultimately forcing Ron to doubt his own mind.

Perfect Balance
Houdini drove the movement of the effect. The critical element of Voldemort’s face, shown at different ages of his and Tom Riddle’s life, was based on MOVA motion capture data, as was Hermione’s face. The MOVA data was first used in Maya and animated to appear at the right position and time to frighten effectively, then moved to Houdini to generate the natural physics of the Horcrux event – that is, swirling within and reacting with the environment.
Starting with the HDRI data from the set as the foundation of their lighting environment, they added some specific key lights to emphasise certain moments in the sequence. Objects left in shadow from the direction these light came from, required some further tweaking to make the event readable.

From original design development to the last shot, one of the last they delivered on the film, the team spent roughly 16 months on the Horcrux to achieve the perfect balance of terror and attraction, while still maintaining the overall ‘Harry Potter’ tone and rating. It had been an exciting challenge for RSP’s team.

Hot Pursuit
RSP were awarded a number of smaller, one-off effects that heighten the presence of magic surrounding the three heroes and filling the environment. Anything could happen at any moment. At the Ministry of Magic for example, the three friends steal the locket of Slytherin holding the Horcrux and run from the courtroom with the Dementors in hot pursuit. They barely escape into an elevator, sliding the door cage closed just in time. To repel their reaching arms, Harry Potter evokes his Patronus in a short blast, another form of this effect, that knocks the Dementors off their feet.

RSP rotoscoped the characters off the plate and created the ball of light in Houdini with an explosive blast, attaching this to the end of Harry’s wand. Then they illuminated the elevator interior with a bright blue light and created sharp shadows from the roto outlines. A Houdini distortion pass warped the image to further the blast effect.

Flushed Away
The decoy detonators in the joke shop, which RSP had developed for an earlier film, are small comical windup toys Harry uses to create a distraction as he prepares to sneak into an office. They had to replicate themselves, take off across the floor, and create a loud noise giving off smoke. The production also needed them to create comic relief in an otherwise dark and sinister story. However when they received the plates, it would have been impossible for them to get across the distance required, as the tiny toys actually had no way to move around. The animation team developed a very comic walk cycle for them.

Another one-off effect in which a character flushes himself down a toilet, turned out to require more trial and error than expected. They considered ideas from green fire to a full body collapsing into water. The director liked Sean’s idea of the actor spinning down into the loo, but rejected any kind of camera move. It had to be a locked off shot. So they began with the real actor on set who pulls a chain before a handover to a digi-double and a clean plate. “That let us spin the double as we needed, with deformations, using Houdini. Then we created the water spinning inside the loo,” said Sean.

Malham Cove
Cinesite’s 2D Supervisor Andy Robinson and 3D Supervisor Holger Voss began working with the production VFX team in London in May 2009, and started on shots in June. They completed more than 90 shots in total.

In one sequence, Harry and Hermione spend a few days at a ‘windswept hill’ location where the environment needed to resemble the distinctive cliffs at Malham Cove in Yorkshire, which feature a remarkable, natural limestone pavement terrain. Cinesite sent Andy, Holger and their photographer to spend five days up in Yorkshire gathering panoramas, textures and photogrammetric detail to almost completely reconstruct the set location to look like Malham Cove.

In the main establishing helicopter shot approaching the hill, for example, they had to rebuild virtually the whole of the cliff. On location they captured enough reference photography and footage from the ground and from the helicopter, to be able to automatically rebuild the geometry and obtain projection cameras for the scene using Cinesite’s csPhotoMesh software, rapidly reconstructing the cliff.

Rock Terrain
Then the desired limestone surface could be modelled over the top. For most of the five days the team spent at Malham Cove, the weather poured with rain, making the mossy lichen-covered rock terrain dangerous and slippery, one reason why the shoot wasn’t done there. But they did shoot enough material and the final day was fine enough to capture better skies and sunlight for the panoramas, which could be stitched together pretty much as shot and used in backgrounds with few matte painting replacements or adjustments required.

When seated on the rocks giving a view out across the surface into the distance, the actors were shot on a set piece built to resemble the rock pavement, on a backlot at the studio. In these cases, Cinesite had to extend the rocky set and add a matte painting for the sky.

Period Buildings
At Godric’s Hollow, the village where Harry’s parents are buried, the team extended the street beyond the actors in a single shot that required considerable effort, mainly due to the edit. The buildings in the extension had to match the practical set perfectly because this shot was cut directly next to a shot with the physical buildings shown directly behind the actors, but no clean plates or reference stills were available.

For this, they needed to create three or four CG buildings from scratch based on the surrounding shots taken further away, and on texture photos taken at Lavenham in Suffolk, a village with similar period buildings. They also added CG snow, falling in the same swirling motion to match the snowfall in the sequence.

While at Godric’s Hollow, Hermione builds a wreath with her wand for Harry’s parents’ grave. A practical wreath had been placed on set, but wasn’t exactly what the production wanted. It needed to appear brighter and more hopeful within the brooding, dark, blue-toned set, which made the wreath look gloomy. The team looked at images of wreaths shot in sunshine to get started on the look.

Once approved, the next step was coordinating the creation of the wreath with Hermione’s circular wand movements. They used a simple L-system to start the ‘growth’ but adjusting the timing of the opening of the individual flowers turned into a manual keyframing task. They wanted to achieve a time-lapse appearance but make it appear to be driven by Hermione’s actions. A small rig was made for each flower.

Snake’s Nose
A primary task for the team was transforming actor Ralph Fiennes’ human nose into Lord Voldemort’s menacing snake’s nose. They started with a generic build for the actor’s face, using a Lidar scan from which to derive the model for his head with his real nose, and a texture shoot to get the diffuse colour texture, specular texture and bump maps required for their proprietary skin shader, csSkinShader.

The starting point for each shot was the full CG head lit to match the plate. To introduce the CG nose, the compositors chose the best areas from either the CG head or the live action to use to hide the seams along the upper lip and natural creases of the face. The look of the effect had been developed originally at MPC, so the established design came from them, which Cinesite updated. The veins and other skin effects were achieved with make-up, enhanced digitally to create a coherent look across the composited head.

The real nose of the actor, of course, obscured parts of the face depending on the camera angle. In low angled shots it might obscure his eye, for example, which would have to be cleaned up and repaired in the composite. Also, whenever the actor turned to the side, his nose had to be removed with clean-up and the missing background painted in.

Facial Tracking
They also built a rig to matchmove his performance with, focussing on the critical area – the cheeks, upper lip and nose – to match the head movement and camera, and to then animate the facial performance. Facial tracking was done with 16 markers placed on the actor’s head over the makeup, to derive the basic motion and shape. They could place the markers in precisely the same position for each take by setting them into a face mask.

The camera was tracked first and placed spatially correctly into the scene to be able to make best use of the lighting for each shot. For the head line up, they did an object track using 3DEqualiser. Then they made a 2D track of the markers on his face, projecting them onto the rigid head track to derive the facial movements.

This work was handed to the soft track team to do the actual matchmoving of the performance, using a standard Maya animation rig. This way, one matchmover was doing the rigid head track as the basis of the lighting, scene set-up and rough blocking for the shots, and later they could decide on how complex and precise the actual facial matchmove and animation had to be to match his vocal performance and expressions.

Once the full workflow was in place and streamlined, it worked well and they could complete the 46 shots of Voldemort’s face fairly rapidly. The major R&D done for the film was the updates made to the csSkinShader for this effect.

From Dog to Doe
Cinesite was also awarded a Patronus, a doe, for which the Art Department hadn’t supplied concept art, so discussions focussed on changes to previous looks. Cinesite went through an iterative process considering many alternatives, evolving with the scene through to the end of production. The story called for an ethereal creature, ‘not entirely in our world’, leaving doubt about whether it was really there or imagined. The team approached it first as a fully built doe, with a fully formed animation running on top of it, and subsequent iterations subtracted from this, blurred it and diffused it, making parts almost transparent.

It also became a source of light in the darkened forest scene. On set, a dog wearing an LED light-suit was shot walking along the path the doe needed to follow. This gave a good approximation of the correct interactive lighting shining onto the leaves and branches. The dog’s performance also gave them a base for blocking the animation. The director could choose the best part of the take for the action and the team would animate on top of that.

The 3D team modelled the complete doe in Maya in full detail with references from production, who requested a graceful elegant doe. It included fur and a muscle rig just as if they were building a photo-real doe, to use as a start point for all CG work. To produce the required translucence, they then filled the doe with fluid and rendered it as a volume for the interior using another proprietary application csFluidShader. Caustics were rendered onto the surface and combined with the photo-real doe render in the composite to achieve the final look.

Animating this model on top of the dog’s performance helped position the doe very accurately into the scene. The animators had a Lidar scan of the set so they could place the doe in space with proper contact points. Animation tests and approvals were done with the fully rendered creature to check exactly how it was working in the scene. The Patronus doe needed a long process of refinement and was one of the first effects they started on, working on it through to the end.

Heroic Elf
Framestore recreated Dobby the house-elf, one of Harry Potter’s oldest allies and a key figure in the earlier films also, and Kreacher, another house-elf. Dobby’s main performance in this movie occurs when he comes to rescue the three friends from the Death Eaters at Malfoy Manor. Although he is called a ‘slave’, Dobby stands up and distinguishes himself by claiming he is a free elf, come to save Harry Potter and his friends, an astonishing thing to say considering the dangerous circumstances. As they escape via a disapparation, Bellatrix throws her knife and strikes Dobby. When the group re-gathers on a desolate beach, Harry rushes to him but discovers he has been mortally wounded, and the heroic elf dies in his arms.

Throughout the dramatic sequence, the audience’s eyes focus on Dobby, which put pressure on the effects team. VFX Supervisor Christian Manz who led the artists over 16 months of work, said a major pre-production decision was that the two elves would be keyframe animated rather than performed through motion capture. This followed a proof-of-concept demo that Christian’s team, Lead Animator Pablo Grillo and CG Supervisor Andy Kind prepared for the director and Tim Burke in February 2009.

Pablo Grillo pointed out that motion capture reveals every small hint or twitch of movement that may make viewers uneasy about the look of fantasy characters. Instead, the animators, working in Maya, developed human performances from appropriate references and staged the sequences to work within the edit and the elves’ design and character. An older staff member provided photographic reference of skin elasticity around the jowls, cheeks and eyes in specific poses under controlled lighting, to dial corresponding shapes into the rig. Tim also suggested filming the voice actors performing on-set as a further guide.

Skin Deformer
“We rebuilt the rigs from the ground up,” said Christian. “Recently, our tools have evolved considerably, and rigs can be rebuilt procedurally at any time, allowing many iterations on the model - essential in refining the skeleton and facial features, and checking how the motions would work. Instead of building rigs inside out from bone to skin, we’re now building rigs from the outside in, working directly on the skin surface to maintain more control. The modelling and rigging work-flows are in fact getting closer - after all, rigging is modelling in motion.

“We relied especially on a skin-like deformer that makes any surface relax or slide like skin over hard objects, but in real-time and without simulation. We broke the body down into anatomical regions, and used the deformer to refine structure, painting and animating the skin behaviour triggered by certain skeleton configurations. We have also refined a 'facial formula' for facial animation. With all these controls, we had facial shapes, deformer set-ups for eyes and lips, and the skin deformer taking all these inputs.”

Humanizing
Before working out the sequence, the elves’ physical appearance had to be finalised. Because both Kreacher and Dobby had appeared in earlier films, no major alterations were called for, but some updates were due. Dobby is a central emotional focus and had to appear human enough to let the audience connect with him. With the original maquette ILM used to create Dobby for the second film and Framestore’s models of Kreacher, the team began a humanizing process based on the elves’ original topologies, smoothing Dobby’s neck and shortening his arms. Kreacher’s nose was shortened and ears trimmed.

Christian said, “For the bodies we created clean base meshes over the original scan data using Topogun, so we could add subdivisions to the model for finer detail work in Mudbox, or reposing the model inside Maya. Delicate areas like eyes and mouth were designed for a range of facial expressions and to look more appealing, and we ensured our modelling work-flow would allow us to transfer large sets of blend shapes onto any model changes, at any point.”

The Real Dobby
For eyes, perhaps the most critical factor, they used the live action reference and took more elements of a human eye into account. “The white of the eye and skin folds around the eyelids are integral to expressions. Wrinkles are notoriously hard to produce in 3D, so we used a combination of deformers and dynamic displacements to achieve this level of realism,” Christain said. “We also spent months determining cornea bulge, the eye diameter and pivots that work with the design and the character. Although these eye tweaks didn’t match the Dobby maquettes as closely as before, we felt we were finding the 'real' Dobby, and able to express finer emotions and acting, to which the audience has reacted very positively.

CG Supervisor Andy Kind worked on lighting for the elves’ scenes, especially interior shots. He said, “We built on our recent skin shading technology, using multiple subsurface scattering techniques. To light scenes, we used mostly ‘bleed cards’ of HDRI textures from the set mapped onto simple geometry on cards. By scaling and positioning this geometry around the characters in Maya, based on the real lights on set, we controlled the intensity of the light and the softness of shadows.”

Matching Textures
When Harry and his companions arrive at the beach and realise that Dobby has been struck down, he collapses into Harry’s arms, and they speak before he dies. The scene employed a Dobby sized body double and a dummy, which became the lighting reference when integrating the animated character into Radcliffe’s arms. The scene was shot on location in a style with lots of free camera movement, a challenge for the paint and roto department who had to remove the dummy and do a lot of body tracking to make so much direct interaction with the CG look believable.

Tim Burke shot a reference and clean plates, but the takes that David Yates chose were those of Harry interacting with the stunt double. To achieve the paintwork, they went to the studio to shoot texture reference of Harry’s wardrobe and hands to help rebuild parts of Harry that the double obscured by re-projecting and animating textures. Where CG and real skin and cloth textures are closely juxtaposed, shaders from a core library allow the artist to build up materials from the same components to preserve continuity of lighting, balanced in the composite.

Out of This World
To dramatise Bellatrix’s knife flying into the group as they transport away, the director asked that the shots look as if they’d been filmed in extreme slow motion. Christian said, “The apparate/disapparate look, developed earlier by MPC, was designed to work over a very few frames - once it was slowed down there were no guarantees that it would look as interesting. It had to be completely redesigned. The only reference given for the look was ‘nebula-like’ – it took our team many iterations to nail.

“What was exciting about the apparation was that there is no real world event like it - the only way to visualize it was by using CG. We originally tried nCloth simulations for the deformations that the apparating characters undergo, but no matter how far the cloth parameters were set, they still resembled cloth too much. So we did further volumetric rendering along the incident vector to give the streaking and fluid movement a nebulous feel, with some cloudy noise to make the characters appear ephemeral. Illumination FX and motion blur were applied over the top of this.”

“We settled on a simpler technique starting with a core or heart generating plasma impacting against the ‘skin’ of the cat, defining its shape, rendered in Maya and 3Delight with a Fresnel pass so you could define the edge but make the centre transparent to reveal the heart. At the edges was the dry ice effect, falling off to make it glow and feel larger than an ordinary cat. These effects were combined as different render passes, resulting in our cat.”

That gave a starting point for the particle simulation. Because the director didn’t want to see him fully resolved throughout the four shots, they started animating patches of his texture that grow over the body of the digi-double and were used to emit and attract Maya particles of the correct colour and position, rendered in RenderMan. The ghost remains obscured and mysterious up until the moment he extends his hand and becomes recognisable as Dumbledore.

The corridor was actually re-built in CG, based on the clean plate which had been lit and shot as it should appear. They could derive their lighting from this and use the recreated geometry to cast shadows and control the interactions with dust and particles against the walls and picture frames.