HARRY POTTER AND THE Deathly Hallows Part 2

The pipeline they set up from the beginning was geared toward the final film. The digital artists and supervision were working well together by then and able to continue straight through to the end. Double Negative’s VFX Supervisor David Vickery, CG Supervisor Rick Leary and Compositing Supervisor Sean Stranks all kept the same roles. “There was a lot of overlap between the two films and for almost nine months we were in full production for both movies simultaneously,” said David.

Gringotts Bank
Within the Gringotts cart ride scene, down through the vast cavern to the vault, Double Negative handled all environments and assets, all set extensions, the foyer, the cart and cave, through to shots of Harry, Ron and Hermione falling from the cart and approaching the dragon in his enclosure. The multiplying treasure sequence done by Tippett Studios is cut into this, but shots from there through the dragon’s heroic climb up out of the vault to the roof, and in the exploding bank foyer, were again the work of Double Negative.

Because so much of the bank foyer set had to be destroyed, Double Negative built a complete cgi version. Once the dragon escapes, the Diagon Alley roofscape where he emerges is digital also. David said, “A beautiful, detailed set had been built for the bank, plus a section of the dragon’s enclosure to provide for actor interaction, but although the photography was helpful, once the artists began adding the dragon, light, interactions with fire, destruction and magical spells to the plates, we realised it would be easier to build a cg environment and work within that. A number of plates had to be replaced almost entirely, only retaining the foreground section of the set where the actors were standing.

“For all the shots featuring the dragon, for example, replacing the live action set gave us absolute control over the integration of the dragon with its surroundings. It was much simpler to get the dragon to perfectly occlude and shadow the floor around it. Magical spell blasts would cast light onto the walls and the dragon’s fiery breath lights up and wraps around the set perfectly.”

Roller Coaster
The cave environment for the three friends’ tumultuous journey down to Gringotts vault required extensive, precise previsualisation. Environment Lead Kieron Helsdon went to Leavesden Studios during preproduction where he worked in the Art Department alongside Production Designer Stuart Craig and Tim Burke to develop the previs for this sequence and design the scope for all of the camera moves. The director wanted to create a thrilling, Indiana Jones-style roller coaster ride, so the whole sequence had to be carefully planned before the shoot as a previs animation.

“We didn't want to use a motion controlled camera in combination with our motion controlled cart rig as it put too many creative constraints on a shoot that was already pressed for time,” explained David. Instead they drove the cart motion using their previs and let the cameramen react to the shots on the day. “This created a much more fluid set of shots as the operators running the cameras could react to directorial changes much faster than we would have been able to re-programme a motion control rig.”

3D Mapping
Double Negative started out with a beautiful maquette of the cavern that the Art Department had built, a clay sculpt 6 feet wide by 10 feet high, mapping out the position of the vault, the waterfall and the precise location of the cart rails running through it. The previs was designed to follow these details exactly. Double Negative developed tools for extracting the animation data from the previs. “Stuart Love, one of our lead 3D artists, built a digital replica of the practical cart rig that SFX supervisor John Richardsons' team had constructed on set. We then wrote Maya scripts that would transfer all their cart previs animation into a format that would drive the practical cart to move in the same way. Thus, the previs animation became the live-action cart ‘animation’,” David explained.

“However, as soon as we shot the live action to match the previs, the editors changed and re-cut the action, moving various shots around which unravelled much of the continuity within the environment. Fortunately, Kieron had built the cave maquette as a modular system. He could move pieces of it around himself to fill new shots in, using it like a Lego cave so that shots could be moved but still create a sequence that felt continuous and coherent.” The cart sequence needed a team of about twenty 3D and 2D artists working for nine months.

Camera Tracking
One of the tricky aspects of translating the previs to live action was trying to figure out where the rails would be in any given shot. The rails were generated in 3D to sit exactly underneath the twisting live action cart axles throughout a shot. “There was a risk that the practical axles would be positioned such that the rails would foul the foreground frame or appear to pass through the camera. To get around this, we removed the axles from all but the extreme close-up shots and replaced them with CG versions, giving absolute control of where the rail and axles would be in any given shot,” David said.

To help with the match move and the eventual manipulation of the green screen elements they positioned two HD witness cameras either side of the cart through the whole shoot, which were locked off and synced with a bloop light. “As the whole cart was not always visible in our plates it wasn't always easy to tell when it was the cart moving and when it was the camera. Getting the relationship between cart and camera correct was critical to make sure the shots worked in stereo. The multiple sources of footage were used to determine what part of the match moves should be cart animation and which were to be camera animation, enabling us to re-create the cart and camera moves very accurately in Maya.”

Rock Formations
Kieron Helsdon was dispatched to Ballachulish on the west coast of Scotland to gather reference imagery of the slate quarries as inspiration for the slate-like rock formations in the cave, later interspersed with towering limestone stalactites. “We started out by building a digital replica of the Art Department’s clay sculpt. It was built in modular sections both to allow multiple artists to work on in concurrently and to make it more versatile when shots had to be recomposed,” said David.

Because Double Negative had no live action elements to composite into early look development tests, the shots they first presented to Tim and David Yates were all POVs. “The lighting was designed to recall a dark, dank cavern with an eerie glow, as if a distant source of light was bouncing off the wet surfaces of the cavern. These early tests guided the lighting for the live action plates.”

Test Renders
For interactive lighting on set, the crew positioned running lights on either side of the cart, flashing past the characters during the journey to give the sensation of motion. “We would always start with the lighting in the live action plates. If it didn’t look right in the context of the shot we would re-work it in comp, reducing the flicker and adding passing shadows to sync up with large passing structures.

“Once we had believable looking shots with the live action elements in place we would show the director for feedback on the overall creative look. In the end they went through half a dozen looks for the sequence, all of which took hundreds of test renders in 2D and 3D to perfect,” David said. “Making something look believable isn’t always the real challenge anymore. It’s delivering a creative solution that the Director and VFX supervisor like. You often have to work hard to understand what people want when they give you creative briefs...”
Double Negative dimensionalised some of their own shots, which added another layer of complication because different techniques were required. Those involving live action required rotoscoping characters then reprojecting the plates onto hand animated digital doubles to create the right eye renders, while full CG shots could be rendered in stereo.

Goblins' Dragon
Creating the dragon for this story was unusual because he had to be designed to emphasise the tragic nature of his existence up to that point, a trapped, enslaved creature prevented by the goblins from fulfilling his majestic destiny. Deprived of light and air, he has even lost the pigment in his skin and never flown, tortured and trained through pain to obey his captors. The arrival of Harry Potter and his friends is his chance to escape and transform himself into a proud dragon sitting atop the roof of Gringotts Bank, almost heraldic.

The extensive concept artwork from production depicted the dragon in a dark cavernous environment. Double Negative’s concept artist Kristin Stolpe used these as the basis for a series of sculpts in Mudbox, creating a 3D model to develop his character and show a poor, mistreated, emaciated wild animal. Work began on the dragon in summer 2008 with a small team dedicated to the build and scripting tools needed to animate the motion control rig used on set. Later the dragon team grew to include almost 100 crew, a small army of  lighting artists, creature FX TDs, compositors, match move and rotoscope artists.

The most challenging aspect of the dragon was developing a way to get the actors to sit on and interact with it believably. Double Negative provided SFX supervisor John Richardson and creature FX supervisor Nick Dudman with the digital model of the creature, which they used to build a life-sized 12ft CNC machined foam sculpt of the dragon’s back, a flexible foam laytex skin and a mechanical rig driven with six moog actuators to control the movement in the dragon’s shoulders, neck, spine and tail. This rig was then mounted on the motion control base at Leavesden.

Taking Flight
“Double Negative wrote a set of tools that would translate our dragon previs into movements that the motion control rig could re-create. The rig had a fantastic range of motion but was so heavy that it was never going to achieve the speeds we were expecting from our dragon. CG supervisor Rick Leary and 2D supervisor Sean Stranks oversaw our dragon shoot. Every shot had to have a new approach to make sure we extracted as much data as possible from the rig – we really squeezed every last ounce of speed out of it!” said David.

Some shots were filmed at 18fps so they could be re-sped later to make the dragon back appear to move faster. Other shots needed to be filmed locked off to allow the resulting plates to be reprojected into 3D camera moves. “Rick and Sean’s presence on set proved invaluable. Every time a shot was filmed, they would take the video rushes and run a quick match move and comp to check that they would be able to make it match the previs. The director was on hand to give immediate.”

Digital Hogwarts
The digital Hogwarts School buildings and surrounding environment that Double Negative created for this Harry Potter film sets it apart from all the others in the series. It was developed from the series’ miniature, which was in fact about 20ft long and 12ft high and had always been shown surrounded with a combination of aerial photography from Scotland and digital environments. The idea for the 3D replacement arose when Tim Burke and Emma Norton realised that Hogwarts appeared in so many sequences that it would need months of planning and shooting on the miniature stage. 

Double Negative’s digital set recreated over 10 miles of terrain at its widest point and used over 3,000 individually painted 4K textures. The school itself comprised over 70 individual assets, all built to three levels of detail as specified in nearly 1,400 blueprints provided to them by Production Designer Stuart Craig’s Art Department.
“Stuart’s team had been crafting the school for the last ten years and we had to live up to that. We were given such a wealth of architectural information that we could have built the school for real if we had wanted. The detail in the plans even covered profiles of the handrails that wound around the inside of the staircases leading up Dumbledore’s tower - nothing was left out.”

“VFX Artist Tania Richard started the creative process through a series of concept studies to present to Tim and David Yates. For the shapes and look of the shield we took inspiration from natural forms - deep sea bio-luminescent creatures and jelly fish. The micro details in the shield mimic the fine structures in insects’ wings and the huge branching cords of magic feeding the shield take their structures from acacia trees.” FX lead Alexander Seaman and his crew worked in Maya and Houdini creating a series of 3D renders and animated geometric structures. These were passed to compositing lead Christine Wong who used Nuke to craft the graceful looks in the finished shots.
The destruction became a creative challenge. “We created the molten fiery surface using a combination of Maya cloth and particle work and procedurally generated animated surfaces exported from Houdini and rendered through PRMAN. We referenced the Hindenburg disaster, not only for scale but because of the way its huge folds of cloth burn and dissipate before they hit the ground. The shield needed to react in a very similar way.”

Multiple Hogwarts
To handle the destruction of Hogwarts, they had to build multiple versions of their model. It was built once in a completely pristine state to use before the battle started, and once after it was destroyed. “These two versions use fairly traditional modelling techniques,” explained David. When we started building the school for our rigid body simulator Dynamite, used in Maya, we had to take a very different approach. If you want to destroy a wall, you can’t just build a texture a polygonal plane to look like a wall. Each brick has to be built and placed individually. Then you have to consider what's inside the wall between the exterior faces and what holds the bricks together. We had to build layers of coarse rubble that filled walls and created dust and debris as the mortar cracked and crumbled.
The detail in the Art Department’s architectural drawings of Hogwarts was invaluable for this work. They included precise roof construction plus the size, shape and profile of the tiles, where the beams supported it underneath, and types of joins – all of which increased the believability of the Dynamite simulations.

Burning Bridges
Double Negative also destroyed Hogwarts wooden bridge in this sequence. “David Yates wanted to blow the wooden bridge up like 'The Bridge on the River Kwai' which gave us a clear idea of the scale and epic style VFX work that we had to push for,” David said. The digital environment was an extension of their existing Hogwarts build but the wooden bridge asset had to be built from scratch four times to allow them  to achieve the wide scope of shots required.

The first model was built from set construction plans and was for shots where they had to use CG to extend the partial set build in live action plates. The second model was built from architecturally accurate scale technical drawings. “We built all the roof timbers, slates, floor joists, down to the individual types of join used to connect the structures to each other. This information was all used in Dynamite to create physically accurate FX simulations as the bridge was blown up. This 3D model was incredibly dense and took a long time to render.
“The third variation on the bridge build was an efficient low resolution render model that could be used in medium and wide shots. The fourth and last build showed the bridge in its post destruction state. These FX simulations looked beautiful as the bridge exploded but it was actually quite difficult to art direct the bridge so that it collapsed into a nice looking structure. We ended up hand placing wood, slate and debris to create the required shapes of twisted tooth-like wooden remains.”

Smoky Plumes
Other creature effects apart from the dragon that Double Negative created were a proportion of the Deatheaters FX work as the fly in or around the school. David said, “The challenge lay in trying to create smoky trails using our fluid solvers because all the fluid solvers try to re-create real world physics. Real smoke dissipates in air very quickly, and moving it through the air rapidly makes it dissipate even faster!”  How do you keep a column of smoke cohesive and intact, but still make it travel quickly through the air? A cloud tank effect wasn’t appropriate either.

David said, “We had to use a couple of different fluid effects in Maya. On top of these were cloth-like ribbons that twisted and spiralled through the smoke and particle effects that gave the tail of the smoke a grittier look. The integration of all of these components in 2D is what gives the trails their beautiful, roiling aesthetic.”

Infinite Staircase
Cinesite built an integral part of the interior of Hogwarts, the great marble staircase where a large portion of the action, battles in particular, takes place. It was built as a very large CG asset that could appear in many scenes through out the film at different times of day, and also be destroyed. CG Digital Supervisor Holger Voss said, “It was developed from a big set piece built at Pinewood Studios. This staircase has appeared in the earlier films, sometimes with rotating steps, but in this film it becomes much larger, extending into the distance within a very large environment.”

The set piece extended about 1 ½ stories, each way, and appeared dressed according to the sequence. Although large, it actually only represented a small section of the staircase. “We took a pretty standard approach, collected all textures, took a Lidar scan and total station survey and built a module that we could re-use and replicate. At the time we re-built it we still didn’t know the level of destruction, type of lighting or the size of the environment, so we tried to be prepared.”

The principle photography for Deathly Hallows Part 2 was done back to back with Part 1, and because the sequences had already been shot when the team started, the practical set had been destroyed and built up again. They had some blueprints showing that it needed to essentially extend into infinity. From these, Cinesite developed the looks and produced basic renders to submit for approval.

Perfect Lighting
The production was often shooting right off the set and into the rafters, because the set was so large that they couldn’t put a green screen above it when the action approached the top of the stairs although the actors might be running back up into the lighting rigs. The team would then have to rotoscope the actors and change lighting set ups. Other challenges arose in the cut. In one shot you might see a large area of the practical set behind an actor, just before cutting to a close up of the same actor with a green screen behind him. The lighting would have to match perfectly and the staircase build needed to accommodate close up viewing.
On set the team spent a full day measuring and photographing this set. Unfortunately, because the plates had already been shot, they couldn’t record scene lighting and instead re-lit their work to match the plate. In fact the lighting had to be adjusted and art directed anyway, for example to show natural light variations in certain areas wherever the stairs were seen receding into the distant background. Also, because the staircase was such a complex construction with so many arches and doorways, they usually chose to light it with global illumination in Renderman to avoid complications with too many light sources.

Flexible Compositors
“Our compositing team was small, only 13, relative to the number of shots. The lead compositor and lighter developed tools allowing them to share set ups for compositing the staircase,” said Andy. “Within each set up were sliders for haze levels, haze colours, defocus and so on. The settings could be saved as a key set up for a scene and loaded into the same node in another shot. They knew that they would have to be working intensely toward the end of production so they created these set ups early, tweaking them whenever they had time. Such measures helped improve our consistency.”

Cinesite’s artists have made the animated portraits in all the Potter films and it has become the specialty of one of their compositors, Karen Wand. The technique, updated this time to work in Nuke, involves a 2D approach, starting with motion analysis on the character and using brush strokes, canvas and varnish textures found on paintings with colour correction to make the colours more muted.

“We have some proprietary motion vector analysis tools, and using this motion analysis data we stick the textures to the characters as they move through the painting to make the canvas itself appear to move. A physical painting is supplied for the background into which the characters are composited. When Ariana walks back into the painting, they shrunk her down slightly,” said Andy.

One shot involves 15 portraits on a wall in the marble staircase environment.  Production shot various scenes of characters fleeing a set piece, and then choreographed the same characters to flee through the set piece of what would be the portrait next to it on the wall. Editorial carefully pieced the vignettes together as a line-up sheet for the team to match. 

Head Tracking
Cinesite had been responsible for replacing actor Ralph Fiennes’ nose area with Lord Voldemort’s snake-like snout in ‘Deathly Hallows Part 1’, and carried on with the task. They completed about 130 such shots, contrasting with only about 50 in the previous one. The same nose asset was used, but with a much larger team to track and light the shots. Andy said, “The changes made for this film were made to the process, not so much to the asset, shader or textures. We were trying to keep the soft tracks with the same artist wherever possible, to keep the facial expressions very similar across all shots within each scene.”
Lead compositor Martin Ciastko and his team joined forces with the lighting and tracking teams to decide how accurate the 3D had to be, shot by shot. As one of the film’s lead characters, Voldemort’s looks and performance were critical to the story. Sometimes the actor’s performance demanded very precise match move work to capture enough subtlety but at other times the model could be loaded directly in the composite in Nuke and the textures re-projected.

Skin Shader
To give the director a clear idea of Volemort’s presence in each scene early on, the team had to make a good start on the replacements across the film. “First, we tracked the camera to the background, then tracked the overall movement of Voldemort’s head according to that camera. Once this movement was approved, we could start on the facial match move,” said Holger. This also meant the basic lighting could be completed and a preliminary element given to the compositors, who cleaned up the tracking markers and live action nose, and produced an initial version of the shot.

The build of Voldemort’s head was based on a Cyberscan of the actor, and the geometry was used to generate a detailed displacement, showing pores and wrinkles. To help preserve the integrity of his performance and achieve an accurate match move, the team built a rig with three levels of animation controls, accessing the full set of tracking markers attached to his head with enough flexibility to match the action in each sequence.

They had developed a skin shader tool to generate the snake nose, generating textures by using specialised photography and lenses, and extracted detailed pore maps. Multilevel subsurface scattering improved the photorealism. Some shot-specific dirt maps and dirt textures were also added where Voldemort picks up dust and dirt during the fights.
Cinesite’s doe patronus this time was created with the same technique as before - rendered using an in-house fur pipeline with interior swirling effects through a custom volume shader using Renderman - but its behaviour was to be quite different. Instead of slowly moving through a dark forest scene, referring to Harry’s mother, it appeared as Snape’s patronus and leapt like a rabbit as it did in ‘The Order of the Phoenix’, and they used both as their reference. New swirling ribbon effects were added in Houdini as a particle simulation converted to geometry.

Green Gunfight
When the team was asked to develop some entirely new wand effects, they put their matte painters and 2D artists to work on concepts. They created some still frames and then multiple frames dissolved together to get a feel for a new effect. They applied several wand effects in a scene the included a chain wrapping around a character, the deflected bolt that blows up part of the staircase, and Ron’s spell fired at an opponent with a flowing ink affect. All of these were a chance to be creative.

For Harry’s and Voldemort’s crtical battle on the staircase, Cinesite created the first portion of an effect that MPC finished with a ribbon effect. This was handled like a gunfight with green bolts that Harry deflects, all done in 2D. "We designed and executed the first part of their wand fight using Nuke, trying to use the green of the Avada Kedavra spell and incorporating some explosive shield effects as Harry tries to defend himself. Harry deflects the wand duel effect into the staircase ceiling.  This allowed for one of our biggest effects in the film, where our staircase completely crashes down onto Voldemort.

Wrapped in Chains
"We used some practical flame and water elements as a basis and heavily warped and treated them in Nuke, mapping them onto spheres which formed the shield around Harry and making them feel explosive with bright bursts of light which augmented practical interactive lighting on set," Andy said. “In contrast, the wand fight between Sape and McGonegal, briefed as a fiery effect, a whip or a flame thrower, was 3D. Each of these has to be planned separately, usually starting off in 2D for a quick turnaround for look development cycles for the director, and we may decide to get into 3D later for effects such as fire.
For this fight in Hogwart’s Great Hall, simple animation rigs were used to drive the particle effect, instead of fluids alone. They chose particles with curl noise fields to achieve a fluid look with less computation time and also re-lit the environment with glow and flashes from Snape’s shield to integrate it into the Hall.

The actor, a Deatheater, who was wrapped in chains was shot reacting to something, as if he had been hit, giving them an opportunity to try various concepts – a boulder, a cloud of dust. His movements required a match move to help animate the chains coming from Harry’s wand. Since there was a lot of cloth movement in the character, they also assisted the match move in 2D for a more fluid approach to stick the chains to the character’s body.  “’Harry Potter’ has such a strong visual sense. That’s what makes it satisfying to be asked to use this as a starting point to come up with something new.

Hogwarts lead environment artist Pietro Ponti devised a rig made up of three Canon 1DS MkIV DSLRs mounted side by side and designed to shoot out of a helicopter. The three cameras were remote triggered with shutters synced. Pietro studied Google Earth and plotted out semi-circular flight paths around all of the mountains the environment required.
The locations were varied and far apart. Loch Schiel, Glen Nevis and the Three Sisters all needed to be shot under similar lighting conditions. Google Earth data also allowed him to determine the lighting and the sun’s location at any time of day and plot the optimum flight paths for the helicopter. He could then hand over the GPS data to the pilot to feed into the aircraft’s computer.

The shoot generated over half a terabyte of photography which Double Negative’s team used inside proprietary geometry reclamation tools to re-build the terrain, using the photography to create textures as well. The result was the mountain mosaic seen in the film around the outside of the school. Thus the Hogwarts location is both invented and real, made up of remodelled, real places, moving the mountains around and blending them together to create the ideal landscape.

The team was able to build the entire exterior structure and terrain, from wide shots to the battle scenes in the main courtyard, in the Great Hall looking out and from the battlements, even the inner courtyard where Harry meets the Grey Lady. In short, whenever the camera is outside the structure.

They had investigated various rigid body solvers to destroy the staircase with, and chose PullDownIt from Thinkinetic in Spain, the group that wrote the first solver for RealFlow. “Cinesite bought a site license for it early on and we extended its functionality to do some set dressing by painting onto the steps where you want to apply certain types of debris or rubble. The solver would drop this material onto the steps accordingly, so that nothing was intersecting or floating, in nice believable piles using physical parameters of gravity, mass and centre of mass,” said Holger.

This process was carried out per scene, depending on plates and the story. 2D Digital Supervisor Andy Robinson said, “When Harry Potter and Voldemort are fighting, Harry deflects a big wand effect up into the staircase and rubble comes crashing down. We never had a definitive shot for it, just a couple of wide shots, plus a medium shot of Harry cowering under a shower of debris. So we came up with some dramatic camera angles and interesting lighting to use in production, backlit and dark, as look development for the destruction in the sequence.
“The shot didn’t get official approval but was cut into the film early enough to build on. The freedom to do this comes from having created such a large, flexible CG asset. We could work with whatever the plates presented.” Holger also built some generic 3D camera angles to take out on second unit shoots of extra characters – such as people fighting, wizards, kids and death eaters - to composite into the backgrounds of the staircase shots.

MORE HARRY POTTER ONLINE:
Appearing on our website will also be features on the top visual effects work that Baseblack, Framestore and IE Effects completed on the final Harry Potter film, plus a retrospective interview with Cinesite’s managing director Antony Hunt, an Executive Producer on all eight films in the series.
See

I.E. Effects Gets Immersed in the World of Harry Potter

Cinesite’s Antony Hunt Looks Back at Harry Potter – and Beyond
Baseblack Brings Ghostly Magic to Harry Potter

Framestore Prepares for the New Harry Potter Generation