The Great Gatsby – Deconstructing New York

Animal Logic was among seven different vendors working on ‘The Great Gatsby’. The team was awarded 590 shots, completed over an 18 month period. The shots involved extensive and varied work creating some of the film’s major New York environments – from Times Square, chaotic party sequences and the Valley of the Ashes, out to Gatsby’s and Tom Buchanan’s expansive mansions on Long Island and the memorable highline view over Manhattan.

Some shots required partial extensions while others needed full CG builds, and tasks included environment backgrounds, crowds, animations, digital doubles and matte painting extensions. The diversity of the environmental work was a key challenge for the team, managing the number of separate environments as well as how different the look and concept for each one had to be.

VFX producer Ingrid Johnston said, “We have worked on productions with lots of environments before, but they had all kept a fairly continuous look and feel from beginning to end of the project. For ‘Gatsby’ however, we go from highly urban settings to landscapes and aerial views.” Even within the single setting of Gatsby’s mansion were exteriors and interiors, wild parties incorporating crowds of guests and quieter scenes when only Jay Gatsby and Daisy were present.

Locating the Story

It became Animal Logic’s job to stitch those settings together visually to create the world for this story, which moves between the extremes existing in and around New York in the mid 1920s, all shot in Australia. The majority of sequences were shot on studio sets and stages although some locations in and around Sydney were also used as sets. The largest of these were the ones used for the Valley of the Ashes, an ash-covered no-mans-land lying between the wealthier New York society of Long Island and the working class neighbourhoods, shot at Balmain and a power station site in Sydney. Smaller portions of Centennial Park and Mt Wilson were used for Nick’s cottage, and parts of Gatsby’s mansion were based on St Patricks Seminary at Manly.

These venues would only serve as partial sets in the film. The seminary was used just for the mansion’s front and ground floor, for example. Above and beyond small areas used as live action, many sets included blue and green screen and were built out digitally in post. Therefore, Animal Logic’s VFX supervisor Andy Brown explained, it was important to find out as much as possible about the sets and locations well in advance of the shoot, gather extensive data and decide what shoot supervision might be needed.

“We captured stills of every inch of each set to prepare for the various replacements and extensions, plus HDR images for lighting,” he said. “Chris Godfrey had data wranglers to record camera, rig and lens information and carry out set surveys with a Total station. LIDAR scans had to be supervised to make sure they were carried out adequately, and more stills had to be captured for the digital characters we created to compose crowds. Meanwhile the client needed a constant stream of concept frames, pieces of pre vis and tech vis to figure out how to shoot and set up the shots.

“Because so much of the live action was shot against green screen, the tracking marker set up, clean plate and element shoots were critical for our team - not just getting them done but monitoring how they were conducted. Tech-viz was important also because it ensured that we had the correct camera angles and coverage. A typical shot would involve two live action components plus a CG environment.”

Historical Reference

The need to make Australia stand in for New York, made it hard to rely completely on the material shot on locations. For example, foliage and trees were tricky because most sites apart from Mt Wilson were full of gum trees. Animal Logic’s two CG supervisors travelled to New York for two weeks in September 2011 during the main unit shoot to capture reference, stills of the key period buildings they needed to include, plus panoramas of Manhattan. They took a GigaPan remote head with them, which allows pan and tile capture of environments. A typical 180° panorama can involve over 100 images, giving a high resolution result.

The team also received numerous specific reference images and concept frames from director Baz Luhrmann and production designer Catherine Martin, including material from the Historical Society in New York and archival sources. Animal Logic’s own concept team contributed concept work, helping with the design of sets at Gatsby’s mansion, the Long Island terrain and more, involving lots of dialogue with the production.

“Having so much reference combined with concepts and style frames from Art Department and artists like VFX art director Daniel Cox was a little overwhelming at first, but it did help narrow down and refine the path we ultimately took. We also had our own concept artists working with us – for example, our team was working on the lower angles while Dan Cox from art department focussed mainly on the high angles the director wanted,” said Ingrid.

“Times Square in particular really had to be built from scratch,” Andy said. “It was nearly all shot on green screen with a minimal set, small pavements, a few cabs and extras, with the small awning you see with the light bulbs underneath. In that case, we were referencing period photos of Times Square, rebuilt all the billboards with signage from production, and had a lot of crowd work to do. We also needed to ensure that the Astoria Hotel exterior was accurate for the period.”

In a Crowd

At nearly every shoot location they shot textures for crowds because such a variety were needed – crowds for the parties, for Times Square, the highline vista and the Valley of the Ashes. For some shots the artists could use sprites, shooting extras against green screen to composite within live action groups, but due to camera moves and the stereo photography, CG crowds were created for the majority of them - crowds generated in Massive to populate some backgrounds, or keyframed characters closer to camera.

Texture shoots were done by placing extras on a turntable, while the hero cast were cyber-scanned – that is, photogrammetry-type photography was used to model the extras but digital doubles were made for the main characters, Gatsby, Tom and Daisy Buchanan, Jordan and Nick. Cyberscans of the actors’ heads and bodies were stitched together and textures applied.

Historically accurate cars, of course, were necessary to complete the period scenes, often appearing in such numbers that they were handled like crowds to fill out the sets. The real vehicles available on set were photographed and modelled for use in either procedural or keyframed crowds. The hero cars, Gatsby’s beautiful yellow Duesenberg and Tom Buchanan’s blue coupe, were LIDAR scanned and photographed in order to build them up to a very high resolution.

Lighting & Shading Pipeline

Lighting lead Matthew Estela pointed out that while some shots placed live action directly next to CG, the artists were mostly creating either entirely CG shots with a small live action insert, or very large, live action shots with CG extending the backgrounds such as the party shots. “The production put less emphasis on matching the live action elements and more on making it bigger, more dramatic, to the point that we ended up replacing live action elements as they started to look dull by comparison. For example, many of the Deusenberg shots ended up replaced by our CG version,” he said.

“On top and behind all of this detail, the matte painters sometimes worked from scratch to generate skies and distant land behind our CG, and at other times took our renders, painted directly on them, then had their work re-projected onto our layers in Nuke.”

The production was keen to see backgrounds filled with vibrant detail. The team at Animal Logic was using a new, physically-based lighting, surfacing and shading pipeline, PHX, that very well for this type of environment, putting out very life-like, photoreal renders much faster than they previously could. All the software was written in-house to work within PRMan. However, because this was among the first shows they used it on, they had quite a bit of development to do and further shaders to build.

Matthew Estela said, “One PHX material in particular was versatile enough to handle buildings, cars, vegetation, oceans, digital doubles, wood, metal and so on. The standardisation allowed our surfacing artists to get through large numbers of assets very rapidly.

Hologram Shader

“A custom shader we developed was a hologram shader for buildings. Since we didn't have time to model interiors for every building in Manhattan, surfacing artist Artur Vill took an idea from video games, starting from our flat windows and shop fronts and letting us dial in parameters for room depth, wallpaper variation, interior lights, curtains - all from a shader that worked correctly in stereo. Viewers barely notice it in the final shots, but would definitely notice if it wasn't there, and just accept that all the windows have a proper room behind them.”

On the tools and simulation front, the lighters re-used many parts of the existing Animal Logic toolkit, developing python scripts into handy tools to distribute lights across crowd cars, for example, or bring multiple lighting layers and their associated AOVs into Nuke.

Matthew said, “Surfacing moved to a centralised library of materials rather than creating materials for every asset from scratch. For example, they could construct eight or nine brick presets and apply those across our hundreds of buildings. If a look change was required, they could make updates to just those eight materials that would automatically propagate across all assets. The more specialised shader attributes wouldn't always transfer into lighters’ scenes, but the lighting TDs had time to get their processes running smoothly by the end.”

HDR Art Direction

Meanwhile, building the very large, varied environments made it a challenge to scale up the pipeline adequately to handle the level of detail in the geometry. “Our first lighting tests had involved a limited number of elements, which now needed to jump to half a city,” said Ingrid. “Fortunately, over the course of this project we had a chance to optimise the processes to be able to cope.”

Using HDRI to light CG objects might match what was on set, but would only be a base starting point and sometimes looked processed and artificial - large chunks of green or blue screen might be visible, practical rigging or pure blue sky. “From here, lighters would heavily art direct the HDRI itself based on shot requirements,” said Matthew, “painting out and replacing blue screen and rigging with more dramatic skies, overlaying gradients to give directionality to the light, or boosting saturation on one side, reducing it on the other. When combined with a few direct lights, the HDRI would give amazing results in the end. Overdriving the light and aggressively shaping and saturating how the environment light behaved, contributed to the hyper-real feel that Baz was after, I think.

Native Stereo

The fact that the film was shot natively in stereo 3D was another new factor for the team. Andy said, “‘Legend of the Guardians’ had been produced in stereo and consequently we already had a stereoscopic infrastructure and pipeline in place.”

“But in this case it was the tracking that was the challenge. We had to overcome the initial hurdle of tracking all shots and understanding that to do this successfully meant taking LIDAR scans or site surveys of every set. We had to be able to track the live action perfectly, and established a large tracking team to handle every shot using 3DEqualizer. Often we had to either refer to other images or create low res tracking geometry based on scans of the shots, which the tracker could use to ‘pin’ the left and right camera streams together.

In F Scott Fitzgerald’s original story, the ‘Valley of the Ashes’ is based on a real place in Queens formerly called Flushing Meadows, variously used over time as a coal ash dump – the purpose Fitzgeral would have known – and later on as the site of two World’s Fairs. In the movie it appears as an industrial wasteland and is the setting of quite a bit more activity than the team’s historical reference showed. The CG assets and backgrounds they created helped add enough interest to key parts of the story occurring there that the production wanted to see.

Flexible Sets

The prominent billboard of Dr Eckleburg’s eyes, an important feature at this location, required a practical stand-in built for certain pickup shots – redressed in post for the right look, although it was not on set for principle photography, and a carefully designed CG version was also built for several composites.

Actually, the designs of certain other props and sets sometimes changed at various points between the concept stage, the shoot and Animal Logic’s final versions of shots. Examples were the pipe organ above the ballroom and Gatsby’s winter garden, which became much more complex over the course of the team’s build and early shot work. In the plates, the changes then had to be blended into a set originally designed for a different look or purpose.

Where such shots had been repeated as pick-ups and re-shot on blue screen, the team would sometimes have to recreate the set from the earlier take, either as a full CG build or a projection of their own reference stills they had captured of the set. Certainly, Andy’s and the on set team’s exhaustive photography from pre-production came in very handy.

High and Wide

Several shots that Animal Logic created in CG are distinctive enough to form signatures for the film. Among these are the very wide high shots with massive camera moves. “In one shot, the camera moves across the Queensboro Bridge and pulls out into an aerial view, and another pulls out over the city from the party at Myrtle’s flat,” Ingrid said. “These were such custom views and such wide vistas with so much information that they needed combinations of very large matte paintings, traditional CG and further 2.5D matte paintings, because they also needed to accommodate the stereo effects and moving camera.”

The financial district was one of the team’s main locations and had to be built out to allow them to create the camera plummeting straight down to meet Nick’s face smiling up from the pavement where he stands. The beautiful highline view that opens the film as the camera comes through cloud to reveal the vista of Manhattan is perhaps the most memorable of these wide high views. It was based on plates captured on a dedicated helicopter shoot over Manhattan, and had first been concepted carefully and followed by tech-vis before taking off. “On returning the images to us, we stitched these plates together to create the base, removed and replaced the modern buildings, added the distinctive finger wharves, plus CG water, boats, smoke and steam to complete the views,” said Andy. 

To read more about post production and VFX for 'The Great Gatsby' see Digital Media World's feature here.