Stephen James and Melaina Mace at DNEG talk about building a terrifying future vision of Seattle for The Last of Us S2 - some of their most complex environment and destruction work yet.
DNEG was one of the lead VFX vendors on HBO series The Last of Us – Season 2. Ellie and Dina have travelled to Seattle, where DNEG’s teams recreated a harsh post-apocalyptic version of the cityscape. Digital Media World spoke to VFX Supervisor Stephen James and DFX Supervisor Melaina Mace about rebuilding the city’s skyline, coastline and familiar sites.
Producing more than 350 shots across 36 sequences, this project represents some of the most complex environment work DNEG has accomplished so far.
Their work ranged from ocean storms to decaying street scenes, where vegetation has been slowly making its way through and across Seattle in a world that has become even more dangerous and unpredictable than it was at the end of the previous series. The team custom-built their assets for large-scale destruction and designed FX that pushed their water and destruction pipelines in new directions.
DNEG joined the production in spring 2023, after the winter shoot. Asset building and production both continued through the summer, before shot turnover in the autumn. “When DNEG got started, HBO’s VFX Supervisor Alex Wang had been working hard on preparation work for this second season of the show, and early on we put effort into supporting production by going out on set wherever Alex couldn’t be present,” Stephen said.
Stephen had also been the company’s VFX Supervisor for the first series. “DNEG planned the scouting and reference gathering on location,” he said. “For Season 1, that meant going over to Boston, but this time it was just down the road from us here in Vancouver.
Three Anchors
The environments are a major driver of the show’s story, showing the passage of time, evidence of danger and destruction, and directly influencing the moods, actions and decisions of the characters. “The primary feature of the show is the characters’ continuous journey, constantly leading them to face challenges from unfamiliar surroundings. So for us, the job for the new series wasn’t just about re-purposing existing work.
“Instead, we are constantly moving on to the next big thing, calling for a large amount of custom VFX work. We had three ‘anchors’ to base our work on. The shoot, much of which took place in and around Vancouver, was the foundation, and we combined that with the extensive photo reference captured on our trips to Seattle – enough to be able to rebuild the city. But even before those two, a third anchor was the ‘Last of Us’ game itself.”
The game concepts, and the production’s concept art based on that, contained abundant material to pull from. For more specific reference, the Showrunner Craig Mazin and VFX Supervisor Alex Wang were fortunately consistent and aligned regarding what they wanted to see. They fleshed out many of the concepts for the teams, so by the time DNEG went down to Seattle, they already knew the camera angles and framing to aim for, the times of day and direction for lighting.
“We informed our work as far as possible with what the real shooting conditions would be. If the characters were going to run out on a rooftop, we went out on various Seattle rooftops and captured images looking down into the desired areas, in the right conditions.”
Data Capture
Evidence of just how complete and dense the production wanted the environment to be can be seen in DNEG’s approach to data capture. They spent six days in Seattle – a day of scouting followed by more days of data capture. “We also worked with a local production team, a drone operator and a LiDAR technician from Clear Angle Studios. We travelled across the city, recording a range of places and buildings with a distinctive look to help viewers locate the story, and establishing views that could be used directly,” said Stephen.
“When we went out onto the water for the Aquarium sequence, we were there really early before sunrise to get the overcast look we wanted. LiDAR helped capture the city streets and the famous spots like the Seattle Great Wheel, the Aquarium and the base of the Space Needle. We also took many all-around shots from the tops of buildings and all the way around locations to use as textures as well as reference.”
Drones were for anywhere they couldn’t reach from the ground. For instance, they allowed them to take a camera out over the water for a wider view of the waterfront, and safely record shots for photogrammetry around those buildings. Drones also made it possible to shoot video by flying very low along the water and ocean to get profile shots of the city and shoot right through the Wheel.
Stephen said, “Although our result was going to be full CG, those shots made beautiful reference for understanding the composition we wanted. Nevertheless, throughout the operation, our drone operator cursed the seagulls attacking his drone.”
Blending Reality with Fiction
Melaina talked about ingesting all of this data and organising it to use in sequences. “For the Aquarium, we combined ground-based LiDAR scans of the real-world location with drone photogrammetry for a full 3D representation of the building and surrounding environment,” she said. “We had additional LiDAR of the set build at the back of the Aquarium where Ellie climbs up. Our job was to design an asset to match the set design and develop the look of the Aquarium for the show.
“So, we combined all of these elements in 3D, lined them up and designed our Aquarium asset to be a combination of the real-world location and the set production design. Having access to the Seattle Aquarium was incredibly helpful, and we were able to use our photography from the location shoots as texture reference.”
Knowing how the real assets were built was a key advantage when recreating well-known sites. For instance, the theatre in the story is based on Seattle’s Paramount Theatre, but the shoot took place on a back lot in Surrey in British Columbia. “The production design of the theatre set was a mirror image of the real building. We LiDAR scanned the real Paramount Theatre, mirrored that in 3D, aligned it to the set LiDAR and made adjustments where necessary for our set extension,” Melaina said.
“Many of the sets were partially built up to two or three stories high, where it became our job to extend the building above that. In Season 1, we worked with the production-side team, who would create their own 3D mock-up of how the rest of the buildings should look. For Season 2, we developed some of the designs ourselves, based on the set design and real-world references. We were present on set for most of our sequences, which gave us the opportunity to shoot additional reference material.
Nature Takes Over
An important part of the environment work involved visualising how nature was reclaiming the streets, sending out CG moss, ivy and overgrowth to spread across the city. To reflect this story point the production often shot on location on sets that included expert set dressing, which DNEG’s work enhanced to look more natural and connected to the world.
There were various considerations. Melaina said, “For a street scene captured on location, production would dress the centre of the street, where the actors are, but may not have been allowed to interfere with the sidewalks or buildings. The challenge was to make all the ground vegetation match local Seattle vegetation as well as the set dressing. Depending on the shoot day, we would often have to simulate wind to match what was captured on set.”
These details all help integrate the CG, and meanwhile the compositors had to plan how the vegetation should spill over the foreground and mix into the background. Every view in every sequence was built off a real-world location, either a built set or a reference. Stephen said, “Our inspiration came from what is in-camera and what works for the whole sequence in the story.
Orchestrated Destruction
“We would embellish and extend whatever was real in the foregrounds to look as if it had been there for 20 years. Often, beyond a city block, we created full CG takeovers – perhaps destroying the buildings, or completely overgrowing them, but always aiming to build off of what is in the plate.
“Our approach was to re-build an object or building as it actually is now, then add a layer of weathering and destruction on top of that to age it,” said Stephen. “We had to re-think the destruction work for Series 2 because Seattle is on the coast. We used more landslide and collapsing effects – tunnels collapsing, water damage causing structures to slide into the ocean, and the sea encroaching onto the buildings. This gave us a new challenge beyond the bombing of the earlier episodes.”
Another factor was that, naturally, showrunners are always looking for something bespoke that hasn’t been seen before. In this case it meant adding lots of complexity on top of a simple foundation. The team did have pre-destroyed bldgs from their library, but the main push was for custom pieces of destruction that could be worked into the composition.
“Directionality was an important idea,” Stephen noted. “In other words, wherever a collapse occurs, we made certain sides of buildings slide into the water first, not just a general collapse. From there, you can scatter debris accordingly, show boats running aground, collapsed roads and cars getting washed into the ocean. All detail would be contributing to that story.
“Because we wanted those scenes to look interesting and engage viewers, our environment team thought through questions like ‘what happened here’? Why? What had occurred over those intervening years? We wanted it to have logic and make sense.
Pipeline Stress Test
From the first season and into the second, DNEG’s pipeline underwent a major change. Melaina said, “Although some workflows for vegetation and destruction kept the same approach, others needed to be completely rebuilt because we shifted to a fully Houdini, Solaris and Rendermanand pipeline. In fact, this is more in line with how our FX team has been working for some time, with a Houdini-based pipeline. Having everything now in the same 3D application has allowed us to be more flexible.
“We’ve had to develop a few new tools for the vegetation and wind simulations as well. But our FX pipeline hasn't changed as much, and making this deliberate shift meant we could build upon that.”
Stephen noted that ‘The Last of Us’ has been a good stress test for the newer aspects. “We knew what we had to reach for, in order to match to Season 1 and go beyond it,” he said. “It was helpful to know the look and level of complexity to aim for – and we couldn’t have had a more dense environment in terms of render management than this show.
“Compared to last time, we had to optimise the assets further to be able to increase the density of buildings across shots, and the complexity within the interiors. At the same time, given our tight schedule, our departments had to work in tandem. The pipeline is set up so that if Lighting has set up a new nighttime look, they can publish it. Meanwhile, if FX are working on a scene they can pick up that look and test how it will work with their FX. All of us can work simultaneously on scenes as they progress and then – hopefully – everyone’s effort will come together at the end.”
Water Management
Of course, some tasks do need to come first. They need consistent layouts in a base environment so that FX have something to simulate against. “Early on, as well as that initial environment work, the team could put a lot of work into postvis of all the ocean sequence work. The oceans were so complex, we knew we couldn’t wait until the environments were complete to get that started,” Stephen said.
This ocean work fell mainly in the series’ finale, DNEG’s biggest sequence. Managing the water was the main challenge for everyone, in fact, both in production and in post. As Melaina commented, “Having the ocean simulations plus the rain sims on top of that, made it more complex.”
The production crew shot the water scenes in several different ways, to get the best possible result on set and in-camera. “For example, we have a shot of Ellie at the shoreline, climbing up and out of the water. We also have shots of her on a boat in a water tank, in a dry dock and on a dry boat rig, accommodating safe manoeuvres for the crew when they needed to capture her up close to the boat,” said Stephen.
“Meanwhile we have footage of Ellie drenched in rain, the shoreline shoot gave us the look of the water, and we had the set lighting as well to use for the storm lightning FX. All these inputs gave us plenty of reference, allowing us to tie our work into the real photography.”
Consistent Storm
Consistency was important during the storms, such as their direction and wave height. The water team had a base ocean that could be used in every shot. The challenge came when working ahead, to match it to a plate that would be shot in a water tank, or on a dry rig on a boat. DNEG’s water had to connect to any motion there. “For that, we relied on strategic timing,” Stephen said. “Alex set up a simulation to show us what was intended to happen on set – like the rise and fall of the boat – that we could match to.”
Despite these complex problems to work through, Stephen really enjoyed working on this sequence. “I hadn’t supervised water much before, but having an expert FX team with recent experience in water work, with an established workflow for oceans that we could push a little further for the storms, was great. We improved looks for the water itself, the spray, rain, rain gusts, rain interacting with water, drips, splashes and ripples. Our team made all the difference.” www.dneg.com
