Sony Pictures’ Colorworks Builds a 4K Workflow for Spider-Man 2

Sony Pictures Imageworks completed about 1,000 visual effects shots, Sony Pictures Post Production provided sound in its first project in Dolby Atmos, and Colorworks performed film scanning, conforming and colour grading - all in 4K - for 'The Amazing Spider-Man 2'. A custom workflow was designed in order to give the teams access to data associated with the production, which in turn supported working together, closely and simultaneously. The goal was improved efficiency and a better, more spectacular cinematic experience for the audience.

Scanning to 4K

To get post production underway, editorial was first sent DNXHD proxies from telecine dailies. Meanwhile at Colorworks, film dailies amounting to more than 1.5 million feet of 35mm film were scanned on DFT Scanity scanners to 4K digital format prior to completion of editorial. This resulted in approximately 1.7 petabytes of original 4K 16-bit film scans plus another 55 TB of digital camera capture files were stored in Sony Pictures Entertainment’s Production Backbone, the studio’s proprietary shared storage system, based on combination of SAN, tape and an internal scalable-storage.

The footage later returned to Colorworks with EDLs and HD reference proxies from editorial for conforming, final colour grading in 4K and mastering in 2D and stereoscopic 3D versions using a combination of Autodesk Flame and FilmLight Baselight. All of the VFX shots were ingested into the Production Backbone as well, so the conform could pull the original 4K scans and digital shots for non-VFX scenes, as well as the final, delivered VFX for effects shots. All files at this point were 16-bit DPX.

Collaboration between the teams was facilitated by the Backbone. Original production elements, along with associated metadata—cumulatively amounting to more than 2.4 PB—were stored on the backbone’s shared-storage environment where it was accessible to sound and picture editors, visual effects artists and others, as needed and in appropriate file formats. The backbone also simplified delivery of elements to external visual effects vendors, who were able to access the backbone through Aspera for secure, high-speed connections, which typically take a few minutes to a couple of hours, at most.

True Digital Workflow

“Scanning all of the original film elements at 4K allowed us to work at the highest quality and implement a true digital workflow for the production,” said Bill Baggelaar, Senior Vice President of Technology for Colorworks and Post Production. “That saved significant time. The Backbone not only provided access to data, it also tracked progress. If an editor or a visual effects artist made a change, it immediately became available to all the other members of the team. This is now becoming more important for quick turn around on a dynamic film like ‘The Amazing Spider-Man 2’.

In this case, the original production elements referred to original scans of film, digital camera sources, VFX plates and the VFX shots created from the film and/or digital camera sources, as well as 3D conversion shots and plates, EDLs and reference QuickTimes. various kinds of production and VFX metadata are associated with these elements. Most of the collection is automated from the Backbone perspective. For original footage, the collected metadata includes everything in the ALE for each element - the complete dailies metadata - as well as all media header metadata, where applicable. For VFX and finals deliverables, the metadata is much sparser but does include CDL values to match the original dailies and mattes, when applicable. 

Shot Tracking

“The production and VFX teams generally had their own shot tracking systems, and Colorworks would interface with the internal Backbone shot tracking system to ensure a seamless flow. As shots landed on the Backbone, they were automatically registered into its database. The tracking system notifies users when items are posted to the PBB, via automated emails. Colorworks is using Shotgun to track and automate many tasks relating to DI and VFX production, but many different systems are tied together based on the particular show we’re working on – each one is a little different.”

Because visual effects teams need to work up to the last minute in order to provide the best quality output that they can, colour correction had to start long before all final VFX deliveries were completed. The VFX download, verification and registration system was carefully refined to work as quickly as possible, so that new VFX shots were ready to pull into the conform straight after they were made available. This was an automated process that ran continuously and allowed the VFX to move into the colour bay very rapidly.

Colour Management Pipeline

As colour is sometimes an issue for the success of effects, Colorworks worked closely with the production and VFX teams on the colour management pipeline to ensure a smooth transition. Bill said, “We also worked along the way to smooth out any bumps that inevitably occur during the production process. Communication is at the heart of a smooth running production. For example, Colorworks created and provided the LUTs for all VFX vendors, managed all colour space conversions, and also did some pre-grading on plates.

“A good colour pipeline is also critical to giving the filmmakers the confidence that what they see in VFX reviews is what they will be seeing in the final colour grading sessions. Mattes can also be useful when colour correcting CG characters in their real-life environments, but in fact the effects work was so good they were hardly used.”

A FilmLight Baselight 8 system was used for all of the final 4K colour grading. Bill explained specific advantages realised by working on the grade in 4K, and some of the challenges they encountered. “Since everything was either sourced from 4K 16-bit scans or 4K 16-bit digital camera sources, anything that wasn't a visual effect was natively at 4K in the Baselight. The 4K supersampling of the 2K VFX plates created sharper, more detailed images for the VFX teams to work from,” he said.

4K Finishing Environment

“It is critical for filmmakers to be able to see their final result in the highest quality that will possibly be viewed by an audience so that they can see the film they are actually making. This high quality 4K theatrical finishing environment has the added benefit of flowing right into the 4K/UHD home video mastering. In the end, we are able to provide 4K deliverables for the various primary and downstream markets much more efficiently by finishing in a full 4K environment.”

Bill noted that the challenges with 4K workflows in general typically relate to data storage because extra capacity and speed are needed for efficient handling. Most of the modern 4K digital camera systems use some type of lightweight compression to achieve more manageable file sizes without compromising quality. “But this show was quite different in that the entire film material was scanned at 4K 16-bit and archived uncompressed to the Production Backbone tape library,” he said.

“Consequently, this was the biggest technical challenge on this show as well, the sheer volume of the 1.7PB of data. Dealing with this efficiently relied heavily on the tightly coupled storage architecture that was developed at Colorworks for the Backbone. In terms of processing the data, the Colorworks and Backbone infrastructures are scaled to deal with 4K on a regular basis and therefore that was never a problem.”

3D Conversion

The 3D conversion was occurring in parallel with the 2D VFX. No pre-grading was done before the 3D. The 3D conversion was part of the same color pipeline and so they were able to use the same information to ensure consistency. 2D grading actually finished first, and the 2D grade was used as the base for the 3D grade.

The same Baselight 8 systems were used for mastering as for the theatrical DI - all of the information and 4K material from the theatrical grading sessions were available for the video mastering. Bill said, “To complete the stereo master, the production's 3D team provided baseline convergence from their Avid. We worked with their team to create a conversion that turned their Avid convergence values into numbers that worked in the Baselight. That information travelled with the EDL and provided the stereographer, Ed Marsh, with a good starting point.”

Bill considered the time that was saved on this project through their collaborative workflow. “Basically the time savings comes from turning a traditional film-based VFX and DI project into a completely digital project,” he said. “Since scans were available before editorial needed to pull the VFX, there were no time delays getting shots into production that would typically occur – that is, order film reels from vault, clean film, load and process keycode lists for scanning, load film reels, scan shots, process scans, QC scans, load scans to drive, deliver to VFX vendor, and so on.

“Since all takes were transferred in dailies and the same timecode was applied to all of the film that was scanned, once it had been QC'd and ingested into the Backbone, the process to pull the VFX plates and pulls for conform were automated tasks driven by submitting an EDL. The automation vetted the pull information against the Production Backbone database and performed the actual restore of the requested frames from tape to a staging area for the particular VFX vendor the shot was intended to go to. An automatically generated notification was then sent to the appropriate production and VFX entities so that they could start to perform their work. This all had to be set up by the technical team that manages our Backbone, but once in place, the  process is repeatable and highly reliable.”  www.sonypictures.com