Pixomondo Links Global Render Farm with Deadline

Since it was founded in 2001, international design and VFX studioPixomondohas continuously evolved its infrastructure and capabilities. The company now operates from six facilities on three continents with teams working on visual effects for film, TV and commercials, interactive digital content, VR, live events, theme park entertainment and original IP.

By sharing render resources across locations, Pixomondo can accomplish its activities more efficiently. Its globally dispersed farm runs around the clock using a proprietary process built on top ofThinkbox Software's Deadline, on which all of Pixomondo’s facilities standardized in 2010.

Patrick Wolf, Pixomondo’s Global Head of Pipeline said, “Part of what influenced us to integrate Deadline so deeply is its open architecture and use of thePythonprogramming language. With the source code for application integration available, we can constantly tweak the software to our own specifications - it’s very extensible. Also, the Thinkbox crew has always been willing to work with us as we expanded.”

Deadline high-volume compute managementsoftware is used to render, manage and process files locally and across the cloud. Pixomondo uses Deadline mainly for remote rendering between branches. Patrick and his team customized Deadline’s job submitters so that users can select the branch or branches to render in. They also extended the Deadline GUI to allow render wranglers to split local jobs to render on one or more remote farms simultaneously, which is especially helpful on tight deadlines.

When a remote job is sent to the farm, event scripts in Deadline automatically determine the job dependencies. Necessary file transfers are queued and then a replica job is created on the remote farm recreating the settings of the user that submitted the job locally as well as remapping job priorities, pools and group configurations. As soon as a frame is rendered remotely, aDeadline after-render eventinforms Pixomondo’sCelery-based global queuing system.

Celeryis an asynchronous, or background, task queue/job queue software based on distributed message passing, usually in real time. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, or theEventletorgeventnetworking libraries. Tasks can execute in the background, or wait until the system is ready for them.

Once informed about the rendered frame, Celery then triggers the transfer of the frame and also marks the task complete on the local side so that the remote render is transparent and visible to the artists. After the remote render is complete and all frames were verifiably transferred back, a cleanup job removes the frames on the remote side to save storage space. Celery is then used to mark the job as completed locally, which allows dependent follow up jobs to run like a proxy image generator. Pixomondo’s production management system,Autodesk Shotgun, is also notified when the render finishes. Currently, the studio solely uses physical computing resources but Patrick is now investigating the integration of cloud rendering to the workflow.

“What is interesting about our remote rendering setup is that every step is a separate dependent Deadline job with automatic retry and failover. Since every job has log files attached, you can easily see and troubleshoot if a connection drops or there is some other issue. You can monitor how a job is performing, whether it’s local or remote,” Patrick said.

The associated log files from Deadline jobs provide useful statistics that help Pixomondo optimize its render farm performance. For ‘Furious 7’ and ‘Game of Thrones’ season 5, supervisors concerned with RAM usage reviewed metrics from Deadline job log files to guide how they used various machines. Another Deadline function is bracketing. For example, rather than waiting days for frames 1 to 100 to render, Pixomondo requests Deadline to render, for example, frames 1, 100, 50, 25, 75 first, then every 5th frame and only then complete the rest. This allows artists to quickly determine render issues before wasting resources on executing a complete pass.

As well as remote rendering, Pixomondo uses Deadline for file transfer and its asset library. With file transfer, coordinators can select a playlist with hundreds of associated versions in Shotgun and queue them for transfer to another branch in Deadline. Deadline then automatically retries failed transfers, outputs detailed logs and notifies users on completion.

Forasset archival, once a show finishes, Pixomondo chooses which assets to archive in Shotgun and then a script generates Deadline jobs that perform the actual archiving in the background on the farm, allowing hundreds of assets to be archived simultaneously, without blocking the computer. The archived assets are then automatically entered into anAsset Library projectin Shotgun. The restore is again triggered from Shotgun and kicks off a custom Deadline asset restore jobs. In case an asset isn’t yet available locally, a transfer job pulls it from a remote branch. On completion of the restore, Deadline automatically notifies the user via email.

“Deadline is very fast. With its new database repository, I can connect to a remote branch easily from my desk and watch the jobs output as it occurs. I don’t have to wait until a job is finished. To maintain consistency, we've committed all our repositories to a version control system so that we can change any of the plug-ins in one location and it will apply the changes across the other branches,” said Patrick.

Deadline’s event framework also simplifies workflow customization by allowing Patrick to hook into any stage of a job on Pixomondo’s farm and tailor its execution. This includes useful functions like the ability to customize jobs directly after submission in one central place, rather than having to implement these code changes in every plug-in.www.thinkboxsoftware.com