Ant Farm Handles Peak Render Loads with Qumulo Data-Aware NAS

Ant Farm’s editors and artists need to work with diverse types of equipment and software, and use a Windows-based computer cluster for rendering, often taking on large format projects that span massive video canvases at resolutions up to UHD 4K.

Heavy Storage Traffic

Each year in June as the gaming industry approaches the annual Electronic Entertainment Expo convention, E3, Ant Farm’s workload intensifies. The company doubles the size of its rendering farm and nearly triples its workforce during this peak season.

Eventually, overall performance of their render farm couldn’t keep up as their back-end storage system slowed down under the increased traffic. “When you have dozens of artists all working off the same storage at the same time, it’s a major challenge,” saidSam Frankiel, IT Director for Ant Farm. “Certain volumes would get hit harder as the work shifted toward high-usage projects, which our existing system couldn’t handle.”

Under the old set-up, storage was limited to an HP network-attached storage unit with one head and eight JBOD (just a bunch of disks) storage systems. When the whole system experienced slowdowns, it was difficult to pinpoint which projects were consuming the most resources. “We’d get calls about performance issues, but from there we relied on guesswork,” Sam said. “The process involved setting OS counters, watching when people were doing work and trying to isolate the problem.”

Visibility and Diagnosis

This shotgun approach to diagnosing issues was time-consuming and disrupted production. They might try rebooting a single node, or even the entire farm, but if that failed, they simply had to wait for artists to close out a project. The company began to look for a scale-out storage system to increase capacity and performance, that would also be able to generate the critical real-time analytics on data usage necessary for better management.

Ant Farm consideredflash-basedsystems andfile accelerationappliances, and products from mainstream providers, one of which was in use higher up in Ant Farm’s corporate structure. Sam explained that Qumulo was ultimately chosen because of visibility it afforded into how their storage system was functioning at any given time, and the data analytics to troubleshoot on a file-by-file or application level. “That was something we hadn’t seen from any other vendor,” he said. “The real-time analytics built directly into the file system allows the IT team to understand quickly which projects are putting a strain on capacity or performance.”

Other aspects of Qumulo that attracted Ant Farm were the interactive web-based REST API they can use to customise the user interface, and the highly scalable nature of the system, which runs on regular commodity hardware. The company runs fourQumulo QC24 hybrid storageappliances, with an Arista 10GbE switch. Qumulo Core’s flash-first hybrid design uses flash memory combined with hard disk drives to create a scale-out NAS, balancing performance, capacity and cost.

Real-Time Analytics

Ant Farm’s ability to use Qumulo’sintegrated analyticsto view real-time data performance on a file, directory or subdirectory level, and by client or by application has helped them make improvements to the way they manage files and workloads.

“The analytics generate reports that show when data thresholds are being hit. We can move files around proactively and alleviate burdens on specific nodes to prevent slowdowns,” said Sam. “When troubleshooting calls come at all hours of the night during our busy season, Qumulo has tools to diagnose and handle problems quickly.” Data is shown broken down by client and file path, so that they can drill down and see which IP address is accessing which file and what level of IOPS is occurring on a project. A user may need to close a file, for example, or a render node may be stuck and causing a slowdown.

Ant Farm also likes Qumulo’scontinuous software developmentand release cycles, which rapidly incorporate customer feedback and software changes. Sam said, “Qumulo’s ability to develop so quickly is very beneficial. If there’s an issue that we have or functionality that we need, we can see it addressed on a week-by-week basis.”  http://qumulo.com