Atomic Cartoons animation studio in Canada has been in operation since 1999, and today produces well known, award-winning children’s content such as ‘Dogs in Space’, ‘LEGO STAR WARS Terrifying Tales’ and ‘Spidey and His Amazing Friends’.

Between their three major facilities, which include the Atomic headquarters in Vancouver, a studio in Ottawa and a location opened recently in Los Angeles, the company hires about 1,000 staff members, mostly animators, directors, producers and writers. Prepared to take risks that trigger innovation, despite their clients’ sometimes aggressive timelines, Atomic’s philosophy is that every project must be a progression from the last, improving in quality and production values that will attract further work.

Their project workflows range across 2D and 3D animation, currently comprising about four petabytes of data files. The 3D files, in particular, have taken up an increasingly significant portion of their storage space over time. While 2D projects involve large numbers of very small files, 3D work produces large files due to complex renders, compositing and simulations.

The teams’ software ranges from Toon Boom Harmony and Storyboard Pro and Adobe CC tools including Substance, to Autodesk Maya, Foundry Katana and Pixar RenderMan and Tractor. As content production has grown in various ways – across new pipelines and covering genres from comedy to action adventure – the demands that Atomic’s artists make on the company's storage system have also continued to increase. Consequently, data management is a major, on-going feature of running the company and needs an evolutionary approach.

Any Kind of Workflow

Robert Crowther is the IT Manager for Atomic Cartoons, and aims to build and maintain a storage system that can be adapted to handle any kind of workflow they might need to set up, especially 3D workflows. The IT team needs to be able to manage their data in a straightforward manner, update the system quickly and generate reports at any moment.

Because his team is small and responsible for numerous priorities and requests as well as hosting the range of software required for different workflows, they looked for a system to support Atomic Cartoons’ full cluster. They have now transitioned more than half of the company’s data storage to Qumulo, which has helped IT support its internal clients, artists who need to focus on deadlines and outputs.

Qumulo’s own development team have their roots in media and entertainment, where they tend to find businesses are more willing than in other industries to innovate regarding data and storage infrastructure. Regularly facing changing demands on their capacity and performance, in order to accommodate new workloads, content creation companies are prepared to look for the best system available at any time, even if it means venturing into something new.

Distributed Workforce, Distributed Data

“As the tools and projects we work on expand, creating bigger and bigger files, IT can use Qumulo to manage, retrieve and distribute Atomic’s file data across locations and a distributed workforce. Having this ability has become a higher priority since the pandemic, when remote work became absolutely necessary,” Robert said.

“As we ran out of space in Vancouver, at times exceeding 85 percent capacity in order to support numerous workflows, it was never a matter of one department or a single project that slowed down, it was everything simultaneously. If we then had to wait for a drive replacement and hit shipping or delivery issues, further production inefficiencies and slowdowns resulted.”

Another challenge were file backups timing out, which stalled server communication. When this occurs, staff are prevented from opening programs, or experience lengthy delays when publishing scenes or shots. In other words, because everyone’s work is connected through the storage, the moment it slows down, all aspects of production are affected. System upgrades typically took up at least a full work day or more, which Robert considers unacceptable for a media and entertainment environment. These combined issues led to staff dissatisfaction, stress for production teams, and frustration for the IT team.

Truly Useable Capacity

These kinds of challenges, arising from inefficient use of capacity, is what led Atomic to Qumulo. In legacy file systems, the need to allow for data protection, for small file storage and handling, and for disk rebuild and recovery operations means that a certain amount of capacity is not going to be available for day-to-day use. Trying to run a facility at 85 percent capacity begins to cost the organisation in terms of data protection and integrity.

Nevertheless, all disks fail eventually, and the data can’t be allowed to suffer or be lost. In many systems, protection generally relies on some form of redundancy or duplication of information such as mirroring across storage devices, which requires space.

Qumulo makes it possible to devote all usable capacity for files – even at petabyte scale – without sacrificing performance or data protection because it protects data at the block level, instead of the file level. Files tend to be of an arbitrary size, which cannot be planned for, whereas Qumulo can use its own block storage design and regulate the file sizes that have to be managed. Having control at this level makes it simpler to rebuild data after a drive fails.

Protected Virtual Address Space

The data Atomic stores in the Qumulo file system is managed through the Qumulo Scalable Block Store, a virtual layer of protected storage blocks. Instead of a system where every file must be accounted for separately, Atomic’s data protection exists beneath the file system, at the block level.

Conceptually the entire storage capacity of a Qumulo cluster is organised into one protected virtual address space, and each protected address within that space stores a 4K block of bytes. Protection comes from the fact that all blocks are recoverable, even if multiple disks were to fail, because the entire file system is stored within the virtual address space, including the directory structure, user data, file metadata, analytics and configuration information.

The protected virtual space serves as an interface between the file system and block-based data recorded on attached block devices, which could be virtual disks formed by combining SSDs and HDDs, or block-storage resources in the cloud. Even though the blocks in the address space are distributed across all of the nodes, or instances, of the cluster, the Qumulo file system only sees and deals with a linear array of protected blocks.

Block-level protection also avoids the use of mirroring that legacy systems use when storing small files. In a Qumulo system, there’s no difference in storage efficiency between large and small files. Furthermore, a Qumulo system sets aside the space it needs for administrative tasks in advance, before it reports usable capacity, instead of expecting the user to leave space free for this purpose. That means you can still recover from drive failures even if you are working at 100% capacity.

Proactive Support

Robert noted that Qumulo’s visual reporting of system status and health is very timely, which was not the case with their earlier systems. “We could generate reports, but by the time they arrived, the system would have moved on and the information would have become irrelevant,” he said. “Now we can produce reports that suit our workflows – for example, visualising render jobs in terms of speed and consumption – in almost real time.”

One aspect of working with Qumulo that Atomic especially favours is their proactive support services. “Not only does Qumulo’s staff work out specific ways to correct Atomic’s storage issues or bugs or answer their questions, but if the support team sees an unusual change in the system, they also work pre-emptively to anticipate requirements without waiting for Atomic to create a ticket,” Robert said. An update that Qumulo is planning to release soon is the automation of various types of fault notifications.

Atomic also have the option to communicate live over Slack with Qumulo’s engineers and support staff, a much quicker way for their work groups to communicate than using email. Because Atomic runs its storage relatively close to capacity, acting quickly helps the artists to continue working within their deadlines. qumulo.com