ITV Preserves Digital Assets for the Future with Spectra Black Pearl

ITV in the UK is launching a long-term archive project to preserve its digital assets for the future. They have anticipated reaching volumes of up to 20PB of content over time, with an estimated increase of about 2PB each year, and intend to store it for up to 30 years. Therefore, they need a durable, high-capacity system, able to scale to accommodate their future growth. They also wish to avoid expensive, proprietary middleware, employ open standards, and maintain enough flexibility to work with other creative ITV departments like sports and news.

At the time they started working on the archive project, ITV was operating three Spectra Logic tape libraries at other locations across the UK. For the archive, they purchased two Spectra T950 Tape Libraries and two Spectra BlackPearl Converged Storage Systems to preserve content at two geographically dispersed locations on different media types, which is becoming a typical approach to long-term preservation archives.

Genetically Diverse Strategy

Called genetic diversity, this strategy stores information across a range of media types such as SSDs, SAS HHDs, SATA HDDs, cloud and tape, to digitally preserve data against hacking and more general attacks on information. Geographic dispersal of storage, a related strategy, requires data to be stored in two or more locations.

A necessary component of this approach to digital preservation is accuracy of replication, which Black Pearl can perform in an automatic or manual scenario, as well as the use of tape which has inherent offsite capabilities. The BlackPearl system writes data on tape in the open LTFS format to make access easier over the longer term.

ITV is currently using Spectra products at their Greenwich and Leeds locations. At Greenwich they have a BlackPearl system and a dual frame T950 tape storage library with five IBM TS1150 drives and media, and at Leeds they have a BlackPearl and a four-frame T950 with six LTO-7 drives and LTO-6 Media.

Interfacing with Black Pearl

Several creative departments within ITV simultaneously move content to and from the organization’s preservation archive automatically via BlackPearl using various integrated partner applications and a custom-built client, and manually via the Spectra BlackPearl Eon Browser, a graphical interface for managing files between BlackPearl and local computers or servers. ITV also expects to deploy a number of new MAM clients within certain workflows in the future.

BlackPearl Converged Storage Systems have a simple, cloud-like interface to on-premise storage, including Spectra tape and disk libraries, and several public cloud integrations. “Spectra’s BlackPearl employs a ReSTful interface, much like Amazon’s S3. Through this interface, client applications can talk to BlackPearl over an HTTP connection,” said Matt Starr, Spectra Logic CTO.

“Instead of multiple ReSTful interfaces, all data flows into the BlackPearl object storage system over the same path, and each client that talks with BlackPearl can maintain its own credentials, buckets, and so forth. A client packages up files that it would want to send to BlackPearl and then initiates a bulk put. This command looks very much like the S3 put, but also handles multiple files in one command. The same can be said for getting data.”

This cloud interface is called Spectra S3, and a software client is required to transfer data to the Spectra S3 interface. Many Spectra S3 clients already exist that are integrated directly into asset management systems. Certified clients are prepared for MAM systems from Avid, EMAM, Vizrt, CatDV and many others. Customers can also build their own clients, called Customer Created Clients, as ITV has done, or modify existing clients to suit their workflows and environments.

Tape, Disk and Cloud

The Spectra certified clients write content to one location, and BlackPearl replicates the content automatically to a second location through its Advanced Bucket Management, which works well for companies such as ITV using a genetically diverse protection strategy. BlackPearl can make one or more copies of data to Spectra's tape, disk and cloud targets. For the disk storage targets, the Bucket Management can also control how long the data resides on disk. This creates a flexible platform and storage management system.

Such integrations are made simpler with BlackPearl’s multi-tenancy capability. Marcel Mester, senior project manager, technology, platforms and broadcast operations at ITV, noted that where a MAM system satisfies a business unit's requirements, the MAM will support multiple workflows. “The multi-tenancy capability ensures that different business units can all share the same Spectra Logic archive, no matter whether their front end are bespoke integrated applications, third party MAMs or manually through drag and drop techniques,” he said.

As mentioned, ITV chose to use an automated Customer Created Client as well. Marcel talked about using this option. “While third party suppliers have built clients into their products, usually MAM applications, where ITV's systems have asset management capability, it is more appropriate to integrate these systems with the Spectra BlackPearl,” he said.

Spectra Logic has set up a Developer Program for developers to create their own custom clients to interface with BlackPearl. The Developer Program contains the necessary components, available for download, and most are open source. Developers can see the source code of these components, which allows them to customize the components for specific storage workflows. 

Cloud-out

Customers using the cloud also need the cloud interface Spectra S3, along with their software client to transfer data to the Spectra S3 interface. Matt said, “When using Spectra S3's 'cloud-out' function, the customer has two choices. Spectra’s BlackPearl can use cloud-out as part of its own storage policy engine. For example, currently you can set a policy to make a copy on disk for 120 days, two copies on tape and eject one of those copies, you can also use cloud-out as part of a storage policy.

“In that case, a customer can create a storage policy that has one copy on disk for 120 days, one copy on tape, and one copy in the cloud. This would provide a fast local copy for 120 days, a cold, local copy forever on tape, and a copy for disaster recovery in the cloud. The customer could also opt not to use BlackPearl cloud-out and make their own copy to the cloud using standard S3.”

Spectra Logic’s archive system makes ITV's digital workflow more direct by involving no middleware and simplifying the management of their assets. ITV can continuously monitor and assess the security of their programming content as well, using Spectra’s certified media and media health reports.

Moving Data

Data can be transferred to a BlackPearl converged storage system, and all of its supported storage targets, via five different workflows spanning backup, manual data movement, automated or partly automated data movement, and purpose-built integrations. Any customer can access any or all of these workflows as a data storage and management resource. Whether an organisation offloads its Tier 1 primary storage, or designs and implements a whole system balancing the number of copies, locations and their tape, disk or cloud storage targets with cost, all this can be handled in a single system.

“Since most media and entertainment customers have a set workflow that is tuned over time, but rarely scrapped and started afresh, Spectra’s processes just plug in to this workflow in almost all cases,” said Matt. “Only at certain times, like system migration, when customers may be changing from one MAM to another, does Spectra need to be more heavily involved with design of the workflow and migration.”   www.spectralogic.com