Spectra Logic StorCycle Manages and Automates Tiered Storage Migration

Spectra Logic StorCycle is storage management software that makes it simpler to configure and take advantage of tiered storage. Although the use of storage tiers is not new, it has been a challenge for companies to organise their data effectively over time between disk, cloud, object storage and tape tiers. Active and inactive data are still being stored together on expensive primary tier storage, even though data is growing faster than budgets and is harder to share outside an organisation when stored in this way.

Data stored unnecessarily in primary storage becomes expensive and harder to manage, and needs to be moved to a more cost-effective location. Maintaining a smaller primary storage tier needs less backup and recovery, and generally increases overall performance. Spectra Logic developed StorCycle to help manage and automate data migration.

Perpetual Tier

StorCycle involves the use of a secure Perpetual Tier of storage,  which can be made up of any combination of cloud storage, object storage disk, network-attached storage (NAS) and tape. Using a combination of storage types protects the data and makes it available and accessible to end users.

The software is based on four main tasks. First it identifies inactive files on primary tier storage before migrating and storing the inactive data according to user-defined criteria to one or more Perpetual Tier storage targets - cloud, disk and/or tape. Files are then protected so that they stay intact and retrievable during long-term storage. Finally, migrated files are accessed and retrieved from the Perpetual Tier.

StorCycle can be implemented as standalone software using public cloud storage or existing network-attached storage, or combined with Spectra Logic storage hardware, to set up and customise a complete storage system that moves data as an automated process.

Scanning and Migration

Manually identifying files that need to be moved may be complex and time-consuming. Through its server, StorCycle is able to scan all files in a configured storage location. The scan identifies information about the files such as size, type and age, which helps users make informed decisions regarding which data they would like to migrate to a perpetual tier. StorCycle is capable of scanning specific directories, or the entire file system of a storage location.

Once a scan is complete, the results show how much data is in the pre-defined primary storage location and gain insight to the location’s contents. The scan results can also be used as part of a migration job to move some or all of the files to the Perpetual Tier of storage.

Because fast scan speed is another critical factor for storage management to be useful, StorCycle has been developed for scan performance. Two limitations are considered – the rate at which information can be collected from the primary storage system about the files being scanned, and the rate at which that collected data can be recorded in the management system’s database. StorCycle focusses on server memory to overcome these limitations. Scan data is collected in memory and then written to the StorCycle database in batches to optimise write performance. StorCycle also uses hundreds of parallel programming threads to collect scan data.

File Replacement

In order to locate the migrated or copied data, StorCycle has user-selectable replacement options that determine how particular data is accessed in the future. One method is to replace the file with a HTML Link file in place of the file, which users click on to restore the source file. A copy of the source data can also be kept on primary storage and another copy made and placed on secondary locations, in effect using StorCycle as backup.

Alternatively, the scanned data can be moved and deleted from the primary source, leaving no replacement file or link behind. The files can then be found and restored via the search tool in the StorCycle web management interface. Following a future release, organisations using NAS will have the option to replace a file with a symbolic link that points to the file on the NAS system.

Project Archiving

StorCycle has a Project Archive function used to tag and move entire project data sets to a perpetual storage tier, which allows the complete project to be accessed for further analysis, categorisation and comparison, preserving it for as long as necessary. In other words, the Perpetual Tier of storage is not limited to older data - it can also be used as an archive tier for large data sets that can be moved immediately after creation or collection.

Organisations like universities and research labs tend to regularly create enormous amounts of data that needs to be managed separately from the traffic of high-performance, primary storage. If users do not have the means to keep various data sets together or track them once they are moved, they may end up leaving them on the primary storage indefinitely.

Instead, they can use StorCycle to automatically migrate the files, folders and directories associated with a given project. The migration task will create a manifest file which shows data migrated, where it was migrated from and where it was migrated to. The archive can also be tagged with searchable information relating to the project.  spectralogic.com