Digital assets are the lifeblood of the Media and Entertainment (M&E) industry and have become the foundation for the global multi-trillion-dollar market 1. M&E content production follows a cyclical lifecycle from capture to creation, production, broadcast (or delivery) and archival, which requires a well understood data management strategy that is similar to contemporary enterprise IT challenges.
Given the diverse capabilities of flash, cloud and shared storage available today, one of the biggest data management problems in M&E is determining where to best locate data and when to migrate it to different storage types that better fit current production, consumption or preservation demands. Digital content creation in the film industry, which encompasses motion picture, television and video production segments, faces constant pressure to increase the pace of studio production, manage growing expenses, efficiently leverage previously produced content, and to cost-effectively protect against content loss with backups while providing long-term retention of assets.
Storage has a wide range of capabilities across performance, protection and price, yet M&E IT architects are routinely limited by solitary storage solutions that force IT to either best match a lifecycle stage or overspend to meet peak demands. In other words, today’s solutions are unable to span the spectrum of storage capabilities in a single system, which forces IT to take a siloed approach to managing digital data through the content lifecycle.
- Enable parallel data and metadata accesses to storage systems for higher bandwidth and reduced latency
- Archive and retrieve older assets to cloud or object storage without modifying applications
- Actively move backups or snapshots of production data to another location for disaster recovery and restoration of deleted or modified files
Figure 1 - M&E production pipelines must support numerous stages from content creation to distribution and archival.
Traditional storage architectures involve deploying a solution from a single vendor, manually moving data between storage types, then adding expensive caching tiers to improve performance, or overprovisioning resources to meet future demands. Primary Data’s DataSphere software can combine different storage types from different vendors to create a global namespace, then automate the movement of data non-disruptively using objectives that align data to the right storage, and ensure that desired service levels are met throughout the content lifecycle.
DATASPHERE: RIGHT DATA, RIGHT PLACE, RIGHT TIME
DataSphere is a metadata engine designed to separate the architecturally rigid relationship between applications and storage to achieve unprecedented improvements in performance, efficiency and scalability. It leverages the latest advancements in NFS 4.2 environments natively, or can be deployed with DataSphere Extended Services (DSX) in legacy, SMB (Windows) or Unix environments to gain insight into how an application makes use of its data. With this knowledge, DataSphere can place, move and distribute data to different storage types or tiers without disrupting an application’s access, even while the data is in-flight.
TAMING THE DIGITAL MEDIA REVOLUTION
Figure 2 - The DataSphere architecture allows the data path to be split from the metadata path, abstracting the logical view of data from its physical location.
With the advent of digital cinema and high definition televisions, studios are continually burdened with the changing demands of creating and managing ever-higher resolution content. The production of native 4K or higher resolutions easily hits the performance limits of today’s scale-out NAS solutions and drives up storage capacity. This often forces studios to use a compressed 4K resolution workflow, rather than the preferred uncompressed (or raw) 4K digital format, as it is both expensive and infrastructure-intensive to provide the amount of storage required and the rate of data bandwidth needed to support uncompressed work. For example, a complete 4K digital movie production may require several petabytes (1,000,000,000,000,000 bytes) of storage. 8K resolution requires almost three to four times more storage, and stereoscopic or 3D images doubles the amount of capacity needed, no matter the resolution.
Figure 3 - Higher resolutions increase the amount of data in each frame, compounding complexity for artists and IT.
Clearly, storage needs to hold a lot of raw data, but the real challenge comes when final composite, color grading, digital intermediate (DI) and editorial stages of production are loading uncompressed 4K image sequences consisting of a single 50MB file for every frame of video. Processing 24 of these files per second requires bandwidth and low latency, with the ability to sustain bandwidths exceeding 10Gb/sec from storage. It doesn’t end there. When creating a digital master from either an uncompressed digital source or filmed originals, some prefer using the metadata rich Tagged Image File Format (TIFF), a standard non-proprietary archival format for lossless storage of image files. TIFF adds metadata for all the spatial (resolution), color, tone, metadata and preservation information required to create a digital master and commonly the basis for digital image processing increasing the storage load. The demands of meeting metadata requests simultaneously with the digital content is challenging for today’s solutions.
In motion picture post-production, the Digital Cinema Initiatives (DCI) specifies several standards for distribution to theaters and for preservation. The good news is that compression ensures the final release is smaller than the amount of data managed in production. For example, the 4K DPX standard has the lowest demand at 4096x2160 pixel resolution, 10-bit color depth, 24 FPS and a maximum of 1.3MB per frame (aggregate of all three-color components including headers) requiring about 6.4Gb/s of bandwidth. The bad news is the amount of content the M&E industry has to store and maintain over time. Major studios and networks have millions of films and tapes, as well as tens of thousands of TV programs representing 50-60,000 petabytes of archived content. More affordable, online and active solutions are needed to store all these assets.
SUPERCHARGE NAS FOR UNCOMPRESSED 4K WORKFLOWS
A modern scale-out NAS system is unable to keep up with the bandwidth demand because it provides an artist or editor’s application with a single shared mounting point using a 10Gb Ethernet network. The problem worsens when several artists work simultaneously, as there is no guarantee that each user’s application will be mounted to a different share. Thus, each share contends for bandwidth. By using DataSphere, a logical cluster with file-granular load balancing can distribute the video frame files so that the client can simultaneously access the files in parallel over multiple 10GbE connections across multiple NAS nodes, increasing performance and reducing latency on existing infrastructure.
Figure 4 - DataSphere can automatically load balance across storage resources to avoid performance bottlenecks.
ACCELERATE PERFORMANCE WITH METADATA MANAGEMENT
Metadata is simply information about data. For the M&E industry, metadata about production content helps to manage digital assets more effectively, whether that data comes from artist information about a single rendered video frame, editors and producers revising a scene, or the corporate tracking of assets.
However, in an NFS environment, metadata operations commonly wait on previously requested NFS data operations within the same NAS storage device or share. This means when a file is accessed, an incoming metadata request must wait until the data operation is completed. For very large files, this could add significant delays that can lead to applications timeouts and impacts on production.
DataSphere offloads metadata access with predictable, low-latency metadata operations by guaranteeing that metadata operations do not get “stuck” in the queue behind other data requests. Rather than having to wait for sequential operations to complete, DataSphere can leverage parallel access with the latest optimizations of the standard NFS v4.2 protocol. This significantly speeds up metadata and small file operations by requiring less than half of the protocol-specific network round trips.
TAKE CANDID SNAPSHOTS FOR SECURITY AND SAVINGS
Many companies augment their traditional backup and recovery strategies with array-based snapshots as a convenient shortcut to continuous backups or restore an entire dataset to a previous point-in-time. This may be risky for production data. Unless snapshots are moved to another storage device, they are exposed to system failures. To ensure data is kept safe, retaining a series of snapshots over time requires companies to purchase additional capacity. Over time, this either gets very expensive, or IT ends up prematurely deleting older snapshots to reclaim capacity on production systems.
With DataSphere, backups or snapshots of production data can be actively moved to another location for disaster recovery and restoration of deleted or modified files. DataSphere enables enterprises to set objectives to move aged snapshots off the production editing infrastructure to a lower cost storage alternatives, such as the cloud. When data needs to be recovered or restored from a series of older snapshots, they can be easily accessed by management software while DataSphere moves the data back to the production environment to complete the recovery entirely automatically.
This means the cloud can now be viewed and used as just a low-cost, lower performance, highly reliable storage resource. However, rather than only limiting the cloud to archival operations, M&E companies can achieve even more savings and agility by leveraging the cloud as a unique tier for IT to store cold data, manage snapshots, ensure data governance, or enable use on-demand usage models.
Figure 5 - DataSphere automatically moves data to the right resource to meet IT-de ned objectives. This allows snapshots to automatically move to the cloud and back again if needed.
BREAK THE BOTTLENECK IN M&E IT INEFFICIENCY WITH DATASPHERE
M&E IT is already challenged with meeting the demands of high definition content production and preservation while keeping costs down. As resolutions continue to increase, audiences and studios demand uncompressed formats, and the preservation of remastered or older content grows, DataSphere helps M&E IT break vendor lock-in and choose new or different types of storage, meet production performance requirements, and seamlessly integrate the cloud for savings and archival. The simple solution is to put the right data in the right place at the right time, automatically.
1. 2015 PWC Entertainment & Media Outlook
Use the Primary Data TCO Calculator on our web site to see how much DataSphere might save you.
Connect With Us
Enter your name and email to receive news and updates from Primary Data. Fields marked with an * are required:
Thank you for contacting us. We will be in touch shortly.
There was an error submitting your request. Please try again.