Canada’s CPAC Leverages DDN Storage Tech for Ballooning Data and News Content 

For the Cable Public Affairs Channel (CPAC), the C-SPAN of Canada, 2015 posed two major challenges – one of them data intensive, the other operational.

On the data side, CPAC’s data management and storage infrastructure was nearing capacity, in part because CPAC was transitioning from standard definition viewing format to the denser high definition format. Operationally, CPAC was tasked with covering Canada’s upcoming federal election, which meant months of extensive political campaign coverage culminating with the election in October.

Like other news outlets in the media and entertainment industry, CPAC runs a real-time, data intensive shop. CPAC programming is delivered by cable, satellite and wireless distributors to more than 11 million households in Canada and around the world via 24/7 streaming through its TV2GO mobile app and its website. Up to 80 hours of new content is ingested daily and over 30,000 hours of archived content is available on its website.

CPAC has been streaming live content for approximately 15 years and can deliver up to 10 simultaneous live event streams at a time, according to Eitan Weisz, senior manager of technical operations. “We generate about 4TB of data daily in ingest, editing, filtering and producing new content. CPAC’s producers can access an additional half petabyte of high-resolution programming in our online data archive.”

Weisz and his technical team collaborate with CPAC editors and producers to review, edit and translate (English programming into French and vice versa) all incoming content, and then render it into final form for CPAC’s linear channel, website and mobile app delivery. CPAC’s long-form programming involves rendering videos that are hours in length, which can be taxing on its storage system. In addition, CPAC permanently archives finished content. It also is participating in an effort to digitize video of proceedings from the Canadian Parliament in its archive.

In 2014, CPAC realized its data requirements were expanding rapidly.

“In the course of an election, content sometimes balloons enormously,” Weisz told EnterpriseTech. “In an IT-centric or file-based environment, more and more content is coming to us from other broadcasters, from other content providers globally through file transfer using FTP, Aspera or other such faster transfer mechanisms that make it easier to acquire content, makes the content more timely. So the amount of content we’re acquiring is increasing, the amount of content we have to edit and produce is increasing, and therefore we are also archiving more content.”

With the decision to go all-HD and with the upcoming election, CPAC decided to make changes.

“We wanted to make sure we had the throughput and the bandwidth capabilities to handle the increased acquisition of content and production of video-on-demand material,” Weisz said. “So we began a major push toward that upgrade and we built it with a fair amount of overhead from a storage capacity perspective to meet our requirements for the next few years.”

CPAC’s content rendering workflow requires a storage infrastructure that can rapidly ingest and store large amounts of raw video data from various news sources, data that is accessed and edited by CPAC staff using the Dalet Digital Media System and other editing platforms. This requires a storage platform that is high performance, highly reliable and scalable.

CPAC’s storage platform for ingest and data access was DataDirect Network’s SFA10K, delivering 4 GB/second performance. For archival storage CPAC used legacy hierarchical storage management (HSM) software to migrate data to CPAC’s robotic tape library. But Weisz said managing the tape library was a demanding and costly process that grew worse as archived content grew exponentially.

“Maintaining the tape library and HSM application became incredibly expensive and labor intensive,” said Weisz. “The cumbersome process of managing both required up to 15 percent of someone’s time each day.” Moreover, tapes frequently went offline or become corrupted. “We simply didn’t have the manpower or the time to stay on top of our tape vaulting demands,” he said, adding that CPAC strives to achieve “five-nines” [99.999%] uptime.

As the organization’s 400-tape library edged closer to capacity, the team considered a move to a new tape format requiring replacement of its tape drives. This posed a significant capital investment along with the time-consuming process of migrating existing content to a different tape format. “It became clear that moving to a disk-based solution would enable us to keep pace with data growth without the cost, reliability concerns or administrative burden of tape management,” said Weisz.

So CPAC embarked on a technology evaluation to find an end-to-end solution, from initial ingestion through to archival storage. Weisz said DDN proposed its SFA12K storage platform, to support CPAC’s requirement of 8 GB/second throughput, integrated with its WOS Object Storage technology for permanent storage.

“Some of the other vendors’ proposal involved putting together two storage solutions from different companies, one for our high-resolution production and editing storage, and another for the archive,” Weisz said. In selecting DDN, Weisz said single vendor accountability was a plus.

“We felt that having a unified solution from a single vendor” was an advantage. “We would have a single point of contact that we would reference, as well as having a single, united monitoring solution that could encompass our production and archival needs,” he said.

DDN WOS

Of the two DDN products, Weisz said, WOS object storage may have had strongest appeal. With WOS, CPAC can leverage the same content for both disaster recovery and collaboration by remotely mirroring data and allowing remote teams to access content independent of the primary site via WOS Gateways located at a secondary site.

“The ability to add DR helped clinch the decision to go with WOS,” said Weisz. “We could quickly build a DR site without the complexity of maintaining and synchronizing a separate file system with the primary site.”

Weisz said the DDN solution positions CPAC to accommodate evolving workflow demands for ingesting, storing, distributing and archiving content. The team uses its upgraded SFA12K platform to ingest data and transfer it to CPAC’s various editing platforms at the same time. Once the edits are complete, the final content is returned to the SFA12K for use within the Dalet asset manager, which transfers it to CPAC’s Harmonic media server for on-air transmission.

After content has been aired, assuming it won’t be used again soon, it is moved to the WOS platform, which then stores and distributes the data as needed. Weisz said that after migrating content from its tape library onto the WOS platform, archival administration dropped to near-zero. He said WOS works in the background retaining hundreds of thousands of hours of content in an active-archive.

“We generally don’t give WOS a thought, which is what we’d hoped for,” said Weisz. “If we need to retrieve footage of a dignitary on the spot, we now have the confidence to pull all the archive footage without delay” because of WOS’s disc speed access. “Previously, there was a large degree of uncertainty whenever we had to retrieve content from tape.” He added that in the future, CPAC may render a video right off WOS.

Weisz said WOS also bolsters data protection and disaster recovery. “We like having the ability to use WOS to mirror to an off-site location for DR purposes,” Weisz said. “Accessing content from anywhere by synchronizing the data transfer between WOS nodes is a major advantage.

Weisz said CPAC determined that even with three years of support, the cost of the new object storage platform is equivalent to what the organization paid in a little over two years of annual support fees on its robotics library and HSM software.

“We were able to increase our production and active archive usable capacity to more than 1PB while lowering costs,” Weisz said.