Weekend Outage Puts Verizon Cloud on Defensive 

Verizon continues to take heat this week following what was apparently a planned 40-hour cloud outage last weekend that was attributed to a massive upgrade of its enterprise-class cloud infrastructure aimed at preventing future outages.

Responding to growing pressure to explain why its "seamless upgrade" resulted in a 40-hour outage, Verizon issued a statement on Thursday (January 15) that provided few details about precisely what happened over the previous weekend.

Verizon CTO Kevin Clarke, noted again that the Verizon cloud was built from the ground up to be a "pure virtualized infrastructure" Added Clarke: "All endpoints (virtual machines, networks, storage volumes and associated snapshots) are virtualized and network addressable."

That means orchestration, firewalls, load balancers, identity management and other subsidiary network functions are also virtualized, Verizon said.

The focus on fully software-defined infrastructure would give Verizon Cloud customers "control over availability and performance management" of cloud primitives like CPU performance, network bandwidth and disk throughput, Clarke added.

Verizon Enterprise Solutions said earlier this week that it had completed the scheduled upgrade of its cloud infrastructure and storage platform last Sunday evening (January 11) to include a "seamless upgrade functionality" that would allow the carrier to perform major systems upgrades in the background without interrupting service or limiting infrastructure capacity.

This approach would eliminate the need for customers to set up virtual machines in multiple regions or upgrade domains. The functionality would also eliminate the need to reboot virtual machines after maintenance or upgrades occur, Verizon explained.

Much of Clarke's summation of the status of the Verizon cloud upgrade reiterated what the company announced in October 2013 when it disclosed plans to take on the likes of Amazon Cloud Services (AWS). Rather than using off-the-shelf software as most public cloud operators would, it decided to start from scratch to create, for example, its own hypervisor and controller.

Clarke, then serving as Verizon's director of cloud engineering, told EnterpriseTech at the time: "We asked ourselves what we would need to change to build a cloud that the enterprise wants, and the answer, regrettably, was everything. So that's what we did."

While Verizon has since been less than transparent about the lengthy weekend outage, it has reported some progress on the enterprise cloud front in recent months. For example, the Enterprise Solutions unit announced in November it had gained federal approval to offer cloud services to government agencies. The approval clears the way for Verizon to initially provide cloud services to the U.S. Department of Health and Human Services.

In September 2014, the Verizon Enterprise unit announced that its Secure Cloud Interconnect service that was working with Microsoft Azure would also work with AWS. The service essentially helps customers decide when and how to move workloads to the cloud.

In December 2014, Verizon announced that the service was extended to Hewlett-Packard's Helion managed cloud service and the Salesforce customer platform.

While providing few details about last weekend's outage, Clarke claimed the downtime was worth the heat the company is currently taking: "Updates and maintenance, even unscheduled maintenance like those required with the Xen hypervisor security flaw last fall, can be performed while client virtual machines continue to run, without requiring a two-zone setup or a machine reboot.

"It’s a little like performing brain surgery on yourself while you’re awake," he added.