Verizon Takes On AWS With Homegrown Cloud Code 

Here is something you don't see every day: A public cloud operator that starts from scratch and creates its own hypervisor and controller rather than using off-the-shelf software. Amazon Web Services did it back in 2006, and now Verizon is doing it with its next-generation cloud.

At the Interop networking conference in New York today, Verizon's Terremark cloud unit will unveil its next-generation cloud platform for enterprise customers, which goes into beta testing this month and which will be available for production workloads in the beginning of next year. Kevin Clarke, director of cloud engineering at the telecommunications giant, talked with EnterpriseTech about the technology choices that Terremark made to build a cloud that is faster and more malleable than its current products or those of its competitors.

"We asked ourselves what would we need to change to build a cloud that the enterprise wants, and the answer we came up with, regrettably, was everything," Clarke said with a laugh. "So that's what we did."

The new cloud bears the Verizon Cloud name, not the Terremark brand, and comes with compute and object storage as its initial offerings. The compute cloud is called Verizon Cloud Compute, and the object storage side is called Verizon Cloud Storage. Clarke is not revealing all of the secrets of the underlying server and storage infrastructure that underpins the Verizon Cloud, but he did talk about some of the in-house development that Verizon has invested in as part of a two-year effort.

The first interesting  – and perhaps shocking – area of investment was the server virtualization hypervisor, and the company has created what it calls an omnivisor, in fact.

"One of the things that we have implemented with our Xen-based omnivisor is the ability to run natively alternate formats of VM images from different hypervisors," explains Clarke. "ESXi is the one that we have implemented right now, so we can import and run VMDK images on Xen. We actually implemented the driver binding stacks with QEMU. Ultimately, we will be doing this with others. Pick your favorite ones – KVM, Hyper-V, we will do all of them because we know how."

The choice of VMware's ESXi as the first format to run on the Verizon Cloud makes sense for two reasons. First, VMware's ESXi hypervisor is by far the most popular server virtualization hypervisor in enterprise data centers for X86 machines. And equally importantly, the current Enterprise Cloud run by Terremark is based on ESXi and VMware's vCloud Director cloud orchestration tools, so the new cloud has to be compatible at the virtual machine level with the current cloud.

The orchestration layer for the Verizon Cloud is all based on Java, and Clarke said that it has implemented REST APIs out the front end "like you kind of have to because it is the lingua franca of the Internet these days." The way this orchestration layer is setup, Verizon is able to support the different orchestration and management APIs from various public clouds, like the proprietary ones created by Amazon Web Services and Google, and open source cloud controllers such as OpenStack and CloudStack, which will be used by various public clouds as well as being deployed in enterprise private clouds.

"My background is at Sun Microsystems, and we believed in open APIs," said Clarke. "We have our own API set now, but in short order we will support CloudStack and OpenStack. And we will support and go right after Amazon. I want their applications. I want the applications that are written to run on these other platforms to run on mine. And I want to change the competition from one about price, where everyone has to be in the same ballpark, to one about operational efficiency, deterministic performance, security, and the value ecosystem around telecommunications."

Verizon is a credible player in the public cloud space. It bought MCI Communications in 2005 for $8.4 billion, using it as a platform to build a global IP network and a large data center hosting business. Two and a half years ago, it paid $1.4 billion to buy Terremark, which was generating close to $350 million in annual sales when the deal was done. Verizon, which had $115 billion in sales last year, does not break out revenues separately for its hosting and network businesses, so we don't know how large they are. The Terremark unit operates 50 data centers and Verizon Enterprise Solutions over 200 data centers of various sizes to providing hosting and co-location services. Its global network has 100 Gb/sec backbones – and has had that high speed for the past four years – and is available in more than 2,700 cities in 150 countries.

The servers underpinning the Verizon Cloud Compute do not have disk drives, but rather are based entirely on solid state drives, which is another surprise. But Verizon wants to compete on performance as much as it does on price.

"I could not get multitenancy and deterministic performance to work if I did not use SSD because of the noisy neighbor problem, the scheduling algorithms, and all that jazz," said Clarke. "A couple of objections always get raised there, and one of them is the wear-leveling problem with flash. Rest assured, I was the system architect at EMC for the VNX arrays, and I kind of get SSDs and I have a lot of storage guys on my team. We have done some things to proactively manage the faults by migrating and replicating data when they hit a certain limit. The other thing we manage is cost. I have always worked at vendors – Sun, EMC, Digital – and now I am at Verizon and I get to say, 'No, that is too much money. I am not going to pay that, it is an outrage.'"

Verizon is not talking about its choices for servers and switches, but EnterpriseTech is working to get some more details on that.

What Clarke did reveal is that the capacities on the compute cloud are scalable, not fixed in T-shirt sizes from small to extra extra large as is done on Amazon Web Services and mimicked on many other clouds. By implementing its own hypervisor, Verizon can independently scale all capacities on the compute cloud. The scheduler in the hypervisor supports virtual clock speeds of 500 MHz, 1 GHz, and 2 GHz. Virtual machines are scaled up based on the cores and threads in the server node (which Verizon did not reveal), and main memory can be scaled up on a virtual machine from 1 GB to 60 GB. Networking bandwidth into the virtual machine scales on 10 Mb/sec increments, and disk I/O can scale from 100 to 5,000 I/O operations per second.

"This is an important thing. This is not just controllable performance on a slider, but raw performance, too. The default platform, without anything special allows a VM to have 5,000 IOPS, and that is faster than anything out there."

The company is also not saying much about the hardware behind the Verizon Cloud Storage services, either. But Clarke said it supports Amazon S3 and OpenStack Swift storage protocols as well as the generic DAV storage service that is part of the HTTP Web server stack. Verizon Cloud Storage is based on storage arrays acquired from a third party vendor and are not based on arrays that Verizon built itself using generic servers and open source file systems. Verizon is not revealing what storage products are being used here, but Clarke did say that Verizon added in software for identity management, metering, and other functions into the arrays.

Pricing for the Verizon Cloud has not been set yet, but will probably be done by the end of the fourth quarter ahead of the production rollout next year, a Verizon spokesperson tells EnterpriseTech. Clarke did confirm that Verizon would have two ways to buy capacity on the new cloud. The first will be directly over the Internet with the list pricing for capacity and payable with a credit card, and the other will be what he called the "martini and golf set" with negotiated pricing on larger amounts of capacity.

Beta customers using the new Verizon Cloud will be served out of Terremark's data center in Culpepper, Virginia, but machinery for this cloud service is already installed in data centers in Englewood, Colorado; Miami, Florida; Santa Clara, California; Amsterdam, the Netherlands; London, England; and Sao Paolo, Brazil. The current capacity of the machines installed is around 50,000 total virtual machines.