Verizon’s New Cloud Is AMD SeaMicro Inside 

Advanced Micro Devices has been looking for a big server maker to adopt its SeaMicro microserver chassis and integrated fabric interconnect, but has found no takers. AMD has, however, inked its first big technology agreement deal with none other than Verizon.

That's right, Verizon, which last week announced a new compute cloud that, it turns out, is based on heavily customized versions of AMD's SeaMicro systems.

The prior Enterprise Cloud built by Verizon's Terremark cloud division was based on rack servers running VMware's ESXi hypervisor and vCloud cloud controller with standard top-of-rack Ethernet switches from a number of different vendors. This time around, Verizon is not only cooking up its own hypervisor and cloud controller, as EnterpriseTech revealed last week, but it is also taking the raw SeaMicro technology from AMD radically reshaping it with lots of low-level code to fit the hardware to own purposes.

Kevin Clarke, director of cloud engineering at the telecommunications giant, says that Verizon has actually filed for 30 patents relating to the software it created to run on AMD's SeaMicro systems to make its new cloud.

Verizon started working with SeaMicro more than two years ago, considerably before AMD paid $334 million to acquire the microserver maker in February 2012. One of the things that made SeaMicro attractive to AMD was the Freedom fabric that SeaMicro had created, which runs at 1.28 Tb/sec and implements a 3D torus connection between nodes in a chassis. The ASICs at the heart of the SeaMicro system also virtualize all of the network and storage I/O in the enclosure, which has 64 processor cards, and also does load balancing across those nodes. The latest machines, the SM15000s, using a mix of stock Intel Xeon E3 and custom AMD Opteron 4300 processors, are the ones that Verizon has chosen as the foundation for its cloud. The SM15000s were announced in September 2012 and they can pack up to 64 eight-core, single-socket processor nodes into a 1U chassis; each node can have up to 64 GB of main memory, which is twice what you see on other microserver designs.

The SM15000s are noteworthy because in those machines the Freedom fabric was extended out of the chassis and allows for up to sixteen disk enclosures to be hung off a single chassis, for a total of 1,408 disks and potentially 5 PB of total capacity. Interestingly, Verizon has not loaded up on disks for its Cloud Compute infrastructure cloud, but is instead only using the SM15000 chassis to house flash-based solid state drives. These flash drives hold local data for virtual machines and provide very high I/O operations per second – up to 5,000 IOPS can be guaranteed per VM with Verizon's homegrown controller. They are not intended for object storage.

As Clarke revealed last week, the Verizon Cloud Storage object store, which supports Amazon Web Services S3 and OpenStack Swift protocols as well as the DAV protocol from HTTP web servers, is based on third party disk arrays that Verizon is declining to identify. These arrays have had their own customizations, too.

"We basically opened the kimono and showed them how to hook into all of our intellectual property," says Andrew Feldman, one of the co-founders of SeaMicro and now general manager of AMD's Data Center Solutions Group. "And the result is that all of the software Verizon wrote programs our hardware directly."

Kevin Clarke, director of cloud engineering at Verizon

"The SM15000 is a really cool product, but I want to be clear about one thing," explains Clarke. "I love the AMD guys and they have been an invaluable partner and resource to us in making the thing that we built work. But we gutted their software solution. We used the chassis, but everything else is all native, homegrown Verizon software. We used ATA over Ethernet as a storage protocol and it is how we fling IOPs around. We implemented our own Layer 2 code in the network processing unit, and it turns out that I happen to have some crazy people on my team that can write assembly code for these devices. We implemented a lot of command and control in their MX management card to make this thing happen. We sent them a check for their source code, and then we started to hack on it, throwing out anything that didn't look like what we wanted. It's like this: How do you carve the statue of David? You start with a block of marble, and then you cut away everything that doesn't look like David."

The Layer 2 network that Verizon created for its new cloud is the key secret sauce, and Clarke says it does a number of things.

First, that network defines a fault containment perimeter, and Clarke says that while this is a technical detail that most people don't worry about it too much, they sure do when it fails.

Second, the network stack created by Verizon to run on the SeaMicro Freedom fabric is "ultra-flat," with the goal of allowing the network spanning multiple SM15000 enclosures to be able to link one VM to another one anywhere on the network in a single hop. That network is designed to link up to 250,000 VMs together, which should work out to be around 250 SM15000 enclosures with about 128,000 cores. This is extreme scale by any measure you want to come up with. The SM15000 chassis has sixteen Ethernet uplinks running at 10 Gb/sec to hook enclosures together and to connect them to the outside world. (It would be better still if the Freedom fabric itself could span multiple enclosures, of course, and this is something that AMD is working on for a future product.)

The custom network created for the Verizon cloud also allows it to do packet forwarding and flow control, and in a sense Verizon has therefore implemented its own software-defined network in that cloud.

"We came to the realization that what SDN does is provide a name and a location where any given object is," says Clarke. "Well, guess what? In my orchestration layer, I already know where things are because I put them there, and I already know what the names are because I gave them the names. So I don't need to incorporate a vendor-based SDN solution. I implemented my own."

The other bit of software development that Verizon did for the SeaMicro machines is to converged storage and networking protocols such that it is able to do flow-based rate control and dynamically set performance for machine-to-machine and VM-to-VM communication and for IOPS coming off the flash storage in the SM15000 enclosures.

Feldman did not disclose the financial details of the partnership with Verizon, but said it was a "monster deal."

The Verizon Cloud Compute and Verizon Cloud Storage Services, which will be in beta before the end of the year, are expected to be in production in early 2014.

Incidentally, both AMD and Verizon can envision a scenario whereby Verizon peddles clones of its cloud hardware to enterprises for their own internal private clouds, linking them directly to the public cloud run by Verizon using its own networks. There is no word from either on when this might happen. But the funny bit is that Verizon could end up being a big server reseller for AMD.

Single Sockets Enough?

It is a bit unusual, perhaps, to be basing a cloud on a single-socket server with only a 64 GB memory footprint, considering that that most data centers use two-socket servers with more processing and main memory expansion. Feldman says that for most cloud customers, a core is a virtual machine and even for larger customers, eight cores is plenty of enough oomph for all but the largest customers. Moreover, if Verizon needs for SeaMicro to build two-socket – or even larger – nodes for the SM15000 chassis, it is perfectly willing and able to do that, with either Opteron or Xeon processors.

The one thing that AMD is not going to do is make a SeaMicro processing card based on Intel's new "Avoton" C2000 processor. Feldman did not elaborate, but it is clear that AMD has its own chips in the Opteron and ARM families that it will want to push inside of the SeaMicro machines. While SeaMicro got its start on Atom processors, the business did not take off until Xeon and Opteron chips with brawnier cores were available.