SoftLayer To Outflank Rivals With Bare Metal, InfiniBand, And Power8 

There are many ways for IBM to get to the $7 billion revenue run rate it has promised Wall Street it can hit as it exits 2015. One way was to take its existing application hosting services and merge them with a true cloud provider, and then make more acquisitions from there to quickly build up its book of business. With its $2 billion acquisition of SoftLayer in June 2013, IBM did the first half but has not done the second. And probably will not for sound economic reasons.

With cross-state rival and OpenStack founder Rackspace Hosting shopping itself around, IBM could double its cloud server fleet and add another $1.5 billion or so to its cloud revenue stream, and add hundreds of thousands of customers by acquiring Rackspace.

Such a move would make Big Blue a serious contender in both bare metal and virtual clouds. It would also make IBM the key developer behind the OpenStack cloud controller, and also an important member of the Open Compute Project open source hardware movement started by Facebook. The latter is an interesting option to consider given that IBM is trying to sell off its System x X86 server business to Lenovo Group for $2.3 billion. IBM is going to shift from being the third largest producer of servers in the world to one of the largest consumers of such machines among enterprises outside of the largest hyperscale players (Google, Amazon Web Services, Microsoft, and Facebook are the biggies). But with only $131 million in earnings and a market capitalization of $4.4 billion, Rackspace is probably too expensive for Big Blue's current budget. Although, to be fair, SoftLayer was running at around a $364 million run rate when IBM bought it for $2 billion last year, which is about 5.5X times revenues, and it could probably pick up Rackspace for as low as $5 billion, and that would work out to about 3.3X times revenues.

Instead of buying Rackspace and adding it to SoftLayer, IBM decided it was cheaper to merge its own fifteen hosting centers with the thirteen datacenters already owned by SoftLayer and then spend another $1.2 billion building these out and adding a dozen new ones until it has 40 datacenters globally linked by SoftLayer's private network and using its own bare metal and cloud controller software. When IBM bought SoftLayer, it had about 21,000 customers, and in the past year it has added another 6,000 more, many of them large enterprises from the IBM systems customer base who are looking for cloudy infrastructure. SoftLayer was a pioneer in bare metal clouds and in getting server nodes equipped with GPU accelerators, and bare metal still accounts for a large portion of its business. Microsoft, Google, and Amazon Web Services do not offer bare metal infrastructure services, which come with hourly billing rates and fast provisioning just like virtual server slices.

That IBM customer base, which is probably on the order of 400,000 strong, is the key asset that is going to grow the SoftLayer business, provided IBM can line up the right infrastructure and platform services at the right price. IBM seems to have had an epiphany moment in 2013 and realized that it needs to build clouds and offer software services on top of them if it hopes to protect its revenue streams; it also seems to believe it doesn't need an X86 server business as much as it needs a cloud business.

The cloud build-out is progressing as planned, says Lance Crosby, CEO at SoftLayer, which is officially a subsidiary of IBM that is tucked under its Global Services behemoth.

George Karidis, chief strategy officer at SoftLayer, tells EnterpriseTech that three new datacenters have opened already in Hong Kong, London, and Dallas, and three more are in the works in Toronto, Melbourne, and Washington DC to open in the third quarter. The new center in Dallas, where SoftLayer had its largest datacenter before the IBM acquisition, and the one in Ashburn, Virginia, outside of the US capital are both going to have FedRAMP certification, which makes it easier for government agencies to deploy there, as well as certification under the Federal Information Security Management Act (FISMA). The idea is that the Dallas and Ashburn sites will be tied together, offering government agencies disaster recovery and load balancing across these two facilities, all within a secure, internal network.

This private network, which was built four years ago by SoftLayer, is a key differentiator for what is now IBM's public cloud, says Karidis. What it means is that customers who deploy servers and storage on the SoftLayer cloud can automatically use this network, which has hundreds of gigabits of traffic coming into every datacenter and two public and two private network cards running at 10 Gb/sec running inside of every server. The private network interfaces are used to hook into the SoftLayer backbone and the two public ones can reach to the outside world of the Internet, and they do not mix. There provisioning, patching, monitoring, and management services running on SoftLayer infrastructure use this private network, and servers link to each other and to outboard storage services through it as well.

"This is not just a carved out virtual private network as you see on other cloud services," says Karidis. "This is actually a dedicated link. And that means you do not have to use the public Internet to get at data that lives in a SoftLayer environment. You just go to the closest point of presence and then link right in, and so we see anywhere from 30 to 40 percent latency improvements because we eliminate all of the public hops." Any cloud provider that wants to add a private network link SoftLayer has would have to retrofit servers with multiple network ports and also build a whole secondary network and link it across all of their datacenters. Google, Amazon, and Microsoft have the private pipes, but as far as anyone knows, do not have public and private interfaces into all of their servers.

The Direct Link service that SoftLayer announced back in June also uses points of presence (POPs in the lingo) in telecommunications hotels to link private datacenters to this internal SoftLayer network, allowing for a seamless connection for hybrid datacenters – but one that is not touching the public Internet in any fashion. These network centers are where SoftLayer also links to all of the telco operators, cable companies, and other network service providers to reach the outside world.

SoftLayer has deployed 56 Gb/sec InfiniBand switches and put InfiniBand adapters into servers in its Houston datacenter as well as a few others. With a direct 10 Gb/sec drop into the SoftLayer network, data transfers between supercomputing centers and other commercial datacenters that run parallel applications are less hampered by the data transfer between those centers and the SoftLayer cloud. InfiniBand, says Karidis, has some scalability limits and can only span about a row of server racks (somewhere around a dozen) before it becomes complicated to deal with. Moreover, says Karidis, because InfiniBand is "basically proprietary technology" it does not play well with others. There are things that need to be done with InfiniBand to make it aware of multi-tenancy, and SoftLayer is working with customers who are already familiar with this low-latency switching option to develop the compute service. Karidis says that given that its backbone is Ethernet, SoftLayer would just as soon use 40 Gb/sec Ethernet, but that some customers are asking for InfiniBand, which is good news for Mellanox Technologies. AWS and a number of other clouds have InfiniBand options for their clouds, and Microsoft uses InfiniBand internally on the Bing Maps service and very likely in other places.

"InfiniBand is available as a proof of concept for certain key customers," says Crosby. "We are exploring options to roll it out to a broader customer set within the next two months or so."

On the systems front, the SoftLayer fleet has held steady at about 120,000 machines since last May, although the company is still replacing old iron with new stuff. SoftLayer makes its own hardware decisions separate from IBM, and is still sourcing its systems and storage from Supermicro, and there is no plan to change that at the moment. The plan is to double the fleet in those expanded 40 datacenters, and it looks like Supermicro can count on getting some big orders. With Intel expected to launch the "Haswell" Xeon E5 v2 processors for two-socket servers sometime this fall, most likely at Intel Developer Forum in early September, it seems likely that SoftLayer is waiting for the shiny new chips to build out the fleet.

SoftLayer is also figuring out how to deploy Power Systems iron in its cloud, and now that Power8 machines support the KVM hypervisor, the OpenStack cloud controller, and Ubuntu Linux as well as running stock Linux from Red Hat and SUSE Linux, it is much easier to weave Power-based systems into the SoftLayer cloud. The Watson question-answer service, which is a mix of code that rides on top of Hadoop, is running on Power system slices in select datacenters in the SoftLayer cloud. A select number of unnamed customers are also running slices of Power8 systems as proofs of concept. But in the long run, says Karidis, Power machines will become peers of X86 iron in all of the SoftLayer facilities.

"I can see a Power Systems cloud that is delivered on a global basis in our datacenters that is as popular as an X86 cloud," says Karidis, looking a few years down the road.

What the research and development team at SoftLayer is looking at is how to take Power-based machines and turn it into what Karidis calls "a fully composable hardware cloud." The futuristic idea is to build Power-based systems in such a way that it is not constrained by networking and racking, and has main memory separate from compute and separate from storage, and allows for systems or clusters to be composed on the fly, through software, as workloads dictate. Because IBM controls the entire Power stack – and has partners that will work with it on the Linux, KVM, and OpenStack front as well – this is something that can be attained, much as Intel is striving to do with its RackScale Architecture and silicon photonics interconnects for server components.

SoftLayer is also keeping an eye on more lightweight virtualization technologies such as LXC containers and Docker containers, according to Karidis, who says he likes the concept behind this technology. (It is hard to ignore anything that has the backing of both Google and Amazon Web Services.) SoftLayer is also keeping an open mind about the possibilities of deploying 64-bit ARM servers on its cloud, and has some experimental machines in its labs just so it can keep on top of this technology as it evolves.