IBM Goes Modular And Flashy With X6 Systems 

IBM, like other system makers, is eager to start selling new machines based on Intel’s forthcoming “Ivy Bridge-EX” Xeon E7 v2 processors. So eager, in fact, that it is rolling out a new high-end design for its System x rack machines and PureFlex converged systems ahead of Intel’s launch of its most powerful Xeon engines.

Because of this, IBM is being cagey about the fact that the new X6 systems being previewed today support the Xeon E7 v2 chips. But trust EnterpriseTech. They most certainly do. In fact, says Alex Yost, vice president in charge of IBM's System x, Blade Center, and PureFlex systems, the new X6 machines have been given a modular design that will allow them to support the next several generations of high-end Xeon processors from Intel.

This is important for both technical and economic reasons. Technically, customers don’t want to have to rip and replace systems each time there is a new processor out from Intel. Moreover, the accounting rules that govern what you can call a system upgrade are based on how much of the system changes, and if you do an upgrade, you don’t have to write off the remaining undepreciated portion of the system as you have to do with a box swap. Because the serial number is attached to the enclosure, as long as you don’t change too many components, you can keep depreciating the system on the normal three-year or five-year schedule that your accountants use. This is distinct from prior X3, X4, and X5 generations of System x machines that were based on Xeon MP, Xeon 7500 and Xeon E7 v1 processors. These machines were not particularly modular.

If Intel doesn’t radically change the chipsets for the Xeon E7s in coming years, it may even be possible for IBM to mix and match different generations of Xeon E7-class processors inside the same X6 enclosures. (IBM was mum on this possibility because it would disclose too much about Intel’s Xeon E7 v2 launch and future processor and chipset roadmaps.)

IBM has worked with Intel on six generations of extensions to the Xeon processors and their related chipsets, and has invested hundreds of millions of dollars since the 1990s to bring mainframe and Unix features to X86 iron. With the X4 generation several years back, when Intel did not support eight-socket systems and its four-socket chipsets were not as efficient as they now are, IBM created its own chipsets for lashing together multiple Xeon processors into a fat memory system. With the X5 machines three years ago, Intel’s chipsets improved in terms of CPU scalability and I/O bandwidth, but IBM extended the memory bus to support more main memory than the Intel chipsets did by themselves.

"The fit for purpose, modular design of the X6 systems enables a bunch of good stuff," Yost tells EnterpriseTech.

The first rack-based X6 machine is called the System x3850 X6, and it comes in a 4U chassis that allows for one storage module and up to four processor modules to be slid into the enclosure. The System x3950 X6 is a double-decker setup that essentially glues together two System x3850 X6 machines with a NUMA connector to create an eight-socket system. Each compute node in these machines has a single processor socket and 24 memory slots, for a maximum of 1.5 TB of main memory using 64 GB DDR3 main memory sticks. Each compute node, which IBM calls a processor book, slides into the enclosure and uses the QuickPath Interconnect (QPI) links on the processor to hook into a backplane that has the Intel chipset (in this case, the C600J chipset) to create a single system image from the nodes. The shared memory coupling of the processors allows for a single instance of an operating system to span all of the compute, memory, and I/O in the system, bringing all of these resources to bear for applications that require more capacity than is available in a workhorse two-socket Xeon E5-2600 v2 system.

Specifically, these new X6 systems are aimed at supporting databases, analytics, ERP and related back-office software, and fat virtualized servers where customers want to be able to have very large virtual machines or cram lots more virtual machines onto a single box. It stands to reason that the X6 machines will also see some use as fat memory nodes in clusters to support various kinds of simulations. It would have been interesting to see IBM push the scalability limits even further, perhaps with an X6 system with 16 or 32 sockets – something that the company is perfectly capable of creating – but Big Blue wants to sell mainframe and Power Systems machines to customers who have such scalability needs.

By the way, IBM has used this processor book approach to Power Systems and System z designs for years, but it does not allow for the mixing and matching of multiple generations of processors in the same system. It is technically possible to do so, but you have to synchronize the components to the slowest common speed to make it all work. Sun Microsystems did this with UltraSparc-III uniboards in its midrange and high-end systems a decade ago, for instance, which made it easier for customers to upgrade their machines piecemeal instead of doing a box swap.

The feeds and speeds of the two rack-based System x X6 servers were not divulged, but we do know a few things more about them. Each X6 enclosure comes with a storage book, which can be equipped with one or two storage controllers; it can have up to eight 2.5-inch SAS drives for a total of 12.8 TB of capacity or up to sixteen 1.8-inch solid state drives (SSDs) for a total of 6.4 TB of capacity.

As is always the case, IBM has some additional goodies it is putting into its machines that provide additional reliability and resiliency beyond that offered in the Intel Xeon and chipset combination. The X6 machines include double chipkill memory scrubbing, which heals around bad bits in main memory without crashing the system, and what IBM calls advanced core recovery, which isolates a failed core, and automated processor failover, which fails over the I/O from the chipset from one socket to another in the event that the processor in the first dies for some reason. These and a number of other features are implemented in firmware or in the backplane of the system.

Flashy Memory – Literally

As EnterpriseTech expected, the new X6 machines also have what IBM calls eXFlash memory channel storage, which puts very fast flash memory on a DDR3 memory stick to get it closer to the processor and running on the memory bus. IBM did not want to reveal the source of this eXFlash, but Brian Cox, senior director of marketing at SanDisk, confirmed to EnterpriseTech that SanDisk was the source of the parts.

The memory channel storage modules were developed by SanDisk in conjunction with Diablo Technologies, and are called UltraDIMM by SanDisk. The modules put flash memory created by SanDisk (which has a flash partnership with Toshiba) that has a SATA interface on a memory stick. Diablo has created a chipset that converts the SATA protocol to the DDR3 main memory protocol, and SanDisk has created a driver for a server BIOS that makes it look like a normal disk storage device to the system and its operating system. SanDisk also has created the wear-leveling algorithms to give the flash memory a long lie in the system. The UltraDIMMs have a 5 microsecond read latency, which is approaching the speed of main memory and which is many times faster than flash memory linked to the system over the PCI-Express bus. It is not clear yet, but the UltraDIMMs should burn less juice and radiate less heat than main memory as well.

The Flex x880 X6, System x3950 X6, and System x3850 X6 servers

The UltraDIMMs come in 200 GB and 400 GB capacities, which are much denser than main memory sticks and which are expected to be considerably less expensive per gigabyte. Cox says that pricing has not been set yet, but the price has to be a lot less than the cost per gigabyte on a 32 GB or 64 GB DDR3 module for it to be worth the effort. Right now, SanDisk is working with BIOS suppliers Phoenix Technology and American Megatrends to get the UltraDIMMs supported as well as server partners to get the flashy DIMMs certified to work in their machines.

IBM has been handing out two-socket servers using the UltraDIMMs for Wall Street firms to test for the past several months because for them, low latency is as important as being able to hold lots of data in a system. On a 24 slot server node, a customer might have eight slots populated with main memory (perhaps using relatively inexpensive 8 GB or 16 GB sticks for 64 GB or 128 GB of capacity) and then put a dozen of the 200 GB UltraDIMMs in the system for data storage (for 2.4 TB of local storage). The server might only need a pair of 2.5-inch or 1.8-inch drives to store the operating system and application software, and might not even have any spinning disk at all.

“Wall Street loves memory channel storage,” says Cox. “And we have all kinds of hedge funds begging us to deliver this.”

Further down the line, Cox says that SanDisk is working on a variant of UltraDIMMs that will look like main memory to the system. This will be particularly useful, he says, for loading up larger in-memory databases than is currently possible, because of both capacity limits and the relatively high cost of memory.

IBM is keen on the UltraDIMMs for this reason, and because it has the jump on the competition. "It is always difficult for us to say precisely what advantage we will have over our competitors, but we expect to have a multiple quarter advantage," says Yost, referring to the eXFlash-branded UltraDIMM memory channel storage.

The X6 systems are not just taking the UltraDIMMs from SanDisk and plugging them in. IBM has a number of add-ons that weave them into the system. The company’s Flash Cache Accelerator, which learns what data is hot, warm, and cold in a system and automagically moves it to the appropriate storage tier, knows about the flashy DIMMs. Another feature called WriteNow commits data to the flash DIMM faster than the default drivers, dropping latency by another 50 percent or so. (How it works is a trade secret.) Data Direct Accelerator allows for software engineers to turn interrupts or polling-driven operations on and off in the system to do custom tuning for applications that hit the flash DIMMs.

IBM fully expects that for many workloads, customers will go all-flash in the X6 systems for both performance and price/performance reasons. Companies massively overprovision their storage arrays, as they have done for decades, so they can get acceptable performance, and it is not uncommon for arrays to be held at 50 percent of capacity or lower. In many cases, companies dig deep into the arrays and “short stroke” the drives, which means putting the data in the outside of the disk and leaving the capacity on the inside of the disk empty. (The outer edge of the disk spins faster under the disk read/write heads, which boosts I/O performance, and for a given unit of capacity it takes the disk arm less back and forth swinging to locate a track, which boosts seek time.) This is a nuisance.

While IBM is not providing an array of performance stats or pricing for the X6 rack servers, Yost will provide this tidbit for a comparison. "You will see, X5 generation to X6 generation, much better performance, especially for I/O hungry applications; something on the order of 46 percent. And customers are going to be able to save 43 percent by moving from an X5 with spinning drives to an X6 with our eXFlash memory channel storage."

The Xeon E7 v2 chip will top out at 15 cores, 50 percent more than the current ten-core Xeon E7 v1, and also has three times the memory capacity per socket, so a 46 percent performance increase is expected between the X5 and X6 server generations, and more for memory-bound workloads. This is a like-for-like comparison with spinning disks. IBM is still doing the performance testing on the eXFlash modules.

That X5 to X6 price comparison above includes fees for both hardware and software for the systems. Further details were not available. Yost did toss out another interesting statistic. A lot of customers, particularly in the Asian markets, are buying four-socket Xeon machines but only putting two processors in them. This gives them a way to boost the performance if a node radically without having to swap out the box. Compared to a machine that puts four sockets on a single system board, the modular X6 machine has a 25 percent lower cost with two nodes in it compared to a half-populated four-way system.

The third member of the X6 family is the Flex x880 X6 node, which is available for the PureFlex converged systems. This Flex x880 V6 node is a double-width server module that has two Xeon E7 v2 sockets, 48 memory slots, and two disk bays. Two or four of these nodes can be lashed together with a NUMA link inside of the Flex System chassis to create progressively larger shared memory systems.

Yost says that IBM has seed System x3850 X6 machines out at early adopter customers now for final testing and that this machine will ship in the first quarter in the wake of Intel’s launch of the Xeon E7 v2 processor. The Flex x880 X6 nodes will be ready for sale in the first quarter as well. IBM is starting to place System x3950 X6 machines with beta customers now, and these boxes are expected to be generally available in the second quarter.