Flash Spreads Around The System, Accelerating As It Goes 

UlltraDIMM memory channel storage, which puts flash memory in the same slots as main memory in servers, is starting to ship from IBM, and its advent has a lot of companies – both users and suppliers – mulling over their flash storage strategies. But don’t think for a moment that UlltraDIMMs are suddenly going to obviate the need for flash cards that hang off the PCI-Express bus. Or, solid state drives that hang off disk controllers or disk drives themselves for that matter.

The excitement around UlltraDIMM memory, which SanDisk has created in partnership Diablo Technologies. The memory channel storage puts 200 GB or 400 GB of flash memory on a DIMM form factor, and Diablo has come up with a chip that can take the SATA protocol coming off the flash chips and convert it to the DDR memory protocol that is used to link main memory to on-chip memory controllers in modern servers. SanDisk has created drivers for Linux that allow this flash DIMM memory to be addressed like any other storage device in the system, and for applications where latency is a key issue – think financial services, for starters – the 5 to 10 microsecond latency on writes is going to be a big attraction.

But neither SanDisk nor Fusion-io, the dominant supplier of PCI-Express flash cards, think that UlltraDIMM spells doom for other kinds of server-attached storage.

“The one thing I have learned over my long career is that media types and media connection points come and go,” Gary Orenstein, senior vice president of products at Fusion-io. “It typically takes longer for them to arrive or to go than anybody ever expects. And so there is no need to get crazy when something shows up and there is no need to declare the death of any media type either because these are elongated waves. We do think that these new efforts are really interesting in highlighting what lies ahead.”

Fusion-io is taking the long view. For years, in the mainframe and then the server business, there was a very hard split between persistent media and more ephemeral media such as main and cache memory. Persistent media spun or whirred and had physical properties as a component of data latencies – such as disk rotation and seek times for disk drives or the time it took to load a tape reel or cartridge and then roll through the tape to find a particular bit of information. Non-persistent memory storage was orders of magnitude faster and you could address data directly, but it was expensive and when the power went off, the data was lost. Flash has come along to become a combination of the two--it is persistent storage but one that can (but often still does not) have the access semantics of main memory.

For right now, though, the low latency on writes is what has server makers and their customers – or those who build their own machines, for that matter – intrigued by UlltraDIMM. But there are other benefits, Steve Fingerhut, vice president of enterprise storage solutions at SanDisk, explains to EnterpriseTech.

“The number one value that UlltraDIMM brings is that it is on the DDR memory bus, and that means extremely low latency, around 5 microseconds for writes,” says Fingerhut. “Call that the flashing billboard on UlltraDIMM. But the other thing is the scaling. The DDR bus has so much performance capability that as you add UlltraDIMMs, the scaling is nearly linear and the latency stays flat. And that means applications can start treating it more like memory.”

SanDisk and its sole server partner thus far, IBM, have not released performance figures for read latencies. There are techniques that can be used to goose writes on flash, but reads tend to be a function of the flash memory itself and it is harder to speed up. “You are going to see read latencies being more standard across vendors, and the applications are less sensitive to that,” says Fingerhut. “Databases in particular are gated on writes; they cannot do a read until a write is committed, so writes become the bottleneck.”

The obvious question is, why would you use anything else?

“There are form factor issues that drive people to different interfaces,” Fingerhut continues. “It may be that your system uses all of its DDR slots for main memory because that is the best for the application, so you move flash to PCI-Express. Or, you may want to use all of your server bays for hard drives and then add a flash layer with PCI-Express. There is no normal in a way – everybody has their own unique situation. Our model is to have all of the options on the table so customers can pick and choose for their given application.”

The primary business at SanDisk is selling SAS solid state drives, and it has two families. The Lightning family are high-end products commonly added to servers and storage arrays, while the Optimus line came through its acquisition of Smart Storage Systems last July. (Smart Storage was also the company that had licensed chips from Diablo Technologies that converts the SATA interface on flash chips to the DDR3 main memory protocol so that flash can be put on regular DDR3 sticks.) The Optimus flash SAS drives are optimized for providing the best bang for the buck, while the Lightning drives focus on the bang; they range in size from 200 GB to 2 TB. SanDisk also got a line of SSDs based on SATA interfaces called Cloudspeed, which have less I/O performance but the lowest cost per unit of capacity; these range in size from 200 GB to 1 TB.

“As for PCI-Express, we are between generations on that,” says Fingerhut. “This is a highly fragmented market, so you have to find the right product for the right segments. We are still finding the right solution given the evolving market.”

Fingerhut separates the flash market into two groups – those who know exactly how the storage will be used, and those that don’t know how it will be used.

“What hyperscale customers want is different from what the server and storage OEMs want,” he says. “The hyperscale customers know exactly how they want to use it, and they are very open to anything that solves their exact problem. Server and storage OEMs cover a broad range of products, so they need an architecture that can do everything reasonably well. Trying to address all of these needs with one product is something nobody in the market has accomplished yet.”

Even within hyperscale customers, the market is highly fragmented, says Fingerhut. Some want flash underneath a traditional database application, and they have a need for very low latency and ACID requirements – something that is rock-solid and they care a lot about reliability. Others use a more modern cloud architecture that has redundancy in the software, and they are doing more coalesced writes and read latency is their number one criteria. Some customers are also looking at cost trade-offs: They are willing to accept a little bit lower performance for a significantly lower price.

UlltraDIMM will have a natural place where applications and databases require high throughput and low latency writes. Until the read performance statistics and pricing are available, it is not completely clear how it will stack up to PCI-Express flash cards.

SanDisk has an exclusive relationship with Diablo Technologies for that SATA-to-DDR conversion chip at the moment, and is not discussing the nature of that deal. SanDisk has not disclosed other server makers it will be partnering with, but Fingerhut says to expect “broader platform adoption through 2014.” The UlltraDIMM technology will have a long life with DDR3 interfaces in four-socket, eight-socket, and larger machines, and when DDR4 memory is available on two-socket machines, SanDisk will be ready with a DDR4 variant of the UlltraDIMM.

At the moment, UlltraDIMMs come in 200 GB and 400 GB capacities, and the MLC NAND flash chips on the units have a duty cycle that can sustain ten full writes per day over five years. While SanDisk is not talking about future capacities on the memory channel storage, it is reasonable to expect that 800 GB and 1.6 TB units are in the works that will track the density of regular SAS SSDs.

Memory maker Micron Technology is working on a hybrid flash-main memory module called NVDIMM, which puts DDR3 main memory and flash backed up by an ultracapacitor to provide data persistence for main memory. This is a slightly different animal from UlltraDIMM. NVDIMMs are available for DDR3 interfaces today and this year Micron is expected to move on up to DDR4 interfaces with this hybrid memory stick.

Fusion-io is not saying if it will be developing a DIMM-based flash memory product, but Orenstein does offer some context to show what the company thinks the issues are for accelerating workloads with flash and any other possible persistent storage media that comes along years hence.

“The design intent behind Fusion-io is not focused on flash and it is not focused on PCI-Express,” explains Orenstein. “It is focused on how we present memory as memory in new and unique ways and how we provide a set of services around that memory, starting with persistence and all of the things that persistent memory needs – power cut failure protection, guaranteeing availability, and so forth. The best way to do that a few years ago was to put flash natively on PCI-Express and, because companies are not going to change their applications overnight, present it as a block storage device while we put ourselves in position for the actual convergence of I/O operations and memory operations. Long term, we believe the world wants more memory-like operations for storage persistent services. And the offerings we see in the market now don’t meet that need. They simply provide another access point with memory viewed as storage. Or they provide an access point to memory viewed as memory – maybe it is not memory but something like flash – but it doesn’t have a set of persistent services.”

SanDisk is thinking along similar lines. Back in January, when IBM’s System X6 machines were launched with the eXFlash-branded UlltraDIMMs, Brian Cox, senior director of marketing at SanDisk, told EnterpriseTech that the company was working on a version of UlltraDIMM that would look like main memory to the server BIOS rather than a storage device. It is not clear if that memory-style flash will have the kind of persistent services that Fusion-io is talking about as its ultimate goal for flash and any other kind of memory-like yet persistent storage that comes along.

Time will tell. One thing is for certain: These developments, and many others, are going to be interesting in terms of accelerating workloads.