Flashtec NVRAM Does 15 Million IOPS At Sub-Microsecond Latency
Hyperscale datacenter operators and enterprise storage suppliers just got a new tool from PMC to help them build out zippier storage infrastructure. PMC, a venerable semiconductor maker founded three decades ago to provide equipment to the telecommunications market, moved into the enterprise storage market seven years ago and has just put together a non-volatile memory card based on a mix of DRAM main memory and flash that can crank through a whopping 10 million I/O operations per second.
That’s about ten times faster than a PCI-Express card based on NAND flash all by its lonesome, and that performance is going to capture the attention of many companies that want to accelerate their applications.
The new NVRAM card is called Flashtec, and it is based on PMC’s own controllers, which support the NVM Express (NVMe) protocol that has been recently announced to boost the performance of flash memory by getting the operating system kernel and drives out of the way as CPUs and main memory try to address flash devices in a system. PMC lunched its first 6 Gb/sec SAS/SATA and 8 Gb/sec Fibre Channel controllers back in 2007, when after it had acquired Adaptec, and added 12 Gb/sec SAS in 2012.
In that same year, the company added flash and NVRAM controllers to its lineup that supported the NVMe protocol, which cuts the latency for I/O operations and the amount of compute required to do them by more than half compared to regular SAS protocols. The these controllers can have up to 8 TB of flash memory attached to them and drive more than 875,000 IOPS of random reads using 4 KB file sizes. The come in two flavors, on that has 16 channels and is aimed at PCI-Express x4 slots and another that has 32 channels and is aimed at x8 slots. PMC also sells SAS host bus adapters and SAS/SATA RAID controller cards, too, which launched last year. PMC had hardly any share of the SAS ports in servers back in 2008, and now it has greater than 50 percent share, Roger Peene, senior director of product marketing at PMC, tells EnterpriseTech. But the big news is at the Flash Memory Summit this week is that PMC was taking the NVMe flash controllers, which were the first ones delivered into the market, and creating the world’s first NVRAM cards that supported the NVMe protocol.
The Flashtec NVRAM drives are PCI Express cards and are distinct, says Peene, from NVDIMM alternatives, which put flash memory onto modules that plug into a server’s main memory slots, in a number of ways. First, NVDIMMs require BIOS support, pull extra power from the memory slots, and jockey with applications for processor cycles and memory bus bandwidth. NVDIMMs are also closer to the processor, and given than the memory bus is much faster than the PCI Express bus, there are those who think that NVDIMM is the way to go. But the NVRAM hybrid that PMC has come up with mixes the best of both worlds and allows for the device to be addresses as either memory or block storage.
This is the really interesting bit, and it allows for the Flashtec cards to be used in a number of different ways. Using block-based access through the NVMe interface, applications use storage semantics – open, read, write, and so on – to access the DRAM on the card in fixed blocks, just as they would access flash or disk storage. In a separate and faster memory-mapped access mode, the card has a direct memory interface with 64-bit memory mapping and addressing and applications use memory semantics – load and store – and DMA engines to access data stored on DRAMs in the card. In either case, the data stored on the DRAM is always backed up immediately to the flash memory.
If you want to use the Flashtec as a write cache, you want to configure it as a block device, but for updates to metadata or database records, then the memory mapping approach is more natural, says Peene. And it is also very fast.
In memory mapped mode, with random 4 KB reads the Flashtec can handle about 4.3 million IOPS peak and can do about 4.6 million on 4 KB random writes. (Having read and write performance that is almost the same is something you can accomplish with DRAM but not with flash alone.) That is about 10X the performance of solid state drives on the reads. If you are focused on writes for those metadata and database workloads, then on 64 byte random writes, the Flashtec can deliver 15 million IOPS, which is about 100X what an SSD can deliver, according to Peene. (The SSDs can do somewhere between 50,000 and 100,000 IOPS.) In block mode, the Flashtec NVRAM drive can do about 1 million IOPS with 4 KB file sizes.
You can partition the Flashtec device and run it in memory mapped and block modes at the same time. Here is how NVRAM fits into the storage hierarchy in terms of latencies:
Flashtec NVRAM cards have 4 GB, 8 GB, or 16 GB of DRAM memory on them. PMC is using Micron Technology as the source for both its memory and flash chips for the device, and is lining up second sources of components. PMC is not being specific about how much flash memory is on the card, but says that it has at least twice the capacity of the DRAM on the card so customers can make at least two copies of the data residing in the DRAM. Peene says that the upper end of capacity needs for both database and metadata caching is 16 GB, so there is no need to try to put 1 TB or 2 TB of DRAM and flash onto a card. (Although this would be a fun engineering exercise and would also probably find a use somewhere. . . . ) It takes under 30 seconds to back up the DRAM to flash and it can handle more than 5,000 backup cycles; it takes less than 60 seconds to restore the data from flash to DRAM.
The NVRAM card from PMC is meant to address the fundamental shortcomings of both DRAM main memory and flash memory. DRAM is not persistent, so when the power is out, the data residing in it is gone. Flash memory is non-volatile, meaning it holds its data for a number of months without any power being supplied to the device, but the flash memory cells wear out with each write cycle. Moreover, flash wears out over time and you have to overwrite sections of flash before you can put new data down on it, which accelerates the wear. Flash SSDs also have an upper limit of about 10 drive writes per day (DW/D), and what enterprises really need, according to Peene, is something between 50 and 1,000 DW/D. Flash memory can’t do that, and until some other kind of non-volatile memory comes to market that can, PMC thinks a mix of DRAM backed up by flash and sporting a supercapacitor module to allow data to be backed up from memory to flash in the event of a server power failure is the way to go. The supercapacitor allows for server makers or the hyperscale operators who design their own machines to create systems that do not need uninterruptible power supplies or battery backups for main memory for mission critical workloads, and this can remove a lot of cost from a rack of machines.
Any application that already has a flash tier in it is a candidate for acceleration from a Flashtec drive. This includes using it as a metadata store in a scale-out file system or as a caching tier in an all-flash array. Journal logs for in-memory databases could also use the Flashtec to boost their performance more than is possible with SSDs or PCI-Express cards based on flash alone.
The Flashtec NVRAM card slides into a PCI-Express 3.0 x8 slot and has a low-profile form factor; it is supported by Red Hat Enterprise Linux 6.5 and 7.0 out of the gate, and clearly other operating systems will follow. PMC says that the Flashtec card will be priced competitively with other NVRAM products on the market, but Peene is a bit vague (on purpose) about the price, saying that the 16 GB variant of the card will cost somewhere between $1,000 and $2,000. The Flashtec cards will be available in the fourth quarter, and are sampling to early adopter customers now.
