How the NVMe solid state disk protocol evolved

A consortium of vendors, including Intel, Samsung, SanDisk, Dell, and Seagate, developed Non-Volatile Memory Express (NVMe) as a communication interface/protocol specifically for solid-state drives (SSDs). The specification defines a register interface, command set, and features for PCI Express (PCIe)-based SSDs to facilitate high performance and interoperability across all conforming NVM subsystems.

The NVMe specification provides an alternative to the Small Computer System Interface (SCSI) and Advanced Technology Attachment (ATA) standards for interfacing between a host system and a peripheral storage device. The SCSI and ATA standards were developed at a time when hard disk drives (HDDs) and tape were the primary storage media. NVMe was designed for use with much faster media such as SSDs.

Other standards such as Serial-Attached SCSI (SAS) and Serial Advanced Technology Attachment (SATA) have been adapted to SSDs. However, these standards do not convey the main benefits of NVMe-based PCIe SSDs:

• Reduced host software stack latency
• Higher input/output operations per second (IOPS)
• Potential lower power consumption

The NVMe specification supports SSDs that use different types of non-volatile memory, including NAND flash and the 3D XPoint technology developed by Intel and Micron Technology. Supported form factors include add-in PCIe cards, M.2 SSDs, and U.2 2.5-inch SSDs. Figure 1 shows the history of NAND evolvement.

Smaller devices look to M.2

M.2, a form factor specification formerly known as the Next Generation Form Factor (NGFF), is primarily for internally mounted computer expansion cards and associated connectors. Its flexible physical specification allows different module widths and lengths. Together with the availability of more advanced interfacing features, this makes the M.2 ideal for solid-state storage applications in general, and for use in small devices such as ultrabooks and tablets. Computer bus interfaces compatible to the M.2 connector are PCIe 3.0, Serial ATA 3.0, and USB 3.0. The M.2 connector has different keying notches that denote various purposes and capabilities of M.2 devices, preventing inadvertent connection of incompatible devices. In addition to the legacy Advanced Host Controller Interface (AHCI) at the logical interface level, the M.2 specification also supports NVMe as the logical device interface for M.2 PCIe SSDs. While the support for AHCI ensures backward compatibility with SATA devices and legacy operating systems (OSs), NVMe is designed to fully utilize the capability of high-speed PCIe storage devices to perform multiple parallel I/O operations.

 

NVMe SSD applications

The SSD market has been running into some significant hurdles, namely legacy storage buses. Serial ATA and SAS offer plenty of bandwidth for hard drives, but the increasingly higher performance of SSDs is pushing them beyond their capabilities. For example, almost any top-flight SATA SSD will run right up against SATA’s 600Gbps ceiling. Even 12 GBps SAS SSD performance is held back at around 1.5 GBps. SSD technology is capable of much more. Not incidentally, a high-bandwidth bus technology was already in place—PCIe. PCIe is the specification for the underlying data transport layer for graphics and other add-in cards, as well as Thunderbolt, a hardware interface standard developed by Intel (in collaboration with Apple) that allows the connection of external peripherals to a computer. PCIe Gen 2 supports approximately 500 MBps per lane, and version 3.x (Gen 3) supports around 985 MBps per lane. Putting a card in an x4 (four-lane) slot will give you 2 GBps of bandwidth with Gen 2 and nearly 4 GBps with Gen 3. That’s a vast improvement. PCIe 3.x provides a wide enough pipe for today’s fastest SSDs.

Intel partnered with Micron to develop a new form of memory technology called 3D XPoint to achieve high reliability. The Intel Optane SSD 905P, half-height, half-length, M.2 format, which supports PCIe 3.0×4 and NVMe, is able to achieve a high 1.6 million hours of mean time between failure (MTBF) life expectancy. Its memory capacity is up to 1.5 TB (Figure 2).

In February, Western Digital announced Ultrastar DC SN630 NVMe SSD, a 2.5-inch small form factor (SFF) enterprise-grade NVMe SSD optimized for cloud storage. The Ultrastar DC SN630 SSD can be configured to deliver a capacity up to 7.68 TB. Additionally, the company’s CL SN720 NVMe SSD is an NVMe, M.2 form factor SSD supporting up to 2 TB. The 1 TB version has a read/write speed of 3,400 MBps and 2,800 MBps accordingly, comparable with the Samsung’s offering with a slightly faster write time (Figure 3).

M.2 adapter selection criteria

Here are some things to consider when choosing M.2 adapters:

Capacity: How many SSDs can the adapter host? A full-height half-length card can host four NVMe SSDs and have a power dissipation limit of 25 W for a PCIe x16 slot. A half-height card can host two SSDs and lower power dissipation.

Support for PCIe x16 or PCIe x8: Say a GPU takes up 16 lanes, but you still want to use four SSDs. You can overcome the lane shortage if the adapter you choose includes an onboard switch to provide an x8 connection.

Cooling the SSDs: Can the onboard cooling devices, e.g. heat sinks, adequately dissipate the heat generated by the SSDs for your application? You may want to consider a cooling fan if your application involves sustained prolonged sequential write operations. But, if the write operation is in short bursts lasting seconds or a few minutes, then a good heat sink with reliable air flow may suffice.

Signal integrity: The challenge here is that PCIe, using high-speed 8-Gbps serial links, can suffer from a multitude of physical phenomena such as crosstalk, impedance discontinuities resulting in reflections that cause jitter, Intersymbol Interference (ISI), and mode conversion due to unbalanced transmission lines—all of which can lead to excessive electromagnetic interference (EMI) emissions in large systems. Manufacturing processes ensuring the high signal integrity of PCIe Gen 3 serial channels are critical if high-speed serial interfaces are to deliver on their promise of greater performance. Signal integrity analysis and compliance testing must be rigorously implemented in production and QA processes to accomplish that goal.

Availability of standard drivers/tools

High-end SSDs had been made using the PCIe bus before NVMe, but the interfaces were non-standard. Standardizing the interface of SSDs to the NVMe specification makes it possible for operating systems to use one driver to work with all conforming SSDs. For example, Windows 10 has its own driver for NVMe SSDs. Drivers are hardware dependent and operating-system-specific. They usually provide the interrupt handling required for any necessary asynchronous time-dependent hardware interface.

One of the best things about NVMe is that you don’t have to worry about driver availability. Linux has had NVMe support since kernel 3.1; Windows 8.1 and Server 2012 R2 both include a native driver, and there’s a FreeBSD driver in the works. When Apple decides to support NVMe, the latter should make it easy to port. However, BIOS support is largely lacking. Without an NVMe-aware BIOS, you can’t boot from an NVMe drive, although anyone with an x4 PCIe slot or M.2 connector can benefit from employing an NVMe drive as secondary storage. An NVMe BIOS is not a big deal, but it does require effort and resources. More abundant and prevalent support has to be there. If you intend to make an SSD boot device, you would need:

• A recent system, such as the Xeon D-1540 or Skylake chipsets (2015 or later).
• Modern OSs that have the NVMe driver built in, like Windows 8.1/10 or Windows Server 2016.
  •  If the OS-provided driver is slow, replace it with a vendor-provided driver once OS is installed.
• An M.2 PCIe x4 (or higher) slot (as in four lanes, key to obtaining the maximum speeds).
• BIOS in UEFI boot mode (A nice bonus here is that it allows your bootable device to be larger than 2 TB, since Windows will use the GUID Partition Table (GPT) rather than the legacy Master Boot Record (MBR) mode.
• Compatibility Support Mode (CSM) to be turned on (as required by some motherboards).
• BIOS upgrades and/or BIOS configuration changes (as required by motherboard manufacturer).
• Optional M.2 NVMe SSD PCIe adapter card for multiple SSDs.

Newer, high-end motherboards will get firmware upgrades to support NVMe so one can boot from the drive.

Samsung offers two models with built-in controllers that support PCIe Gen 3.0 x4 and NVMe 1.3 in a M.2 format. They support storage capacity up to 1 TB (970PRO) and 2 TB (970VRO). The read speed is 3,500 MBps for both models, while the sequential write speed is 2,500 MBps and 2,700 MBps accordingly (Figure 4).

Note that NVMe SSDs are nearly seven times faster than SATA SSDs. A single NVMe M.2 SSD can provide read speeds as high as 3,500 MBps. To maximize the SSD performance, one may want to select a M.2 adapter separately. For example, one of Aplicata’s expansion adapters, the Quattro 400 PCIe x8 model, provides a read speed of 7.2 GBps when combined with four 1 TB Samsung 970PRO NVMe SSDs. At this speed, it is close to the PCIe Gen 3 x8. These inexpensive adapter cards will fit in the PCIe slots if the motherboard lacks a M.2 connector (Figure 5).

Conclusion

NVMe is a communication interface/protocol developed specifically for SSDs by a consortium, while M.2 is primarily for internally mounted computer expansion cards and associated connectors. SSDs come in pre-configured controllers. To achieve higher read/write performance, one may consider the option of integrating SSDs using separate M.2 adapters. In selecting an adapter, consider the supported storage capacity, whether it supports PCIe x16 or x8, the cooling of the SSDs, signal integrity, and, finally, the availability of standard drivers and tools.