7 ms·
NVMe, the fast future for SSDs
- seatonist 12y agoI can remember putting ISA "hardcards" in my 286s and similar. Full circle!
- jjoonathan 12y agoI'm just happy that for once we got to see standards converge (SATA PHY ditched for PCIe) rather than proliferate. BTW, does anyone know if NVMe uses the ATA command set for side channel stuff like configuring encryption and the like?
- justincormack 12y agoNo I dont think so, it has a new command set.
- nzmsv 12y agoThe NVMe command set is pretty close to the existing SCSI spec though. In case of security the commands are the SCSI Security Protocol IN/OUT. Shameless plug: if you want to work with this stuff, check out purestorage.com/jobs
- jjoonathan 12y agoHah, I wonder if it'll come full circle and we'll get to see AT commands sent over SCSI (all inside of NVMe). Revenge for ATAPI :P
- nzmsv 12y agoThere is already a standard for doing exactly that: passing through ATA commands wrapped in SCSI. This can come in handy when a SATA drive sits behind some bridge that speaks SCSI.
- ChuckMcM 12y agoBut did they conform to the LIM[1] spec ? :-) They also had RAM drives you could buy. The point then as now is that increasing the "high performance" working set space of a program, increases the amount of transactional data that can be "in flight" during an operation, and that increases the overall size of the data set you can work with. I've been waiting for these boards to come down in price for about 4 years now. I started talking with Intel about them early on (we used their XM-25 SSDs because it was a price point for flash that was "enough" better than spinning rust that it made sense) and they insisted on trying to sell us the same flash chips on a PCIe card for 10x the dollars, I (and many others apparently) refused to pay that. Sure if you have a 'cost is no object' data base or something but for a large internet working set where revenue differences are measured in cents per thousand transactions? Not so much. I know one company that went so far as to design and build their own PCIe Flash card. I have heard it did great stuff for them. [1] LIM - Lotus-Intel-Microsoft spec for extended memory on IBM PC compatible machines.
- yuhong 12y agohardcards have nothing to do with LIM.
- ChuckMcM 12y agoWow, so much for relying on my memory. http://en.wikipedia.org/wiki/Expanded_memory http://en.wikipedia.org/wiki/Expanded_memory vs https://books.google.com/books?id=KjwEAAAAMBAJ&pg=PA61&lpg=PA61&dq=286+hardcard&source=bl&ots=tUPGkGS8mi&sig=_7k1A79zs_A5nNFSmqmR2YTFXYY&hl=en&sa=X&ei=jBAiVYH1OMitogTfmIHwAQ&ved=0CB8Q6AEwAA#v=onepage&q=286%20hardcard&f=false https://books.google.com/books?id=KjwEAAAAMBAJ&pg=PA61&lpg=P... I was thinking of the plug-in expanded memory cards rather than the plug in hard drive cards.
- higherpurpose 12y agoWasn't someone here saying these SSDs consume an abnormal amount of power to achieve those (2x?) faster speeds than current PCIe SSDs? Or was that applicable just to Intel's SSDs?
- Dylan16807 12y agoThat's just a design decision for those models. It's still PCIe, nothing is causing wasted power.
- jjoonathan 12y agoLast time I looked at the SATA PHY it seemed pretty wasteful. During every transaction the back channel was running at full speed blasting "R_OK R_OK R_OK" to the transmitting party. Not a checksum, mind you, just a bunch of magic DWORDs indicating that the drive was receiving (it would use a checksum too once the transmission was over, of course). I guess it's an easy way to keep the PLLs locked but it seemed like a pretty huge piece of low-hanging fruit towards the goal of reducing power consumption. In comparison, PCIe is a much more sophisticated serial protocol. It's an entire damned packet network with addresses, subnets (so to speak), routing, retry, a credit system for bandwidth sharing. Unlike ethernet, which typically tops out at achieving ~75% of its theoretical bandwidth, PCIe typically tops out at ~95% of its theoretical bandwidth. Crazy stuff. I'd bet good money that the PCIe PHY can beat the crap out of the SATA PHY on energy/bit and that the disparity will only widen with time.
- vardump 12y agoI'm usually getting 95%+ with gigabit ethernet. More than that with jumbo frames. Well, not with Realtek, but Intel and Broadcom chipsets are just fine.
- petrbela 12y agoSamsung claims the opposite: NVMe SSD provides low energy consumption to help data centers and enterprises operate more efficiently and reduce expenses. Power-related costs typically represent 31% of total data center costs, with the memory and storage portion of the power (including cooling) consuming 32% of the total data center power. NVMe SSD requires lower power (less than 6W active power) with energy efficiency (IOPS/Watt) that is 2.5x as efficient as SATA. http://www.samsung.com/global/business/semiconductor/product/flash-ssd/nvmessd http://www.samsung.com/global/business/semiconductor/product... http://www.pcworld.com/article/2866912/samsungs-ludicrously-fast-pcie-ssd-uses-almost-no-power-in-standby-mode.html http://www.pcworld.com/article/2866912/samsungs-ludicrously-...
- ziedaniel1 12y agoThe numbers in the article are all wrong. SATA's ceiling is 600 MBps (megabytes per second), not 600 Gpbs (gigabits per second). SAS goes up to 12Gbps, not 12GBps, which is actually the same thing as 1.5GBps. At least the PCIe numbers look right.
- carlob 12y agoapparently it was already corrected once, but it's still chock-full of mistakes.
- ChuckMcM 12y agoYes they are all wrong. 6 Gbps, 600 MBps (the capital B is supposed to indicate 'Bytes' versus 'bits') the encoding is 8b/10b which is 10 bauds per 8 bit byte. PCIe 2.0 has 2.5Gbps "lanes" PCIe 3.0 has 5Gbps "lanes" they can be ganged together for additional bandwidth. (x1, x2, x4, x8, x16) it is also 8b/10b so you divide by 10 to get Bytes per second (250MBps/500MBps). Both SATA and PCIe have a 'transaction limit' which is a function of the controller, which limits the total number of operations per second (IOPs). The product of the IOPs and the size of the transaction can never exceed the bandwidth of the channel. But it often is under. For example a typical SATA disk control (prior to the popularity of SSDs) would do about 25,000 IOPs, and if you had 512 byte (.5K) block reads and writes, you could read and write 25,000 * .5 or 12,500K or 12 MBps (which was much lower than the theoretical bandwidth of 200MBps on 2Gbps SATA II channels. Optimizing channel utilization requires that you figure out how many IOPs your OS/Controller can initiate and then sizing the payload to consume the max bandwidth. Large payloads and you'll push IOPs down, smaller payloads and you won't use all the bandwidth. One of the nicer aspects of ATM was that it was designed and specified for full channel utilization with 64 byte packets which made it possible to reason about the performance and latency of an arbitrary number of streams of data moving through it.
- p1esk 12y agoPCIe 3.0 uses 128b/130b line encoding, not 8b/10b.
- 12y ago
- nqzero 12y agoone of the fundamental problem with SSDs is the impedance mismatch introduced by emulating HDDs. NVMe doesn't appear to help with that at all we need an interface that allows us to bypass the FTL and access the underlying erase blocks
- skrause 12y agoIs there actually any evidence that this would improve performance significantly? Removing the translation layer means that every OS and their file systems have to do good wear leveling because otherwise you'll destroy blocks quickly.
- baruch 12y agoIt's hard to come up with evidence since there are very few options to create your own SSD/NVMe firmware for a real-world like device. The closest I've come up with is OpenSSD and that required $3000 and was for an old controller with very little documentation behind it.
- chongli 12y agoThat's pretty easy to do. Just use a log-structured filesystem[0]. The abstraction we use now is antiquated. It's very much reminiscent of the impedance mismatch in graphics APIs (such as OpenGL) which is now being solved (with Vulkan). [0] https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system
- justinsb 12y agoI think this is the idea behind FusionIO: doing the translation on the host CPU is faster, and allows you to expose additional commands (like atomic writes, or direct key-value interfaces).
- deelowe 12y agoWhy?
- otterley 12y agoNVMe is about as simple an interface to a block device as it gets: read blocks, write blocks, and a few health/diagnostic commands. Commands and geometries that originate in spinning disks have been completely extricated. I'm not sure that I'd want the protocol to get involved in the intricacies of backing store housekeeping as you propose. Newer generations of SSDs may not even have erase blocks or translation layers; do we want to have yet another protocol when the technology changes?
- exabrial 12y agoSFF-8639 cables look pretty cool! Reminds me of what Apple/Intel was trying to do with thunderbolt. I can imagine some creative people will find other uses for this connector.
- jccalhoun 12y agoHere's a good review with lots of pics and benchmarks: http://www.pcper.com/reviews/Storage/Intel-SSD-750-Series-12TB-PCIe-and-25-SFF-Review-NVMe-Consumer http://www.pcper.com/reviews/Storage/Intel-SSD-750-Series-12...
- ccleve 12y agoSo the important question is, how does the affect how we write applications? Is the api different, or are we still reading/writing disk files? Should we do memory mapping of the files or not? Should we parallelize access to different sections of big files? Or write a ton of small files? How does this affect database design? Current big data apps emphasize large append-only writes and large sequential reads (think LSM trees). Does this make sense any more? What does disk caching mean in the context of these new drives?
- pjc50 12y agoAPI is the same. Memory map if you prefer that access style and it suits your OS/language preferences. The OS will almost certainly not let you map straight across into the device's PCI memory mapped window so you'll incur a copy to userspace penalty either way. Benchmark. There is probably no longer any advantage to a sequential write, but you still have a per-syscall and per-IOP overhead, so large writes will be faster than N small ones. Disk caching is still there and still decreases latency but isn't so critical.
- MCRed 12y agoWhy not simply make an SSD Controller that has a thunderbolt port? Since Intel is building thunderbolt into its support chips now, this seems like a good way to get quick performance, plenty of bandwidth without having to come up with new standard, separate drivers, etc. Thunderbolt ports could be put on motherboards fairly easily, etc. Is there something I'm missing? Plus this would have the advantage of driving down prices for thunderbolt and increasing adoption.
- pjc50 12y agoWhy Thunderbolt? Its license fees are the main barrier.
- eurleif 12y agoIsn't Thunderbolt just externalized PCI Express?
- TazeTSchnitzel 12y agoThunderbolt is PCIe and DisplayPort.
- mrmondo 12y agoNVMe is one of the most important changes to storage over the past decade. I'm currently replacing all our SANs with storage servers filled with NVMe SSDs (as the tier 1 storage, commodity SATA SSDs for second tier). I've posted the link to my first blog post about it in the comments on another NVMe post in the past: http://smcleod.net/building-a-high-performance-ssd-san/ http://smcleod.net/building-a-high-performance-ssd-san/ I'm close to writing the next post around the actual build, my findings, benchmarks etc... Hopefully I'll have that done next week - but the system comes first. I'm a little disappointed with this article as I think it could do with a) some technical review and b) some more detailed information.
- zsmith928 12y agoawesome read, can't wait to see your results on performance, etc.
- deleted 12y ago[deleted]
- mojoe 12y agoAnother cool thing about NVMe is the smaller command set -- only about 10 commands (excluding admin commands), vs the 200+ that SCSI has grown to over the years. It's fairly quick to learn.
- Derbasti 12y agoIs this relevant for consumers? SSDs were a huge improvement for everyday computing; would NVMe be a similar jump?