5 ms·

Thoroughly surprised the lack of difference between read latency on SATA SSD and NVMe SSD. I thought NVMe was significantly more performant than SATA. Guess the

Thoroughly surprised the lack of difference between read latency on SATA SSD and NVMe SSD. I thought NVMe was significantly more performant than SATA. Guess the difference is all on the writing end, so if you had some sort of mostly read workload it's fine.

lathiat 5y ago
I mean ultimately SATA and NVMe are just an interconnect. The biggest difference is throughout. Yes there can be some minor latency or round trip gains in the protocol but throughout is the biggest difference.
- anarazel 5y ago
  The number of non-cacheable writes for SATA/AHCI is iirc 3x of NVMe command submission (1). Which does increase the lower bound of latency that is achievable. That doesn't matter for slower drives, but on the very low latency end it's a measurable impact. Lower confidence: I think there's also a latency differences due to better control over interrupts with NVMe than SATA. Especially on a multi socket system, or even a higher core count single CPU system, that's quite a win.
  - hardwaresofton 5y ago
    Hey I think you might have forgotten to add your source for (1) to this analysis!
    - anarazel 5y ago
      I can't quite tell if you're cheekily suggesting I should provide sources for my claims, or whether you misunderstood (1) to be a reference to some source? If the latter, it was just intended to document the number of uncached writes for NVMe.
      - anarazel 5y ago
        If the latter, I think I just read the specs at some point & traced the linux drivers. There's prominent references to needing just one MMIO write for NVMe command submission in the NVMe spec: "Does not require uncacheable / MMIO register reads in the command submission or completion path. A maximum of one MMIO register write is necessary in the command submission path." Page 7 of https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2020.09.21-Ratified.pdf https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2... I didn't quickly re-find the relevant SATA/AHCI spec parts, but there's some references in other places, e.g. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ : "The NVMe specification is designed to significantly reduce the number of required MMIO compared to older specifications like AHCI. AHCI requires as many as 7 total MMIO operations (many of which are reads) per I/O, whereas NVMe requires an MMIO write only to ring the doorbell both on the submission and completion side. MMIO writes are posted, meaning the CPU does not stall waiting for any acknowledgement that the write made it to the PCIe device. This means that MMIO writes are much faster than MMIO reads. That said, they still have a significant cost." And even that one doorbell write (for each of submission and completion) can be batched. I'm not sure why I remember three - that might just have been the average number in simple scenarios, rather than the max.
      - hardwaresofton 5y ago
        Definitely the latter! Thought (1) was a pointer to some source
netflixandkill 5y ago
NVME will pull ahead with higher concurrency due to the much larger command queues. With single or small sets of reads the overall bus throughput doesn't matter much and the limitation is how fast the control board on the SSD can pull in data and put it on the bus.
cperciva 5y ago
NVMe is a higher performance interface, so initially drive manufacturers used it on their higher performance SSDs.
digikata 5y ago
The generation of flash chip in the drive can make a major difference there, as well as fill, the wear of the drive and it’s internal firmware architecture. The makers also target different tiers of performance with different model lines too and that can be a huge difference.
derefr 5y ago
It's IOPS, not throughput, that NVMe gets you. (Or rather, if you're not being bottlenecked by IOPS, then NVMe won't get you anything over a JBOD of SATA SSDs.) SATA SSDs will serve you just fine if you're Netflix or YouTube, streaming a bunch of video files to people. High throughput, low IOPS — no advantage to NVMe. But you'll get a huge boost from NVMe if you're doing a lot of little concurrent random reads from a much-larger-than-memory dataset. For example, if you're Twitter, fetching updates to users' timelines. If you've got higher concurrency, you'll want several NVMe SSDs behind a hardware RAID controller, in RAID0. And if you've got really high concurrency, you'll want several of these NVMe SSD RAID controllers, on separate PCI-e lanes, under software RAID0. That's the point when you're starting to reach those "basically identical to memory" figures that are the use-case for Intel's Optane.
- hardwaresofton 5y ago
  Thanks for this -- this is what I'm missing. I was starting to essentially suspect that NVMe is really more of a benefit for "hyperscaler"/IaaS providers... This lays it out well for me, and the how too. I rent my hardware so I don't have access to the hardware RAID stuff but this really filled out holes for me. > If you've got higher concurrency, you'll want several NVMe SSDs behind a hardware RAID controller, in RAID0. Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? > That's the point when you're starting to reach those "basically identical to memory" figures that are the use-case for Intel's Optane. I need to do more reading on this but thanks for pointing this out
  - derefr 5y ago
    > Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. The NVMe pool can be thought of as holding a "working" representation of the data — sort of like how, if you have a business layer that fetches a JSON document from a document store, then the business-layer node's main memory then holds a deserialized working representation of the data. You wouldn't back up/protect the data on your NVMe pool, any more than you'd back up the memory backing the deserialized JSON. If you wanted to modify the canonical data, you'd modify it in the canonical store. (For JSON, the document database; for our use-case, the WAL segments + base-backups in the object store.) † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record.
masklinn 5y ago
> Thoroughly surprised the lack of difference between read latency on SATA SSD and NVMe SSD. You did notice that the scales are different yes? At the low end the best case latencies are similar but nvme has much better worst cases: > For a 4 kilobytes block size the average time improved only a little, but the 99 percentile is two times lower. and as size increases nvme benefits.
- hardwaresofton 5y ago
  Do you mean the logarithmic scale? yeah I did notice that, but I definitely did overlook the p99 2x performance difference, thanks for drawing my attention to that. Internally I think I just had a mostly wrong intuition on how much of a technology shift NVMe was compared to SATA -- it's a bit hard to tell if we're approaching theoretical limits or if SATA was just very good for it's time. As a side note I wish this post didn't try to fit so much in one graph, but I can easily imagine someone complaining about the exact opposite so that's neither here nor there I guess. Separation by color and shade/intensity of color rather than the line patterns might have helped. Or maybe just completely separate charts for some of the dimensions... not sure.
  - pdimitar 5y ago
    I don't have scientific benchmarks but I did observe noticeable performance differences in data ingestion workflows between a machine with a SATA 3 drive and a machine with a NVMe drive (and it was a mid-range one, with 2200/1700 MB/s speeds). But the gains are only coming if you properly setup the machine, e.g. a NVMe drive can't make much use of its high concurrency stats if you don't have a strong multicore CPU with a newer motherboard that allows for more traffic.
anarazel 5y ago
I'm confused by how high their latencies are, something isn't right (/me goes and spends too much time that I instead should spend on paperwork). The biggest contributor is that they tested random reads - that'll likely end up bottlenecked by the FTL, not the protocol & interface latency. Which, especially on consumer devices, can add a lot. Testing sequential, non-buffered, 4k IO with fio, on my boot disk, a Samsung SSD 970 PRO 1TB, I see: Average latency of 13.8us (99.9th percentile 33us), with fio running on the socket the NVMe is attached to, and 16.22us on the remote node (99.9th 37us). With polling that goes down to 8.19 / 8.79us. If I instead test random reads, I see 69.68 / 70.63us (99.9th 147us/ 176us) local/remote node respectively, if I use a 128MB file. Unsurprisingly the local/remote difference nearly vanishes, as the transfer itself isn't the bottleneck anymore. Nor does polling help much. With larger files (I tested up to 128GB, didn't have enough free space for more), that increases a bit, but not by much. Even taking the effect of randomness into account, my disk is several years old, and the latencies reported in the post (and linked PDF) are well above what I see, on a two socket system, with plenty other things going on at the same time. I see considerably lower latencies on two different fast server grade SSDs as well (not quite to the optane level though). The PDF's benchmark description says: > The main goal of this work is to evaluate the hardware and not the filesystem. On each device we create a file of size approximately 90% that of a device size. We fill it with specific data to be sure that the space is allocated and that the reading takes place. To assert the latter we verify that received bytes match the ones we expect. I don't really understand why they're doing that. On most disks that will ensure that the FTL cannot operate from cache, even if that cache is generously sized. IME even random workloads tend to have more locality than a uniform distribution over 90% 4TB. But more importantly, this seems to counter the author's stated goals: > In this work we present a detailed overview and evaluation of modern storage reading performance with regard to available Linux synchronous and asynchronous interfaces. But the test setup chooses the scenario that is least likely to be influenced by the choice of interface? There's a lot of odd stuff in the later sections. It's not at all clear what 7.1 really measures. The numbers are too high for QD1 sequential reads (even at 10us latency one would just be a bit below 1GB/s). But too low for high QD sequential reads. So is it buffered - but then I see higher throughput at 4kB? Then 7.2 talks about "AIO" in the graphs but I think just means doing one read per thread. But how would that be "asynchronous reading"?

lathiat 5y ago

I mean ultimately SATA and NVMe are just an interconnect. The biggest difference is throughout. Yes there can be some minor latency or round trip gains in the protocol but throughout is the biggest difference.

anarazel 5y ago
The number of non-cacheable writes for SATA/AHCI is iirc 3x of NVMe command submission (1). Which does increase the lower bound of latency that is achievable. That doesn't matter for slower drives, but on the very low latency end it's a measurable impact. Lower confidence: I think there's also a latency differences due to better control over interrupts with NVMe than SATA. Especially on a multi socket system, or even a higher core count single CPU system, that's quite a win.
- hardwaresofton 5y ago
  Hey I think you might have forgotten to add your source for (1) to this analysis!
  - anarazel 5y ago
    I can't quite tell if you're cheekily suggesting I should provide sources for my claims, or whether you misunderstood (1) to be a reference to some source? If the latter, it was just intended to document the number of uncached writes for NVMe.
    - anarazel 5y ago
      If the latter, I think I just read the specs at some point & traced the linux drivers. There's prominent references to needing just one MMIO write for NVMe command submission in the NVMe spec: "Does not require uncacheable / MMIO register reads in the command submission or completion path. A maximum of one MMIO register write is necessary in the command submission path." Page 7 of https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2020.09.21-Ratified.pdf https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2... I didn't quickly re-find the relevant SATA/AHCI spec parts, but there's some references in other places, e.g. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ : "The NVMe specification is designed to significantly reduce the number of required MMIO compared to older specifications like AHCI. AHCI requires as many as 7 total MMIO operations (many of which are reads) per I/O, whereas NVMe requires an MMIO write only to ring the doorbell both on the submission and completion side. MMIO writes are posted, meaning the CPU does not stall waiting for any acknowledgement that the write made it to the PCIe device. This means that MMIO writes are much faster than MMIO reads. That said, they still have a significant cost." And even that one doorbell write (for each of submission and completion) can be batched. I'm not sure why I remember three - that might just have been the average number in simple scenarios, rather than the max.
    - hardwaresofton 5y ago
      Definitely the latter! Thought (1) was a pointer to some source

netflixandkill 5y ago

NVME will pull ahead with higher concurrency due to the much larger command queues. With single or small sets of reads the overall bus throughput doesn't matter much and the limitation is how fast the control board on the SSD can pull in data and put it on the bus.

cperciva 5y ago

NVMe is a higher performance interface, so initially drive manufacturers used it on their higher performance SSDs.

digikata 5y ago

The generation of flash chip in the drive can make a major difference there, as well as fill, the wear of the drive and it’s internal firmware architecture. The makers also target different tiers of performance with different model lines too and that can be a huge difference.

derefr 5y ago

It's IOPS, not throughput, that NVMe gets you. (Or rather, if you're not being bottlenecked by IOPS, then NVMe won't get you anything over a JBOD of SATA SSDs.) SATA SSDs will serve you just fine if you're Netflix or YouTube, streaming a bunch of video files to people. High throughput, low IOPS — no advantage to NVMe. But you'll get a huge boost from NVMe if you're doing a lot of little concurrent random reads from a much-larger-than-memory dataset. For example, if you're Twitter, fetching updates to users' timelines. If you've got higher concurrency, you'll want several NVMe SSDs behind a hardware RAID controller, in RAID0. And if you've got really high concurrency, you'll want several of these NVMe SSD RAID controllers, on separate PCI-e lanes, under software RAID0. That's the point when you're starting to reach those "basically identical to memory" figures that are the use-case for Intel's Optane.

hardwaresofton 5y ago
Thanks for this -- this is what I'm missing. I was starting to essentially suspect that NVMe is really more of a benefit for "hyperscaler"/IaaS providers... This lays it out well for me, and the how too. I rent my hardware so I don't have access to the hardware RAID stuff but this really filled out holes for me. > If you've got higher concurrency, you'll want several NVMe SSDs behind a hardware RAID controller, in RAID0. Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? > That's the point when you're starting to reach those "basically identical to memory" figures that are the use-case for Intel's Optane. I need to do more reading on this but thanks for pointing this out
- derefr 5y ago
  > Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. The NVMe pool can be thought of as holding a "working" representation of the data — sort of like how, if you have a business layer that fetches a JSON document from a document store, then the business-layer node's main memory then holds a deserialized working representation of the data. You wouldn't back up/protect the data on your NVMe pool, any more than you'd back up the memory backing the deserialized JSON. If you wanted to modify the canonical data, you'd modify it in the canonical store. (For JSON, the document database; for our use-case, the WAL segments + base-backups in the object store.) † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record.

masklinn 5y ago

> Thoroughly surprised the lack of difference between read latency on SATA SSD and NVMe SSD. You did notice that the scales are different yes? At the low end the best case latencies are similar but nvme has much better worst cases: > For a 4 kilobytes block size the average time improved only a little, but the 99 percentile is two times lower. and as size increases nvme benefits.

hardwaresofton 5y ago
Do you mean the logarithmic scale? yeah I did notice that, but I definitely did overlook the p99 2x performance difference, thanks for drawing my attention to that. Internally I think I just had a mostly wrong intuition on how much of a technology shift NVMe was compared to SATA -- it's a bit hard to tell if we're approaching theoretical limits or if SATA was just very good for it's time. As a side note I wish this post didn't try to fit so much in one graph, but I can easily imagine someone complaining about the exact opposite so that's neither here nor there I guess. Separation by color and shade/intensity of color rather than the line patterns might have helped. Or maybe just completely separate charts for some of the dimensions... not sure.
- pdimitar 5y ago
  I don't have scientific benchmarks but I did observe noticeable performance differences in data ingestion workflows between a machine with a SATA 3 drive and a machine with a NVMe drive (and it was a mid-range one, with 2200/1700 MB/s speeds). But the gains are only coming if you properly setup the machine, e.g. a NVMe drive can't make much use of its high concurrency stats if you don't have a strong multicore CPU with a newer motherboard that allows for more traffic.

anarazel 5y ago

I'm confused by how high their latencies are, something isn't right (/me goes and spends too much time that I instead should spend on paperwork). The biggest contributor is that they tested random reads - that'll likely end up bottlenecked by the FTL, not the protocol & interface latency. Which, especially on consumer devices, can add a lot. Testing sequential, non-buffered, 4k IO with fio, on my boot disk, a Samsung SSD 970 PRO 1TB, I see: Average latency of 13.8us (99.9th percentile 33us), with fio running on the socket the NVMe is attached to, and 16.22us on the remote node (99.9th 37us). With polling that goes down to 8.19 / 8.79us. If I instead test random reads, I see 69.68 / 70.63us (99.9th 147us/ 176us) local/remote node respectively, if I use a 128MB file. Unsurprisingly the local/remote difference nearly vanishes, as the transfer itself isn't the bottleneck anymore. Nor does polling help much. With larger files (I tested up to 128GB, didn't have enough free space for more), that increases a bit, but not by much. Even taking the effect of randomness into account, my disk is several years old, and the latencies reported in the post (and linked PDF) are well above what I see, on a two socket system, with plenty other things going on at the same time. I see considerably lower latencies on two different fast server grade SSDs as well (not quite to the optane level though). The PDF's benchmark description says: > The main goal of this work is to evaluate the hardware and not the filesystem. On each device we create a file of size approximately 90% that of a device size. We fill it with specific data to be sure that the space is allocated and that the reading takes place. To assert the latter we verify that received bytes match the ones we expect. I don't really understand why they're doing that. On most disks that will ensure that the FTL cannot operate from cache, even if that cache is generously sized. IME even random workloads tend to have more locality than a uniform distribution over 90% 4TB. But more importantly, this seems to counter the author's stated goals: > In this work we present a detailed overview and evaluation of modern storage reading performance with regard to available Linux synchronous and asynchronous interfaces. But the test setup chooses the scenario that is least likely to be influenced by the choice of interface? There's a lot of odd stuff in the later sections. It's not at all clear what 7.1 really measures. The numbers are too high for QD1 sequential reads (even at 10us latency one would just be a bit below 1GB/s). But too low for high QD sequential reads. So is it buffered - but then I see higher throughput at 4kB? Then 7.2 talks about "AIO" in the graphs but I think just means doing one read per thread. But how would that be "asynchronous reading"?