4 ms·

The number of non-cacheable writes for SATA/AHCI is iirc 3x of NVMe command submission (1). Which does increase the lower bound of latency that is achievable. T

The number of non-cacheable writes for SATA/AHCI is iirc 3x of NVMe command submission (1). Which does increase the lower bound of latency that is achievable. That doesn't matter for slower drives, but on the very low latency end it's a measurable impact. Lower confidence: I think there's also a latency differences due to better control over interrupts with NVMe than SATA. Especially on a multi socket system, or even a higher core count single CPU system, that's quite a win.

hardwaresofton 5y ago
Hey I think you might have forgotten to add your source for (1) to this analysis!
- anarazel 5y ago
  I can't quite tell if you're cheekily suggesting I should provide sources for my claims, or whether you misunderstood (1) to be a reference to some source? If the latter, it was just intended to document the number of uncached writes for NVMe.
  - anarazel 5y ago
    If the latter, I think I just read the specs at some point & traced the linux drivers. There's prominent references to needing just one MMIO write for NVMe command submission in the NVMe spec: "Does not require uncacheable / MMIO register reads in the command submission or completion path. A maximum of one MMIO register write is necessary in the command submission path." Page 7 of https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2020.09.21-Ratified.pdf https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2... I didn't quickly re-find the relevant SATA/AHCI spec parts, but there's some references in other places, e.g. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ : "The NVMe specification is designed to significantly reduce the number of required MMIO compared to older specifications like AHCI. AHCI requires as many as 7 total MMIO operations (many of which are reads) per I/O, whereas NVMe requires an MMIO write only to ring the doorbell both on the submission and completion side. MMIO writes are posted, meaning the CPU does not stall waiting for any acknowledgement that the write made it to the PCIe device. This means that MMIO writes are much faster than MMIO reads. That said, they still have a significant cost." And even that one doorbell write (for each of submission and completion) can be batched. I'm not sure why I remember three - that might just have been the average number in simple scenarios, rather than the max.
  - hardwaresofton 5y ago
    Definitely the latter! Thought (1) was a pointer to some source

hardwaresofton 5y ago

Hey I think you might have forgotten to add your source for (1) to this analysis!

anarazel 5y ago
I can't quite tell if you're cheekily suggesting I should provide sources for my claims, or whether you misunderstood (1) to be a reference to some source? If the latter, it was just intended to document the number of uncached writes for NVMe.
- anarazel 5y ago
  If the latter, I think I just read the specs at some point & traced the linux drivers. There's prominent references to needing just one MMIO write for NVMe command submission in the NVMe spec: "Does not require uncacheable / MMIO register reads in the command submission or completion path. A maximum of one MMIO register write is necessary in the command submission path." Page 7 of https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2020.09.21-Ratified.pdf https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4b-2... I didn't quickly re-find the relevant SATA/AHCI spec parts, but there's some references in other places, e.g. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ : "The NVMe specification is designed to significantly reduce the number of required MMIO compared to older specifications like AHCI. AHCI requires as many as 7 total MMIO operations (many of which are reads) per I/O, whereas NVMe requires an MMIO write only to ring the doorbell both on the submission and completion side. MMIO writes are posted, meaning the CPU does not stall waiting for any acknowledgement that the write made it to the PCIe device. This means that MMIO writes are much faster than MMIO reads. That said, they still have a significant cost." And even that one doorbell write (for each of submission and completion) can be batched. I'm not sure why I remember three - that might just have been the average number in simple scenarios, rather than the max.
- hardwaresofton 5y ago
  Definitely the latter! Thought (1) was a pointer to some source