5 ms·

Wendell from Level1Techs has been beating his drum about pushing IOPS on these newer hardware platforms, and has some great videos about his journey and what is

Wendell from Level1Techs has been beating his drum about pushing IOPS on these newer hardware platforms, and has some great videos about his journey and what is fundamentally broken under the hood with how software interacts with and fails to anticipate hardware as fast as the new Optane drives. https://www.youtube.com/watch?v=RfrMnVpPuVw https://www.youtube.com/watch?v=RfrMnVpPuVw - The initial video attempting to clear 20 million IOPS on Threadripper https://www.youtube.com/watch?v=WEMszbMUj68 https://www.youtube.com/watch?v=WEMszbMUj68 - 15 million IOPS on Xeon 8380s (you can see how this is going) https://www.youtube.com/watch?v=9Gu_rT8N0-U https://www.youtube.com/watch?v=9Gu_rT8N0-U - Discussion with Allyn Malventano of Intel My broad interpretation of this is that the software layer was not ready for things like 64+ core systems on a single socket, and PCI-E 4 drives moving data this fast. Nice to see it's starting to catch up.

jeffbee 5y ago
It is not only the software layer that was not prepared for so many. Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). Some NICs only have 32 receive queues. It's now trivial to build a box with 256 threads, so these numbers don't look great.
- wtallis 5y ago
  > Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). You're talking specifically about server environments there, right? Because consumer NVMe devices started out with far lower queue counts (eg. 7) and even the top consumer SSD product lines have struggled to keep pace with growing Threadripper core counts. 64 queue support is still something of a luxury among consumer NVMe devices.
  - jeffbee 5y ago
    That sucks. I can't recall having personally seen one that didn't support at least 64 submission queues.
    - wtallis 5y ago
      Samsung's current flagship 980 PRO consumer drive supports 128 queues, but the previous generation (970 PRO/EVO/EVO Plus) only supported 32. Their first two generations were limited to 8 and 7 queues. I wouldn't be surprised if these limits also applied to their entry-level enterprise SSDs that used the same controllers.
      - MR4D 5y ago
        How can we find this information out easily (not just for Samsung, but for others)? Even Anandtech only tested 32 queues on the 980 Pro [0]. [0] - https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/5 https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...
        wtallis 5y ago
        Number of queues is not the same as the number of operations in a queue. Supporting more queues means that you need less communication and synchronization between CPU cores; ideally, you have at least one queue per core so that each CPU core can submit commands to the SSD independently. I didn't test queue depths higher than 32 when I reviewed the 980 PRO, because it's a consumer drive and consumer workloads hardly ever produce double-digit queue depths. Even going all the way up to QD32 is fairly unrealistic. Server workloads can be quite different, and some of them genuinely can result in a large number of simultaneous IO requests. Consumer workloads have barely even started to evolve to take advantage of SSDs, and hard drives aren't quite entirely gone from the consumer PC market. As for how to check how many queues a NVMe drive supports: the easy way is to install the drive in a system with a very large number of CPU cores, and see how many queues the Linux NVMe driver creates (check the drive's sysfs directory). My understanding from reading the NVMe spec is that you can't necessarily query the drive itself for the number of supported queues after the operating system's NVMe driver has already done that and set up its queues.
        antongribok 5y ago
        Is this not the correct way to do it? $ sudo nvme get-feature /dev/nvme0 -f 7 get-feature:0x7 (Number of Queues), Current value:0x7f007f
wtallis 5y ago
I think the most fundamental way in which the software ecosystem is unprepared doesn't require Optane storage or 64+ core systems to demonstrate. When CPU core counts first started growing, storage was still effectively single-threaded: a hard drive can only work on servicing one request at a time (though you can get a small throughput increase by keeping several requests in the queue), and scaling up to arrays of many drives was much more effective at increasing sequential throughput than random IO performance. With flash-based storage, the SSD now has far more parallelism than your CPU. Having one pending IO operation per CPU core isn't enough to keep your SSDs busy. Having one pending IO per software thread and doing lots of context switches to run more threads than you have cores still won't keep your drives busy, unless you're only doing IO in large chunks. So synchronous, blocking IO APIs are unable to make full use of your storage performance. Optane's low latency doesn't save you from this problem; it just highlights that context switches are also too expensive.
- eptcyka 5y ago
  Funny that, some people argue that the future of I/O is synchronous memory-mapped devices.
  - wtallis 5y ago
    That can potentially work, but only by getting the filesystem/block storage/PCIe layers of the IO stack out of the way, and even then it's only plausible for Optane or similarly low-latency storage. It'll never work for flash, because the latency of flash will never be low enough.

jeffbee 5y ago

It is not only the software layer that was not prepared for so many. Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). Some NICs only have 32 receive queues. It's now trivial to build a box with 256 threads, so these numbers don't look great.

wtallis 5y ago
> Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). You're talking specifically about server environments there, right? Because consumer NVMe devices started out with far lower queue counts (eg. 7) and even the top consumer SSD product lines have struggled to keep pace with growing Threadripper core counts. 64 queue support is still something of a luxury among consumer NVMe devices.
- jeffbee 5y ago
  That sucks. I can't recall having personally seen one that didn't support at least 64 submission queues.
  - wtallis 5y ago
    Samsung's current flagship 980 PRO consumer drive supports 128 queues, but the previous generation (970 PRO/EVO/EVO Plus) only supported 32. Their first two generations were limited to 8 and 7 queues. I wouldn't be surprised if these limits also applied to their entry-level enterprise SSDs that used the same controllers.
    - MR4D 5y ago
      How can we find this information out easily (not just for Samsung, but for others)? Even Anandtech only tested 32 queues on the 980 Pro [0]. [0] - https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/5 https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...
      - wtallis 5y ago
        Number of queues is not the same as the number of operations in a queue. Supporting more queues means that you need less communication and synchronization between CPU cores; ideally, you have at least one queue per core so that each CPU core can submit commands to the SSD independently. I didn't test queue depths higher than 32 when I reviewed the 980 PRO, because it's a consumer drive and consumer workloads hardly ever produce double-digit queue depths. Even going all the way up to QD32 is fairly unrealistic. Server workloads can be quite different, and some of them genuinely can result in a large number of simultaneous IO requests. Consumer workloads have barely even started to evolve to take advantage of SSDs, and hard drives aren't quite entirely gone from the consumer PC market. As for how to check how many queues a NVMe drive supports: the easy way is to install the drive in a system with a very large number of CPU cores, and see how many queues the Linux NVMe driver creates (check the drive's sysfs directory). My understanding from reading the NVMe spec is that you can't necessarily query the drive itself for the number of supported queues after the operating system's NVMe driver has already done that and set up its queues.
        antongribok 5y ago
        Is this not the correct way to do it? $ sudo nvme get-feature /dev/nvme0 -f 7 get-feature:0x7 (Number of Queues), Current value:0x7f007f

wtallis 5y ago

I think the most fundamental way in which the software ecosystem is unprepared doesn't require Optane storage or 64+ core systems to demonstrate. When CPU core counts first started growing, storage was still effectively single-threaded: a hard drive can only work on servicing one request at a time (though you can get a small throughput increase by keeping several requests in the queue), and scaling up to arrays of many drives was much more effective at increasing sequential throughput than random IO performance. With flash-based storage, the SSD now has far more parallelism than your CPU. Having one pending IO operation per CPU core isn't enough to keep your SSDs busy. Having one pending IO per software thread and doing lots of context switches to run more threads than you have cores still won't keep your drives busy, unless you're only doing IO in large chunks. So synchronous, blocking IO APIs are unable to make full use of your storage performance. Optane's low latency doesn't save you from this problem; it just highlights that context switches are also too expensive.

eptcyka 5y ago
Funny that, some people argue that the future of I/O is synchronous memory-mapped devices.
- wtallis 5y ago
  That can potentially work, but only by getting the filesystem/block storage/PCIe layers of the IO stack out of the way, and even then it's only plausible for Optane or similarly low-latency storage. It'll never work for flash, because the latency of flash will never be low enough.