4 ms·

Your NVMe SSD gets 1 million IOPs with overlapping requests. That means each new request issued does not wait for the previous one to finish. There's a queue

Your NVMe SSD gets 1 million IOPs with overlapping requests. That means each new request issued does not wait for the previous one to finish. There's a queue depth, and keeping it full enough is needed to reach 1 million IOPS throughput. The latency will be higher than 1µs for each request. RAM also does overlapping requests and has a queue depth, so that's a reasonable comparison. The headline of this article says they are standardising RAM with a throughput of 17600 million IOPS. If you have 96 CPU cores per socket, and 6 channels per socket going to separate MRDIMMs, that's 1100 million IOPS per core. At 128 cores, 825 million IOPS per core. So, this new RAM standard is 825-1100 times faster throughput than your laptop SSD, when counted in IOPS (without regard to the data size). If you want to compare latency instead of throughpt, the IOPS measurement for that occurs when the sender waits for each request to complete before issuing the next one. In that case you're right that it's will be much lower IOPS per CPU core, but this also makes it lower IOPS for the SSD so that comparison still favours RAM. Each CPU core is independent, so each core would likely wait for its own previous request before issuing the next one (this actually happens when a CPU core is following a linked list), but the cores are doing this independently of each other, in parallel. So with 96 CPU cores per socket, each core waiting for its own request to complete before issuing the next one, and assuming ~100ns latency, that's 960 million IOPS total, and 10 million IOPS per core. With 128 CPU cores it's still 10 million IOPS per core.

imtringued 4y ago
People don't seem to understand that IOPS is a meaningless measure from a practical perspective. It is just a weird way to express throughput. An SSD that loads 4KB at a time has 64 times less IOPS than RAM that loads 64 bytes at a time assuming both have the same bandwidth.
- Dylan16807 4y ago
  The average fetch is not that big. IOPS and GB/s are both very important numbers. From a practical perspective, the IOPS of RAM is critical for running programs. And if you're doing anything other than big file transfers, an SSD plugged into USB 2.0 at 35MB/s will beat a hard drive on USB 3.0 at 200MB/s. Even better is IOPS at queue depth 1, IOPS at queue depth 16 or 32, and max bandwidth.
MrFoof 4y ago
Moreover, at least last time I played around with it on reasonable hardware (3rd gen EPYC), your first real limitation is your operating systems’ IO scheduler, not the hardware — at least on a “single node.” At least a year and change ago, you started slamming into a wall around 15-16M random IOPS (from your backing store). Wasn’t CPU bound. Wasn’t memory bound or a NUMA issue. Wasn’t an issue of PCIe lanes. 100% the IO scheduler as dtrace very much laid that bare. We’re very much in a situation where the software is catching up to the hardware. For example, look at the proposed NEST Scheduler for Linux where they were seeing 10-100% (yes, as much as double!) performance improvement with a 10-15% reduction in power consumption, mostly by focusing on keeping, “hot cores hot.” IO scheduler and process scheduler improvements will offer truly material gains in coming years, even if you still use the same high-end hardware you have today.
jiggawatts 4y ago
I love the responses in this thread. Turned out that my "Fermi estimate" was wrong, by about a factor of 100x. A simple back-of-the-napkin maths is that for a system with 30GB/s memory bandwidth and 32-byte cache lines works out to about a billion I/O operations per second. This would be a typical laptop system, or the like. However, 100ns per read is still a valid scenario for a single-threaded program "chasing pointers" as in a linked-list. I think it's still fair to say that an un-optimised single-threaded program accessing memory randomly is only 10x faster than a parallel program efficiently utilising a modern NVMe SSD. The corollary is that there is (still) no scenario where an SSD would "beat" the performance of RAM since the worst-case for RAM still beats the best-case for SSD.

imtringued 4y ago

People don't seem to understand that IOPS is a meaningless measure from a practical perspective. It is just a weird way to express throughput. An SSD that loads 4KB at a time has 64 times less IOPS than RAM that loads 64 bytes at a time assuming both have the same bandwidth.

Dylan16807 4y ago
The average fetch is not that big. IOPS and GB/s are both very important numbers. From a practical perspective, the IOPS of RAM is critical for running programs. And if you're doing anything other than big file transfers, an SSD plugged into USB 2.0 at 35MB/s will beat a hard drive on USB 3.0 at 200MB/s. Even better is IOPS at queue depth 1, IOPS at queue depth 16 or 32, and max bandwidth.

MrFoof 4y ago

Moreover, at least last time I played around with it on reasonable hardware (3rd gen EPYC), your first real limitation is your operating systems’ IO scheduler, not the hardware — at least on a “single node.” At least a year and change ago, you started slamming into a wall around 15-16M random IOPS (from your backing store). Wasn’t CPU bound. Wasn’t memory bound or a NUMA issue. Wasn’t an issue of PCIe lanes. 100% the IO scheduler as dtrace very much laid that bare. We’re very much in a situation where the software is catching up to the hardware. For example, look at the proposed NEST Scheduler for Linux where they were seeing 10-100% (yes, as much as double!) performance improvement with a 10-15% reduction in power consumption, mostly by focusing on keeping, “hot cores hot.” IO scheduler and process scheduler improvements will offer truly material gains in coming years, even if you still use the same high-end hardware you have today.

jiggawatts 4y ago

I love the responses in this thread. Turned out that my "Fermi estimate" was wrong, by about a factor of 100x. A simple back-of-the-napkin maths is that for a system with 30GB/s memory bandwidth and 32-byte cache lines works out to about a billion I/O operations per second. This would be a typical laptop system, or the like. However, 100ns per read is still a valid scenario for a single-threaded program "chasing pointers" as in a linked-list. I think it's still fair to say that an un-optimised single-threaded program accessing memory randomly is only 10x faster than a parallel program efficiently utilising a modern NVMe SSD. The corollary is that there is (still) no scenario where an SSD would "beat" the performance of RAM since the worst-case for RAM still beats the best-case for SSD.