Moreover, at least last time I played around with it on reasonable hardware (3rd gen EPYC), your first real limitation is your operating systems’ IO scheduler,

Moreover, at least last time I played around with it on reasonable hardware (3rd gen EPYC), your first real limitation is your operating systems’ IO scheduler, not the hardware — at least on a “single node.” At least a year and change ago, you started slamming into a wall around 15-16M random IOPS (from your backing store). Wasn’t CPU bound. Wasn’t memory bound or a NUMA issue. Wasn’t an issue of PCIe lanes. 100% the IO scheduler as dtrace very much laid that bare. We’re very much in a situation where the software is catching up to the hardware. For example, look at the proposed NEST Scheduler for Linux where they were seeing 10-100% (yes, as much as double!) performance improvement with a 10-15% reduction in power consumption, mostly by focusing on keeping, “hot cores hot.” IO scheduler and process scheduler improvements will offer truly material gains in coming years, even if you still use the same high-end hardware you have today.