4 ms·
Linux Achieves 5.1M IOPS Per-Core with AMD Zen 3 and Intel Optane
- COGlory 5y agoWendell from Level1Techs has been beating his drum about pushing IOPS on these newer hardware platforms, and has some great videos about his journey and what is fundamentally broken under the hood with how software interacts with and fails to anticipate hardware as fast as the new Optane drives. https://www.youtube.com/watch?v=RfrMnVpPuVw https://www.youtube.com/watch?v=RfrMnVpPuVw - The initial video attempting to clear 20 million IOPS on Threadripper https://www.youtube.com/watch?v=WEMszbMUj68 https://www.youtube.com/watch?v=WEMszbMUj68 - 15 million IOPS on Xeon 8380s (you can see how this is going) https://www.youtube.com/watch?v=9Gu_rT8N0-U https://www.youtube.com/watch?v=9Gu_rT8N0-U - Discussion with Allyn Malventano of Intel My broad interpretation of this is that the software layer was not ready for things like 64+ core systems on a single socket, and PCI-E 4 drives moving data this fast. Nice to see it's starting to catch up.
- jeffbee 5y agoIt is not only the software layer that was not prepared for so many. Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). Some NICs only have 32 receive queues. It's now trivial to build a box with 256 threads, so these numbers don't look great.
- wtallis 5y ago> Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). You're talking specifically about server environments there, right? Because consumer NVMe devices started out with far lower queue counts (eg. 7) and even the top consumer SSD product lines have struggled to keep pace with growing Threadripper core counts. 64 queue support is still something of a luxury among consumer NVMe devices.
- jeffbee 5y agoThat sucks. I can't recall having personally seen one that didn't support at least 64 submission queues.
- wtallis 5y agoSamsung's current flagship 980 PRO consumer drive supports 128 queues, but the previous generation (970 PRO/EVO/EVO Plus) only supported 32. Their first two generations were limited to 8 and 7 queues. I wouldn't be surprised if these limits also applied to their entry-level enterprise SSDs that used the same controllers.
- MR4D 5y agoHow can we find this information out easily (not just for Samsung, but for others)? Even Anandtech only tested 32 queues on the 980 Pro [0]. [0] - https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/5 https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...
- wtallis 5y agoNumber of queues is not the same as the number of operations in a queue. Supporting more queues means that you need less communication and synchronization between CPU cores; ideally, you have at least one queue per core so that each CPU core can submit commands to the SSD independently. I didn't test queue depths higher than 32 when I reviewed the 980 PRO, because it's a consumer drive and consumer workloads hardly ever produce double-digit queue depths. Even going all the way up to QD32 is fairly unrealistic. Server workloads can be quite different, and some of them genuinely can result in a large number of simultaneous IO requests. Consumer workloads have barely even started to evolve to take advantage of SSDs, and hard drives aren't quite entirely gone from the consumer PC market. As for how to check how many queues a NVMe drive supports: the easy way is to install the drive in a system with a very large number of CPU cores, and see how many queues the Linux NVMe driver creates (check the drive's sysfs directory). My understanding from reading the NVMe spec is that you can't necessarily query the drive itself for the number of supported queues after the operating system's NVMe driver has already done that and set up its queues.
- wtallis 5y agoI think the most fundamental way in which the software ecosystem is unprepared doesn't require Optane storage or 64+ core systems to demonstrate. When CPU core counts first started growing, storage was still effectively single-threaded: a hard drive can only work on servicing one request at a time (though you can get a small throughput increase by keeping several requests in the queue), and scaling up to arrays of many drives was much more effective at increasing sequential throughput than random IO performance. With flash-based storage, the SSD now has far more parallelism than your CPU. Having one pending IO operation per CPU core isn't enough to keep your SSDs busy. Having one pending IO per software thread and doing lots of context switches to run more threads than you have cores still won't keep your drives busy, unless you're only doing IO in large chunks. So synchronous, blocking IO APIs are unable to make full use of your storage performance. Optane's low latency doesn't save you from this problem; it just highlights that context switches are also too expensive.
- eptcyka 5y agoFunny that, some people argue that the future of I/O is synchronous memory-mapped devices.
- wtallis 5y agoThat can potentially work, but only by getting the filesystem/block storage/PCIe layers of the IO stack out of the way, and even then it's only plausible for Optane or similarly low-latency storage. It'll never work for flash, because the latency of flash will never be low enough.
- thepenis 5y agoWow! This is OFF TOPIC. Hackernews is only for covid and anal sex conversations. Anything to do with computing is off-topic.
- willis936 5y agoI genuinely want to know more. What specific model Optane drive was used? A lot of the newer stuff is greatly hampered by the QLC backend.
- wtallis 5y agoThere are no products that mix second-generation Optane with QLC. The only two products that mix first-generation Optane with QLC are consumer drives that are don't work on AMD platforms. Jens is testing using the second-generation server Optane drives (P5800X), and getting performance that's completely impossible with any of the consumer Optane products, because all of those are still limited to PCIe gen3 speeds.
- MR4D 5y agoLinux dominance in the datacenter is about to grow even more. Awesome news!
- ThinkBeat 5y agoI need some context. Linux developers have made improvements to specific parts of the IO sub system. These changes will give Linux better IOPS performance. Prior test are quoted at 3,8 IOPS. They ran the old code on a MD Ryzen Threadripper 3970X at 3,8 IOPS. Now he got a better / fast CPU AMD Ryzen 9 5950X and it achieves an impressive 5.1M It seems to me that the new test does not make it clear how much code changes helped and how much a new CPU helped? Running the old version on the new CPU or the other way around would make it easier to compare?
- wtallis 5y agoMy understanding from Jens Axboe's tweets is that he ran the same software test on his old hardware and his new hardware. Separately, he's been using this test for quite a while to track the progress of software optimizations.
- lathiat 5y agoBecause hes only using a single CPU core I'm not sure how much the CPU would have helped but you're right he hasn't quantified this. He's the lead developer of io_uring a new kernel mechanism for doing I/O that is much more effecient for these large IOPS workloads and it itself has been getting lots of optimisation in every recent kernel so it could be both.