12 ms·

Modern storage is plenty fast, but the APIs are bad

512 points by harporoeder 6y ago

Ericson2314 6y ago
From the author's previous piece: https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/ https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-wi... > Our CTO, Avi Kivity, made the case for async at the Core C++ 2019 event. The bottom line is this; in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another. There are good reasons why network programming is done asynchronously, and you should consider that for your own application development too. > > It fundamentally changes the way Linux applications are to be designed: Instead of a flow of code that issues syscalls when needed, that have to think about whether or not a file is ready, they naturally become an event-loop that constantly add things to a shared buffer, deals with the previous entries that completed, rinse, repeat. As someone that's been working on FRP related things for a while now, this feels very vindicating. :) I few like as recently as a few years ago, the systems world was content with it's incremental hacks, but now the gap between the traditional interfaces and hardware realities has become too much, and bigger redesigning is afoot. Excited for what emerges!
- mehrdadn 6y ago
  > in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another. Interesting take, and NUMA CPUs have felt networked to me when I've used them, but typical multicore UMA CPUs sure haven't... is there a reason to believe this will change (or already has), or did the author mean to only talk about NUMA?
  - ddorian43 6y ago
    Both, even multicore. Same ideas are used by Red Panda (open source kafka++ clone).
  - jeffbee 6y ago
    I guess it depends on what you consider to be "typical". If you look at a many-core chip from AMD you'll see a gradient of access latency from one core to another. You'll see the same on any Intel Skylake-X descendant, although the slope of that gradient is less. Your software will need to be very highly optimized already before you start to sweat the difference, though.
    - masklinn 6y ago
      Here's what jeffbee is talking about: https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-dive-review-5950x-5900x-5800x-and-5700x-tested/5 https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-di...
      - jeffbee 6y ago
        Yep. If you are sweating a microsecond, 100 nanoseconds is significant chunk of your budget. For this and possibly other reasons, a many-core CPU isn't always a great choice for hosted storage. If your goal is to export NVMe blocks over a network interface, you might be better off with an easier-to-program 4- or 8-core CPU. I don't like seeing 128 cores and a bunch of NVMe devices in the same box because it just causes trouble.
        magicalhippo 6y ago
        With p2pdma[1], it seems you don't even need a fancy CPU to push GB/s. [1]: https://www.youtube.com/watch?v=LDOlqgUZtHE https://www.youtube.com/watch?v=LDOlqgUZtHE (9:40)
- hinkley 6y ago
  > that have to think about whether or not a file is ready, they naturally become an event-loop that constantly add things to a shared buffer, deals with the previous entries that completed, rinse, repeat. This is the exception, not the rule, and it bugs me when APIs default to this. Most consumers of data are not looking at the stream of data, and in many cases where streaming is what I want, there are tools and APIs for handling that outside of my application logic. Much of the time I’m dealing with units of data only after the entire unit has arrived. Because if the message is not complete there is no forward progress to be made. My tools should reflect that reality, not what’s quickest for the API writers to create. In fact, if I remember my queuing theory properly, responsiveness is improved if the system prioritizes IO operations that can be finished (eg, EOS, EOF) over processing buffers for one that is still in the middle, which can’t happen with an event stream abstraction.
- pjmlp 6y ago
  The problem is forcing everyone to program in asynchronous way. WinRT tried to go down that route (only asynchronous APIs), to drive developers into that path, but eventually they had to support synchronous as well due to the received resistence.
  - justicezyx 6y ago
    Before Herb Sutter's free lunch is over, not many writes in multi-threading code. Then today, everyone is writing multi-threading code, one way or another; and majority indirectly in newer languages like Go and Rust, or better frameworks like Actors, message passing, coroutines, and some noble souls who are capable enough, in classic pthread and other threading APIs. Of course everyone is going to program in an async way; that's how nature works. But it certainly will not be in a fashinon that is repulsive to you, just give it some time. Maybe 10 years.
    - pjmlp 6y ago
      Agreed, it is also why Microsoft was one of the biggest contributors to the co-routines support in C++, and have frameworks like Orleans and Coyote.
ivoras 6y ago
> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of flash-based storage just yet.
- gravypod 6y ago
  Depending on the storage technology the comparison to RAM is not that far off. Intel is trying to market it that way any way [0]. It's obviously not RAM but it's not the <500GB 5200RPM SATA 3GB/s disk I started programming on. [0] - https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html https://www.intel.com/content/www/us/en/architecture-and-tec...
  - smcameron 6y ago
    Yeah, back in 2014, I worked at HP on storage drivers for linux, and we got 1 million IOPS (4k random reads) on a single controller, with SSDs, but we had to do some fairly hairy stuff. This was back when NVME was new and we were trying to do SCSI over PCIe. We set up multiple ring buffers for command submission and command completion, one each per CPU, and pinned threads to CPUs and were very careful to avoid locking (e.g. spinlocks, etc.). I think we also had to pin some userland processes to particular CPUs to avoid NUMA induced bottlenecks. The thing is, up until this point, for the entire history of computers, storage was so relatively slow compared to memory and the CPU that drivers could be quite simple, chuck requests and completions into queues managed by simple locking, and the fraction of time that requests spent inside the driver would still be negligible compared to the time they spent waiting for the disks. If you could theoretically make your driver infinitely fast, this would only amount to maybe a 1% speedup. So there was no need to spend a lot of time thinking about how to make the driver super efficient. Until suddenly there was.
    - smcameron 6y ago
blindm 6y ago
Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?
- rcxdude 6y ago
  I've had one (very early and cheap) SSD fail on me. Other than that I don't think I've seen or heard of any issues across a large range of more modern SSDs. The reliability and endurange issues which occured on earlier SSDs no longer seem to be a problem (this is in part because flash density has skyrocketed: because each flash chip can operate more or less independently, the more storage an SSD has the faster it can run and the more write endurance it has).
- tjoff 6y ago
  The solution for this problem is raid. Way cheaper and far superior in terms of reliability than your solution. Or if that isn't an option (laptop?) a good backup-solution that that runs daily or more often is also a better and cheaper alternative. Drives may dail at any time, but they don't age the way your post would suggest.
  - blindm 6y ago
    I've looked into RAID. It seems a bit complicated to use. Is it trivial to create a RAID array in Linux with zero fuss and the whole thing 'just working' with very little knowledge of the filesystem itself other than it keeps your data 'safe' and redundancy baked in?
    - dfinninger 6y ago
      Do you mean “use” or “setup”? RAID is trivial to use. Mount the volume to a directory and use it like normal. The setup is a bit more involved, but really not that bad. It’s a couple commands to join a few disks in an array and then you make a file system and mount it. https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-crea...
kibwen 6y ago
Prior discussion from /r/rust where the author is present to answer questions: https://www.reddit.com/r/rust/comments/k16j6x/modern_storage_is_plenty_fast_it_is_the_apis_that/ https://www.reddit.com/r/rust/comments/k16j6x/modern_storage...
jeroenhd 6y ago
I suppose _modern_ storage is fast, but how many servers are running on storage this modern? None of mine are and my work dev machine is still rocking a SATA 2.5" SSD. We're probably still a few years off from being able to switch to this fast I/O yet. With the new game consoles switching over to PCIe SSDs I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Even with SATA drives like mine though, there's really not that much performance loss from doing IO operations. I've run my OS with 8GiB of SSD swap in active use during debugging and while the stutters are annoying and distracting, the computer didn't grind to a halt like it would with spinning rust. Storage speed has increased massively in the last five years, for the love of god fellow developers, please make use of it when you can! That said, deferring IO until you're done still makes sense for some consumer applications because cheap laptops are still being sold with hard drives and those devices are probably the minimum requirement you'll be serving.
- rcxdude 6y ago
  Well, if the whole async 'I/O is the bottleneck' principle which was the refrain from a few years ago is actually true, then servers running databases should be focusing on upgrading their storage to these levels, since that's where the most bang for buck comes from in terms of performance gain is (of course now the big thing is running everything on clouds like AWS where everything is dog slow and really expensive, so perhaps it doesn't actually matter). (In fact the main reason for the API changes covered in the article is because the CPU and RAM can no longer run laps around storage).
- scribu 6y ago
  > how many servers are running on storage this modern? AWS allows you to provision EC2 instances with NVMe, without much fanfare. Cost is only ~20% more than SATA.
- wtallis 6y ago
  > I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.
mehrdadn 6y ago
I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this piece I had the immense pleasure of getting early access to one of the next generation Optane devices, from Intel. The entire blog post is complaining about how great engineers have misconception about modern storage technology and yet to prove it the author had to obtain benchmarks from early access to next-generation devices...?! And to top it off, from this we conclude "the disconnect" is due to the APIs? Not, say, from the possibility that such blazing-fast components may very well not even exist in users' devices? I'm not saying the conclusions are wrong, but the logic surely doesn't follow... and honestly it's a little tasteless to criticize people's understanding if you're going to base the criticism on things they in all likelihood don't even have access to.
- arka2147483647 6y ago
  I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.
  - Miraste 6y ago
    Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.
    - mehrdadn 6y ago
      3 years I think: https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint > It was announced in July 2015 and is available on the open market under brand names Optane (Intel) and subsequently QuantX (Micron) since April 2017. For comparison, look at how many decades it took SSDs to become commonplace: https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_SSDs https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_...
deleted 6y ago
[deleted]
Thaxll 6y ago
How those modern API runs on old HW?
- jlokier 6y ago
  > How those modern API runs on old HW? Very well. The API Glauber (the author) describes has always been a good API for performance even with old HDDs and old CPUs. In fact with very old CPUs the reduction in memory copies and system calls was more significant than it is now. The extra control over memory use with direct I/O is beneficial on some lower-end embedded systems too, e.g. video streaming from HDD on low memory systems. It's just that before fast SSDs, there wasn't as much motivation to get the "best" I/O performance via complex APIs, because most of the time overhead was dominated by the HDD performance itself. (This was less true for big, fast RAIDs though.) There was some benefit in a small amount of parallelism with HDDs, to improve block sorting, but that was usually best achieved with a few threads or processes, which is still a simple API to use, just read() and write() syscalls. The only things really worth doing a complex API for before were O_DIRECT+AIO together, and even then they need a well-written, I/O-aware application to make them really worth using. In general, those applications have tended to be databases and VM hypervisors, but I've also seen it done for optimized file streaming to/from HDD on embedded systems. Though beneficial, the Linux AIO implementation had problems for a long time, even with O_DIRECT in some cases (such as filling holes in sparse files and extending files); so it was never reliably "true" async I/O. And there was no memory-buffer transfer and system call elision as there is with io_uring. Now there is more motivation, so the API has been improved. O_DIRECT+AIO+io_uring is a better combination. The benefits are increasingly worth the effort for more kinds of applications due to the faster SSDs, and the SSDs' ability to handle large I/O request queues. But they would have been a good API combination for performance 20 years ago too.
bob1029 6y ago
One thing I have started to realize is that best case latency of an NVMe storage device is starting to overlap with areas where SpinWait could be more ideal than an async/await API. I am mostly advocating for this from a mass parallel throughput perspective, especially if batching is possible. I have started to play around with using LMAX Disruptor for aggregating a program's disk I/O requests and executing them in batches. This is getting into levels of throughput that are incompatible with something like what the Task abstractions in .NET enable. The public API of such an approach is synchronous as a result of this design constraint. Software should always try to work with the physical hardware capabilities. Modern SSDs are most ideally suited to arrangements where all data is contained in an append-only log with each batch written to disk representing a consistent snapshot. If you are able to batch thousands of requests into a single byte array of serialized modified nodes, you can append this onto disk so much faster than if you force the SSD to make individual writes per new/modified entity.
- wtallis 6y ago
  On Linux, it's already a NVMe driver option to enable polling for (high priority) IO completion rather than sleeping until an interrupt. The latency of handling an interrupt and doing a couple of context switches is higher than the best-case latency for fast SSDs. The io_uring userspace API also has a polling mode.
zackmorris 6y ago
I agree with the premise, but disagree with the conclusion. For a little background, my first computer was a Mac Plus around 1985, and I remember doing file copy tests on my first hard drive (an 80 MB) at over 1 MB/sec. If I remember correctly, SCSI could do 5 MB/sec copies clear back in the mid-80s. So until we got SSD, hard drive speed stayed within the same order of magnitude for like 30 years (as most of you remember): http://chrislawson.net/writing/macdaniel/2k1120cl.shtml http://chrislawson.net/writing/macdaniel/2k1120cl.shtml So the time to take our predictable deterministic synchronous blocking business logic into the maze of asynchronous promise spaghetti was a generation ago when hard drive speeds were two orders of magnitude slower than today. In other words, fix the bad APIs. Please don't make us shift paradigms. Now if we want to talk about some kind of compiled or graph-oriented way of processing large numbers of files performantly with some kind of async processing internally, then that's fine. Note that this solution will mirror whatever we come up with for network processing as well. That was the whole point of UNIX in the first place, to treat file access and network access as the same stream-oriented protocol. Which I think is the motive behind taking file access into the same problematic async domain that web development is having to deal with now. But really we should get the web back to the proven UNIX/Actor model way of doing things with synchronous blocking I/O.
tobias3 6y ago
Intuitively one should be able to approach the max speed for sequential reads via some tuning (queue/read_ahead_kb) even with the traditional, blocking posix interface. This would require a large enough read-ahead and large enough buffer size. Not poisoning the page cache/manually managing the page cache is an orthogonal issue and only relevant for some applications (and the additional memory copy barely makes a difference in OPs post). One advantage of using high level (Linux) kernel interfaces is that this "automatically" gets faster with newer Linux versions without a need of large application level changes. Maybe in a few years we'll have an extra cache layer, or it stores to persistent memory now. Linux will (slowly) improve and your application with it. This won't happen if it is specifically tuned for Direct I/O with Intel Optane in 2020. But yeah, random IO is (currently) another issue, and as said the usual advice is to avoid them. And with the old API this still holds. If one currently wants fast random IO one needs to use io_uring/aio (with Direct-IO) or just live with the performance not being optimal and hope that the page cache does more good than bad (like Postgresql).
- jorangreef 6y ago
  The page cache is not reliable, and actually does more bad than good, especially in the case of PostgreSQL: https://www.usenix.org/conference/atc20/presentation/rebello https://www.usenix.org/conference/atc20/presentation/rebello
kazinator 6y ago
This is a really poor article. Only in very rare circumstances can developers change the API's. API's are not "bad"; they are built to various important requirements. Only some of those requirements have to do with performance. > “Well, it is fine to copy memory here and perform this expensive computation because it saves us one I/O operation, which is even more expensive”. "I/O operation" in fact refers to the API call, not to the raw hardware operation. If the developer measured this and found it true, how can it be a misconception? It may be caused by a "bad" I/O API, but so what? The API is what it is. API's provide one requirement which is stability: keeping applications working. That is king. You can't throw out API's every two years due to hardware advancements. > “If we split this into multiple files it will be slow because it will generate random I/O patterns. We need to optimize this for sequential access and read from a single file” Though solid state storage doesn't have track-to-track seek times, the sequential-access-fast rule of thumb has not become false. Random access may have to wastefully read larger blocks of the data than are actually requested by the application. The unused data gets cached, but if it's not going to be accessed any time soon, it means that something else got wastefully bumped out of the cache. Sequential access is likely to make use of an entire block. Secondly, there is that API again. The underlying operating system may provide a read-ahead mechanism which reduces its own overheads, benefiting the application which structures its data for sequential access, even if there is no inherent hardware-level benefit. If there is any latency at all between the application and the hardware, and if you can guess what the application is going to read next, that's an opportunity to improve performance. You can correctly guess what the application will read if you guess that it is doing a sequential read, and the application makes that come true.
- cahooon 6y ago
  I didn't get the impression that the author was suggesting to throw out the old APIs. It seems to me like the article is a proof of concept of new approaches that could be added as new APIs, only expected to be used by people who need them, using an approach that takes advantage of modern storage technology. > "Random access may have to wastefully read larger blocks of the data than are actually requested by the application. The unused data gets cached, but if it's not going to be accessed any time soon, it means that something else got wastefully bumped out of the cache. Sequential access is likely to make use of an entire block." I may have misread it, but I thought he addressed this in the article. > "Random access files take a position as an argument, meaning there is no need to maintain a seek cursor. But more importantly: they don’t take a buffer as a parameter. Instead, they use io_uring’s pre-registered buffer area to allocate a buffer and return to the user. That means no memory mapping, no copying to the user buffer — there is only a copy from the device to the glommio buffer and the user get a reference counted pointer to that. And because we know this is random I/O, there is no need to read more data than what was requested."
Cojen 6y ago
I found this to be a good read, but I wish the author discussed the pros/cons of bypassing the file system and using a block device with direct I/O. I've found that with Optane drives the performance is high enough that the extra load from the file system (in terms of CPU) is significant. If the author was using a file system (which I assume is the case) which was it?
papi_bichhu 6y ago
For a moment, I thought it's my personal list of "I don't know these"
quelsolaar 6y ago
This is such a big deal! The assumptions made when IO APIs where designed are so out-of-step with today's hardware that it really is a time to have a big rethink. In graphics, the last 20 years of API development have very much been focused on harnessing a GPU that have again and again outgrown the CPUs ability to feed it. So much have been learned, and we really need to apply this to both storage and networking.
amelius 6y ago
What I hate about Unix filesystems: the fact that you can't take a drive, put it in another computer and have permissions (user/group-ids) working instantly. Same for sharing over nfs. Of course, people have tried to solve this, but I think not well enough. It's a huge amount of technical debt right there in the systems we use every day.
- bufferoverflow 6y ago
  Because permissions is an OS-enforced feature, not the filesystem feature. If you have access to the drive, permissions are meaningless.
  - amelius 6y ago
    Permissions (and ownership info) can be useful even if you have complete access to a filesystem. By the way, assume you have root permission. How would you replace a single file in a random tar-file, without changing any of the permissions/userids/groupids inside the tar-file? You can't untar it because the users inside the tar file don't correspond with the ones on your system. So, you'll have to use special tools, which is (only) one demonstration of the inadequacy of the permissions mechanism of our filesystems.
    - pixl97 6y ago
      Don't the files on disk just keep the current uid/gid anyway. You'd change one file and zip it back up. Seen plenty of files with just a number instead of an owner in the past.
      - amelius 6y ago
        There are two problems with this. 1. Tar also stores user and group names, which you see when you use the "v" and "t" options. 2. If you try this as non-root user, you run into permission problems.
    - wokkel 6y ago
      No, you use tar --numeric-owner and forget about the names.
forgotmypw17 6y ago
Because I am a "glue" programmer, and I realize that all storage options suck, I've decided to wait on any infrastructure choices for now, and just use the filesystem as a key-value store when developing my projects. When I need indexing, I use SQLite, but I limit myself to very basic subset of SQL that would work in any of Oracle, Maria, Microsoft stores without changes.
LeoNatan25 6y ago
This is why I am excited about DirectStorage, hoping it can usher a new era of IO APIs. We’ll see. https://devblogs.microsoft.com/directx/directstorage-is-coming-to-pc/ https://devblogs.microsoft.com/directx/directstorage-is-comi...
scottlamb 6y ago
Nice article. This part puzzled me though: > The operating system reads data in page granularity, meaning it can only read at a minimum 4kB at a time. That means if you need to read read 1kB split in two files, 512 bytes each, you are effectively reading 8kB to serve 1kB, wasting 87% of the data read. SSDs (whether SATA or NVMe) all read and write whole sectors at a time, right? I'm not sure what the sector size is, but 4 KiB seems like a reasonable guess. So I think you're reading the 8 KiB no matter what; it may just be a question of what layer you drop it at (right when it gets to the kernel or not). Also, doesn't direct IO require sector size-aligned operations?
- deleted 6y ago
  [deleted]
- wtallis 6y ago
  All SATA SSDs and almost all NVMe SSDs use 512-byte LBAs by default, but many NVMe SSDs can be reconfigured to use 4096-byte LBAs by default. The underlying NAND flash memory these days typically has a native page size of 16kB.
  - scottlamb 6y ago
    Thanks. What are the tradeoffs with NVMe sector size? There must be some reason to want 4096 if they added that option. Is there less write amplification as you get closer to the native page size or does the firmware avoid that anyway with its write leveling?
dkdk8283 6y ago
I think this is a loaded article and underscores the importance of low level engineers who understand your workload to guide purchasing. No longer will fringe benefits and bribes be enough.
AtlasBarfed 6y ago
AWS EBS gp2 is far from an NVMe drive. Everyone architects for AWS these days. So the limitations of EBS still dominate I/O: limited IOPS, limited bandwidth. Yes there are ephemerals. Which are basically little more than ramdisks/caches to actual AWS design.
sriku 6y ago
John Ousterhout, in the RAMCloud project, bet that 2.5μs latency for a read across the network would be possible. NVMe-s seem to be inching towards that too - like < 20μs at this point. Would be interesting to see how these numbers play out.
dr_j_ 6y ago
I stopped reading this when it became evident that the author was just promoting a library he’d written. I might have been a tiny bit more interested if I were a Rust developer.
tutfbhuf 6y ago
I don't think that io_uring is a bad API, developers just have to start using it.

512 points by harporoeder 6y ago

From the author's previous piece: https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/ https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-wi... > Our CTO, Avi Kivity, made the case for async at the Core C++ 2019 event. The bottom line is this; in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another. There are good reasons why network programming is done asynchronously, and you should consider that for your own application development too. > > It fundamentally changes the way Linux applications are to be designed: Instead of a flow of code that issues syscalls when needed, that have to think about whether or not a file is ready, they naturally become an event-loop that constantly add things to a shared buffer, deals with the previous entries that completed, rinse, repeat. As someone that's been working on FRP related things for a while now, this feels very vindicating. :) I few like as recently as a few years ago, the systems world was content with it's incremental hacks, but now the gap between the traditional interfaces and hardware realities has become too much, and bigger redesigning is afoot. Excited for what emerges!

mehrdadn 6y ago
> in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another. Interesting take, and NUMA CPUs have felt networked to me when I've used them, but typical multicore UMA CPUs sure haven't... is there a reason to believe this will change (or already has), or did the author mean to only talk about NUMA?
- ddorian43 6y ago
  Both, even multicore. Same ideas are used by Red Panda (open source kafka++ clone).
- jeffbee 6y ago
  I guess it depends on what you consider to be "typical". If you look at a many-core chip from AMD you'll see a gradient of access latency from one core to another. You'll see the same on any Intel Skylake-X descendant, although the slope of that gradient is less. Your software will need to be very highly optimized already before you start to sweat the difference, though.
  - masklinn 6y ago
    Here's what jeffbee is talking about: https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-dive-review-5950x-5900x-5800x-and-5700x-tested/5 https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-di...
    - jeffbee 6y ago
      Yep. If you are sweating a microsecond, 100 nanoseconds is significant chunk of your budget. For this and possibly other reasons, a many-core CPU isn't always a great choice for hosted storage. If your goal is to export NVMe blocks over a network interface, you might be better off with an easier-to-program 4- or 8-core CPU. I don't like seeing 128 cores and a bunch of NVMe devices in the same box because it just causes trouble.
      - magicalhippo 6y ago
        With p2pdma[1], it seems you don't even need a fancy CPU to push GB/s. [1]: https://www.youtube.com/watch?v=LDOlqgUZtHE https://www.youtube.com/watch?v=LDOlqgUZtHE (9:40)
hinkley 6y ago
> that have to think about whether or not a file is ready, they naturally become an event-loop that constantly add things to a shared buffer, deals with the previous entries that completed, rinse, repeat. This is the exception, not the rule, and it bugs me when APIs default to this. Most consumers of data are not looking at the stream of data, and in many cases where streaming is what I want, there are tools and APIs for handling that outside of my application logic. Much of the time I’m dealing with units of data only after the entire unit has arrived. Because if the message is not complete there is no forward progress to be made. My tools should reflect that reality, not what’s quickest for the API writers to create. In fact, if I remember my queuing theory properly, responsiveness is improved if the system prioritizes IO operations that can be finished (eg, EOS, EOF) over processing buffers for one that is still in the middle, which can’t happen with an event stream abstraction.
pjmlp 6y ago
The problem is forcing everyone to program in asynchronous way. WinRT tried to go down that route (only asynchronous APIs), to drive developers into that path, but eventually they had to support synchronous as well due to the received resistence.
- justicezyx 6y ago
  Before Herb Sutter's free lunch is over, not many writes in multi-threading code. Then today, everyone is writing multi-threading code, one way or another; and majority indirectly in newer languages like Go and Rust, or better frameworks like Actors, message passing, coroutines, and some noble souls who are capable enough, in classic pthread and other threading APIs. Of course everyone is going to program in an async way; that's how nature works. But it certainly will not be in a fashinon that is repulsive to you, just give it some time. Maybe 10 years.
  - pjmlp 6y ago
    Agreed, it is also why Microsoft was one of the biggest contributors to the co-routines support in C++, and have frameworks like Orleans and Coyote.

ivoras 6y ago

> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of flash-based storage just yet.

gravypod 6y ago
Depending on the storage technology the comparison to RAM is not that far off. Intel is trying to market it that way any way [0]. It's obviously not RAM but it's not the <500GB 5200RPM SATA 3GB/s disk I started programming on. [0] - https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html https://www.intel.com/content/www/us/en/architecture-and-tec...
- smcameron 6y ago
  Yeah, back in 2014, I worked at HP on storage drivers for linux, and we got 1 million IOPS (4k random reads) on a single controller, with SSDs, but we had to do some fairly hairy stuff. This was back when NVME was new and we were trying to do SCSI over PCIe. We set up multiple ring buffers for command submission and command completion, one each per CPU, and pinned threads to CPUs and were very careful to avoid locking (e.g. spinlocks, etc.). I think we also had to pin some userland processes to particular CPUs to avoid NUMA induced bottlenecks. The thing is, up until this point, for the entire history of computers, storage was so relatively slow compared to memory and the CPU that drivers could be quite simple, chuck requests and completions into queues managed by simple locking, and the fraction of time that requests spent inside the driver would still be negligible compared to the time they spent waiting for the disks. If you could theoretically make your driver infinitely fast, this would only amount to maybe a 1% speedup. So there was no need to spend a lot of time thinking about how to make the driver super efficient. Until suddenly there was.
  - smcameron 6y ago

blindm 6y ago

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

rcxdude 6y ago
I've had one (very early and cheap) SSD fail on me. Other than that I don't think I've seen or heard of any issues across a large range of more modern SSDs. The reliability and endurange issues which occured on earlier SSDs no longer seem to be a problem (this is in part because flash density has skyrocketed: because each flash chip can operate more or less independently, the more storage an SSD has the faster it can run and the more write endurance it has).
tjoff 6y ago
The solution for this problem is raid. Way cheaper and far superior in terms of reliability than your solution. Or if that isn't an option (laptop?) a good backup-solution that that runs daily or more often is also a better and cheaper alternative. Drives may dail at any time, but they don't age the way your post would suggest.
- blindm 6y ago
  I've looked into RAID. It seems a bit complicated to use. Is it trivial to create a RAID array in Linux with zero fuss and the whole thing 'just working' with very little knowledge of the filesystem itself other than it keeps your data 'safe' and redundancy baked in?
  - dfinninger 6y ago
    Do you mean “use” or “setup”? RAID is trivial to use. Mount the volume to a directory and use it like normal. The setup is a bit more involved, but really not that bad. It’s a couple commands to join a few disks in an array and then you make a file system and mount it. https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-crea...

kibwen 6y ago

Prior discussion from /r/rust where the author is present to answer questions: https://www.reddit.com/r/rust/comments/k16j6x/modern_storage_is_plenty_fast_it_is_the_apis_that/ https://www.reddit.com/r/rust/comments/k16j6x/modern_storage...

jeroenhd 6y ago

I suppose _modern_ storage is fast, but how many servers are running on storage this modern? None of mine are and my work dev machine is still rocking a SATA 2.5" SSD. We're probably still a few years off from being able to switch to this fast I/O yet. With the new game consoles switching over to PCIe SSDs I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Even with SATA drives like mine though, there's really not that much performance loss from doing IO operations. I've run my OS with 8GiB of SSD swap in active use during debugging and while the stutters are annoying and distracting, the computer didn't grind to a halt like it would with spinning rust. Storage speed has increased massively in the last five years, for the love of god fellow developers, please make use of it when you can! That said, deferring IO until you're done still makes sense for some consumer applications because cheap laptops are still being sold with hard drives and those devices are probably the minimum requirement you'll be serving.

rcxdude 6y ago
Well, if the whole async 'I/O is the bottleneck' principle which was the refrain from a few years ago is actually true, then servers running databases should be focusing on upgrading their storage to these levels, since that's where the most bang for buck comes from in terms of performance gain is (of course now the big thing is running everything on clouds like AWS where everything is dog slow and really expensive, so perhaps it doesn't actually matter). (In fact the main reason for the API changes covered in the article is because the CPU and RAM can no longer run laps around storage).
scribu 6y ago
> how many servers are running on storage this modern? AWS allows you to provision EC2 instances with NVMe, without much fanfare. Cost is only ~20% more than SATA.
wtallis 6y ago
> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

mehrdadn 6y ago

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this piece I had the immense pleasure of getting early access to one of the next generation Optane devices, from Intel. The entire blog post is complaining about how great engineers have misconception about modern storage technology and yet to prove it the author had to obtain benchmarks from early access to next-generation devices...?! And to top it off, from this we conclude "the disconnect" is due to the APIs? Not, say, from the possibility that such blazing-fast components may very well not even exist in users' devices? I'm not saying the conclusions are wrong, but the logic surely doesn't follow... and honestly it's a little tasteless to criticize people's understanding if you're going to base the criticism on things they in all likelihood don't even have access to.

arka2147483647 6y ago
I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.
- Miraste 6y ago
  Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.
  - mehrdadn 6y ago
    3 years I think: https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint > It was announced in July 2015 and is available on the open market under brand names Optane (Intel) and subsequently QuantX (Micron) since April 2017. For comparison, look at how many decades it took SSDs to become commonplace: https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_SSDs https://en.wikipedia.org/wiki/Solid-state_drive#Flash-based_...

deleted 6y ago

[deleted]

Thaxll 6y ago

How those modern API runs on old HW?

jlokier 6y ago
> How those modern API runs on old HW? Very well. The API Glauber (the author) describes has always been a good API for performance even with old HDDs and old CPUs. In fact with very old CPUs the reduction in memory copies and system calls was more significant than it is now. The extra control over memory use with direct I/O is beneficial on some lower-end embedded systems too, e.g. video streaming from HDD on low memory systems. It's just that before fast SSDs, there wasn't as much motivation to get the "best" I/O performance via complex APIs, because most of the time overhead was dominated by the HDD performance itself. (This was less true for big, fast RAIDs though.) There was some benefit in a small amount of parallelism with HDDs, to improve block sorting, but that was usually best achieved with a few threads or processes, which is still a simple API to use, just read() and write() syscalls. The only things really worth doing a complex API for before were O_DIRECT+AIO together, and even then they need a well-written, I/O-aware application to make them really worth using. In general, those applications have tended to be databases and VM hypervisors, but I've also seen it done for optimized file streaming to/from HDD on embedded systems. Though beneficial, the Linux AIO implementation had problems for a long time, even with O_DIRECT in some cases (such as filling holes in sparse files and extending files); so it was never reliably "true" async I/O. And there was no memory-buffer transfer and system call elision as there is with io_uring. Now there is more motivation, so the API has been improved. O_DIRECT+AIO+io_uring is a better combination. The benefits are increasingly worth the effort for more kinds of applications due to the faster SSDs, and the SSDs' ability to handle large I/O request queues. But they would have been a good API combination for performance 20 years ago too.

bob1029 6y ago

One thing I have started to realize is that best case latency of an NVMe storage device is starting to overlap with areas where SpinWait could be more ideal than an async/await API. I am mostly advocating for this from a mass parallel throughput perspective, especially if batching is possible. I have started to play around with using LMAX Disruptor for aggregating a program's disk I/O requests and executing them in batches. This is getting into levels of throughput that are incompatible with something like what the Task abstractions in .NET enable. The public API of such an approach is synchronous as a result of this design constraint. Software should always try to work with the physical hardware capabilities. Modern SSDs are most ideally suited to arrangements where all data is contained in an append-only log with each batch written to disk representing a consistent snapshot. If you are able to batch thousands of requests into a single byte array of serialized modified nodes, you can append this onto disk so much faster than if you force the SSD to make individual writes per new/modified entity.

wtallis 6y ago
On Linux, it's already a NVMe driver option to enable polling for (high priority) IO completion rather than sleeping until an interrupt. The latency of handling an interrupt and doing a couple of context switches is higher than the best-case latency for fast SSDs. The io_uring userspace API also has a polling mode.

zackmorris 6y ago

I agree with the premise, but disagree with the conclusion. For a little background, my first computer was a Mac Plus around 1985, and I remember doing file copy tests on my first hard drive (an 80 MB) at over 1 MB/sec. If I remember correctly, SCSI could do 5 MB/sec copies clear back in the mid-80s. So until we got SSD, hard drive speed stayed within the same order of magnitude for like 30 years (as most of you remember): http://chrislawson.net/writing/macdaniel/2k1120cl.shtml http://chrislawson.net/writing/macdaniel/2k1120cl.shtml So the time to take our predictable deterministic synchronous blocking business logic into the maze of asynchronous promise spaghetti was a generation ago when hard drive speeds were two orders of magnitude slower than today. In other words, fix the bad APIs. Please don't make us shift paradigms. Now if we want to talk about some kind of compiled or graph-oriented way of processing large numbers of files performantly with some kind of async processing internally, then that's fine. Note that this solution will mirror whatever we come up with for network processing as well. That was the whole point of UNIX in the first place, to treat file access and network access as the same stream-oriented protocol. Which I think is the motive behind taking file access into the same problematic async domain that web development is having to deal with now. But really we should get the web back to the proven UNIX/Actor model way of doing things with synchronous blocking I/O.

tobias3 6y ago

Intuitively one should be able to approach the max speed for sequential reads via some tuning (queue/read_ahead_kb) even with the traditional, blocking posix interface. This would require a large enough read-ahead and large enough buffer size. Not poisoning the page cache/manually managing the page cache is an orthogonal issue and only relevant for some applications (and the additional memory copy barely makes a difference in OPs post). One advantage of using high level (Linux) kernel interfaces is that this "automatically" gets faster with newer Linux versions without a need of large application level changes. Maybe in a few years we'll have an extra cache layer, or it stores to persistent memory now. Linux will (slowly) improve and your application with it. This won't happen if it is specifically tuned for Direct I/O with Intel Optane in 2020. But yeah, random IO is (currently) another issue, and as said the usual advice is to avoid them. And with the old API this still holds. If one currently wants fast random IO one needs to use io_uring/aio (with Direct-IO) or just live with the performance not being optimal and hope that the page cache does more good than bad (like Postgresql).

jorangreef 6y ago
The page cache is not reliable, and actually does more bad than good, especially in the case of PostgreSQL: https://www.usenix.org/conference/atc20/presentation/rebello https://www.usenix.org/conference/atc20/presentation/rebello

kazinator 6y ago

This is a really poor article. Only in very rare circumstances can developers change the API's. API's are not "bad"; they are built to various important requirements. Only some of those requirements have to do with performance. > “Well, it is fine to copy memory here and perform this expensive computation because it saves us one I/O operation, which is even more expensive”. "I/O operation" in fact refers to the API call, not to the raw hardware operation. If the developer measured this and found it true, how can it be a misconception? It may be caused by a "bad" I/O API, but so what? The API is what it is. API's provide one requirement which is stability: keeping applications working. That is king. You can't throw out API's every two years due to hardware advancements. > “If we split this into multiple files it will be slow because it will generate random I/O patterns. We need to optimize this for sequential access and read from a single file” Though solid state storage doesn't have track-to-track seek times, the sequential-access-fast rule of thumb has not become false. Random access may have to wastefully read larger blocks of the data than are actually requested by the application. The unused data gets cached, but if it's not going to be accessed any time soon, it means that something else got wastefully bumped out of the cache. Sequential access is likely to make use of an entire block. Secondly, there is that API again. The underlying operating system may provide a read-ahead mechanism which reduces its own overheads, benefiting the application which structures its data for sequential access, even if there is no inherent hardware-level benefit. If there is any latency at all between the application and the hardware, and if you can guess what the application is going to read next, that's an opportunity to improve performance. You can correctly guess what the application will read if you guess that it is doing a sequential read, and the application makes that come true.

cahooon 6y ago
I didn't get the impression that the author was suggesting to throw out the old APIs. It seems to me like the article is a proof of concept of new approaches that could be added as new APIs, only expected to be used by people who need them, using an approach that takes advantage of modern storage technology. > "Random access may have to wastefully read larger blocks of the data than are actually requested by the application. The unused data gets cached, but if it's not going to be accessed any time soon, it means that something else got wastefully bumped out of the cache. Sequential access is likely to make use of an entire block." I may have misread it, but I thought he addressed this in the article. > "Random access files take a position as an argument, meaning there is no need to maintain a seek cursor. But more importantly: they don’t take a buffer as a parameter. Instead, they use io_uring’s pre-registered buffer area to allocate a buffer and return to the user. That means no memory mapping, no copying to the user buffer — there is only a copy from the device to the glommio buffer and the user get a reference counted pointer to that. And because we know this is random I/O, there is no need to read more data than what was requested."

Cojen 6y ago

I found this to be a good read, but I wish the author discussed the pros/cons of bypassing the file system and using a block device with direct I/O. I've found that with Optane drives the performance is high enough that the extra load from the file system (in terms of CPU) is significant. If the author was using a file system (which I assume is the case) which was it?

papi_bichhu 6y ago

For a moment, I thought it's my personal list of "I don't know these"

quelsolaar 6y ago

This is such a big deal! The assumptions made when IO APIs where designed are so out-of-step with today's hardware that it really is a time to have a big rethink. In graphics, the last 20 years of API development have very much been focused on harnessing a GPU that have again and again outgrown the CPUs ability to feed it. So much have been learned, and we really need to apply this to both storage and networking.

amelius 6y ago

What I hate about Unix filesystems: the fact that you can't take a drive, put it in another computer and have permissions (user/group-ids) working instantly. Same for sharing over nfs. Of course, people have tried to solve this, but I think not well enough. It's a huge amount of technical debt right there in the systems we use every day.

bufferoverflow 6y ago
Because permissions is an OS-enforced feature, not the filesystem feature. If you have access to the drive, permissions are meaningless.
- amelius 6y ago
  Permissions (and ownership info) can be useful even if you have complete access to a filesystem. By the way, assume you have root permission. How would you replace a single file in a random tar-file, without changing any of the permissions/userids/groupids inside the tar-file? You can't untar it because the users inside the tar file don't correspond with the ones on your system. So, you'll have to use special tools, which is (only) one demonstration of the inadequacy of the permissions mechanism of our filesystems.
  - pixl97 6y ago
    Don't the files on disk just keep the current uid/gid anyway. You'd change one file and zip it back up. Seen plenty of files with just a number instead of an owner in the past.
    - amelius 6y ago
      There are two problems with this. 1. Tar also stores user and group names, which you see when you use the "v" and "t" options. 2. If you try this as non-root user, you run into permission problems.
  - wokkel 6y ago
    No, you use tar --numeric-owner and forget about the names.

forgotmypw17 6y ago

Because I am a "glue" programmer, and I realize that all storage options suck, I've decided to wait on any infrastructure choices for now, and just use the filesystem as a key-value store when developing my projects. When I need indexing, I use SQLite, but I limit myself to very basic subset of SQL that would work in any of Oracle, Maria, Microsoft stores without changes.

LeoNatan25 6y ago

This is why I am excited about DirectStorage, hoping it can usher a new era of IO APIs. We’ll see. https://devblogs.microsoft.com/directx/directstorage-is-coming-to-pc/ https://devblogs.microsoft.com/directx/directstorage-is-comi...

scottlamb 6y ago

Nice article. This part puzzled me though: > The operating system reads data in page granularity, meaning it can only read at a minimum 4kB at a time. That means if you need to read read 1kB split in two files, 512 bytes each, you are effectively reading 8kB to serve 1kB, wasting 87% of the data read. SSDs (whether SATA or NVMe) all read and write whole sectors at a time, right? I'm not sure what the sector size is, but 4 KiB seems like a reasonable guess. So I think you're reading the 8 KiB no matter what; it may just be a question of what layer you drop it at (right when it gets to the kernel or not). Also, doesn't direct IO require sector size-aligned operations?

deleted 6y ago
[deleted]
wtallis 6y ago
All SATA SSDs and almost all NVMe SSDs use 512-byte LBAs by default, but many NVMe SSDs can be reconfigured to use 4096-byte LBAs by default. The underlying NAND flash memory these days typically has a native page size of 16kB.
- scottlamb 6y ago
  Thanks. What are the tradeoffs with NVMe sector size? There must be some reason to want 4096 if they added that option. Is there less write amplification as you get closer to the native page size or does the firmware avoid that anyway with its write leveling?

dkdk8283 6y ago

I think this is a loaded article and underscores the importance of low level engineers who understand your workload to guide purchasing. No longer will fringe benefits and bribes be enough.

AtlasBarfed 6y ago

AWS EBS gp2 is far from an NVMe drive. Everyone architects for AWS these days. So the limitations of EBS still dominate I/O: limited IOPS, limited bandwidth. Yes there are ephemerals. Which are basically little more than ramdisks/caches to actual AWS design.

sriku 6y ago

John Ousterhout, in the RAMCloud project, bet that 2.5μs latency for a read across the network would be possible. NVMe-s seem to be inching towards that too - like < 20μs at this point. Would be interesting to see how these numbers play out.

dr_j_ 6y ago

I stopped reading this when it became evident that the author was just promoting a library he’d written. I might have been a tiny bit more interested if I were a Rust developer.

tutfbhuf 6y ago

I don't think that io_uring is a bad API, developers just have to start using it.