10 ms·

Async hazard: MMAP is blocking IO

106 points by mmastrac 2y ago

PaulHoule 2y ago
No secret. Reading from memory is synchronous and always has been, at least in a normal computer. (Sometimes I think of how you could fit a fancy memory controller in a transport triggered architecture but that’s something different)
- 01HNNWZ0MV43FF 2y ago
  By blocking they mean that it can take ballpark non-volatile storage times instead of ballpark RAM times
  - xoranth 2y ago
    I believe they mean that since it bypasses the (Tokio) scheduler, so if you use it in async code you lose the main benefit of async code (namely, the scheduler is able to switch to some other task while waiting for IO to complete.). Basically the same behavior you'd get if you called a blocking syscall directly.
  - guerrilla 2y ago
    GP is aware. mmap makes files act like memory. Memory is always synchronous, thus blocking, so mmaped files are always blocking. I'm surprised OP even found this surprising. It should be completely obvious.
    - dataflow 2y ago
      At first I thought the title meant the mmap() call itself blocks, which I figured could be slightly surprising. But it seems they're referring to I/O on the mapped file? I'm also baffled, how could it possibly not block?
      - usefulcat 2y ago
        > how could it possibly not block? When the bytes being read are already in the cache. Hence the later part of the article where the author shows that reading mapped memory can be significantly faster.
        magicalhippo 2y ago
        It still blocks. It just completes orders of magnitudes faster.
        icedchai 2y ago
        Do you consider reading from a normal array (one not backed by a memory mapped file) to also be blocking?
        okr 2y ago
        Ha. Exactly.
        magicalhippo 2y ago
        In the languages and platforms I use, absolutely yes. Do you have some examples where a normal memory read is async?
        icedchai 2y ago
        Your definition of blocking is a bit different from my own. Synchronous is not always blocking. If the data is there, ready to go, there is no "blocking." If you consider all memory reads to be "blocking", then everything must be "blocking". The executable code must, after all, be read by the processor. In an extreme case, the entire executable could be paged out to disk! This interpretation is not what most people mean by "blocking."
        magicalhippo 2y ago
    - jerf 2y ago
      The term "blocking" has diverged between various communities and it is important to recognize those differences or you'll have dozens of people talking past each other for hundreds of messages as they all say "blocking" and think they mean the same thing, and then get very confused and angry at all the other people who are so obviously wrong (and in their context, they are) but just can't see it. It is obvious that a given "execution context", which is my generalized term for a thread and an async job and anything else of a similar nature, when it reaches for a value from an mmap'd file will be blocked until it is available. However, different communities have different ideas of an "execution context". Threaded language users tend to see them as threads, so while a given thread may be blocked the rest of the program can generally proceed. (Although historically the full story around file operations and what other threads can proceed past has been quite complicated.) Async users on the other hand are surprised here because the operation is blocking their entire executor, even though in principle it ought to be able to proceed with some other context. Because it's invisible to the executor, it isn't able to context-switch. In this case, the threaded world view is reasonably "obvious" but it can be non-obvious that a given async environment may not be able to task switch and it may freeze an entire executor, and since "one executor" is still a fairly common scenario, the entire OS process. (I am expressing no opinion about whether it must block an executor. System calls come with a lot of flags nowadays and for all I know there's some way an async executor could "catch" a mapped access and have an opportunity to switch. I am taking the original article's implicit claim that there isn't one happening in their particular environment at face value.) As long as you do not distinguish how various communities use the term "blocking", you will get very, very deeply nested threads full of arguments about something that, if you just are careful with your terminology, isn't complicated for anyone from any subculture to understand.
      - deleted 2y ago
        [deleted]
    - dbaupp 2y ago
      Yeah, of course a synchronous call that might block the thread is blocking, I agree... but, if I didn't have the context of "we're in a comment thread about a blog post about mmap", I'm pretty sure I wouldn't flag `x[i]` on `x: &[u8]` (or any other access) as a synchronous call that is worth worrying about. Hence the discussion of subtlety in https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a-distance https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a... It's obvious when pointed out, but I don't think it's obvious without the context of "there's an mmap nearby". (Author here, this behaviour wasn't surprising to me, but felt subtle enough to be worth investigating, and get some concrete data.)
  - bhawks 2y ago
    Same thing would happen if your memory has been paged to disk too.
- xoranth 2y ago
  On Linux, you might be able to use userfaultfd to make it async...
  - dividuum 2y ago
    I don't see how that would work. The memory access causing the page fault still blocks, but now another thread handles paging in the requested data. So without coordination between those two, nothing really changes. Sounds easier to just use nonblocking reads directly. Thanks for the pointer to userfaultfd. Didn't know that existed.
    - cmrdporcupine 2y ago
      Yeah. Part of the problem is that userfaultfd isn't itself quite flexible enough. What you might want to do is release the faulted thread to do some other work, letting it "know" what it can come back later when the data is available, but there's no mechanism to make that happen. Instead it's going to be entirely blocked until the fault can be resolved.
mjb 2y ago
I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap between happy case and worst case performance is so large. That makes benchmarking hard, adds to the risk of stability bugs, and complicates taming tail latency. It's behavior also varies a lot between OSs. It's also nice to see all the data in this post. Too many systems design conversations are just dueling assertions.
- a-dub 2y ago
  it outsources buffer management and user thread i/o scheduling to the kernel. for some use cases it's a great way to simplify implementation or boost performance. for others it may not perform as well. the blog post points (in my mind) at some more general advice when programming which is not to mix and match paradigms unless you really know what you're doing. if you want to do user space async io, cool. if using kernel features tickles your fancy, also cool. mixing both without a deep understanding of what's going on under the hood will probably give you trouble.
  - rbanffy 2y ago
    Making it work asynchronously would require the compiler to split the memory access into two parts, a non-blocking IO dispatch and a blocking access to the mapped address. The OS would need to support that, however, and the language would need to keep track of what is a materialised array and what’s not.
    - a-dub 2y ago
      as i understand, mmap is only efficient because it can leverage hardware support for trapping into the kernel when a page needs to be loaded to satisfy an access attempt. i think adding software indirection to every access in the mapped region would be really slow. i think a better answer would be to impose more structure on the planned memory access, then maybe given some constraints (like say, "this loop is embarrassingly parallel") the system could be smarter about working on the stuff in ram first while the rest is loaded in.
Retr0id 2y ago
> This is thus a worst case, the impact on real code is unlikely to be quite this severe! I think the actual worst-case would be to read the pages in a (pseudo-)random order.
- dbaupp 2y ago
  (Author here.) Good point. Do you think the difference will be observable? Will it be observable on an SSD (vs. HDD)?
  - Retr0id 2y ago
    I think so, yes - only one way to find out, though!
lowbloodsugar 2y ago
- register - shadow register - L1 - L2 - L3 - RAM - GPU/SPU/RSP - SSD - Network - HDD The line is drawn depending on what you are doing and how. Edit: moved Network above HDD. :-)
- tux3 2y ago
  I would put networks above HDDs (depending on how many miles you need to send your emails)
  - mjb 2y ago
    Modern data center networks offer RTTs about 100x lower than hard drive latency, and comparable to local SSD. It depends, of course, how far over the network you're going, and how fast the other side responds, but <100us is very achievable.
correnos 2y ago
IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.
- deleted 2y ago
  [deleted]
- dan-robertson 2y ago
  I don’t find this argument super strong, fwiw. It could just mean ‘be wary of doing blocking operations with async, and note map makes reading memory blocking (paging in) and writing memory blocking (CoW pages)’ I think there are reasons to be wary but to me, debugging comes first (this goes two ways though: if you have a single ‘actual’ thread then many races can’t happen) because debuggers/traces/… work better on non-async code. Performance comes second but it’s complicated. The big cost with threads is heavy context switches and per-thread memory. The big cost with async is losing cpu locality (because many syscalls on Linux won’t lead to your thread yielding, and the core your thread is on will likely have more of the relevant information and lots of cache to take advantage of when the syscall returns[1]) and spending more on coordination. Without io_uring, you end up sending out your syscall work (nonblocking fd ops excepted) to some thread pool to eventually pick up (likely via some futex) load into cache, send to the os on some random core, and then send back to you in a way that you will notice such that the next step can be (internally) scheduled. It can be hard to keep a handle on the latency added by all that indirection. The third reason I have to be wary of async is that it can be harder to track resource usage when you have a big bag of async stuff going on at once. With threads there is some sense in which you can limit per-thread cost and then limit the number of threads. I find this third reason quite weak. All that said, it seems pretty clear that async provides a lot of value, especially for ‘single-threaded’ (I use this phrase in a loose sense) contexts like JavaScript or Python where you can reduce some multithreading pain. And I remain excited for io_uring based async to pick up steam. [1] there’s this thing people say about the context switching in and out of kernel space for a syscall being very expensive. See for example the first graph here: https://www.usenix.org/legacy/events/osdi10/tech/full_papers/Soares.pdf https://www.usenix.org/legacy/events/osdi10/tech/full_papers... . But I think it isn’t really very true these days (maybe spectre & co mitigations changed that?) at least on Linux.
akira2501 2y ago
> How do other mmap/madvise options influence this (for instance, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_POPULATE, MADV_POPULATE_READ, mlock)? (Hypothesis: these options will make it more likely that data is pre-cached and thus fall into fast path more often, but without a guarantee.) That probably should have been the first thing to try. Too mad the mmap2 crate does not expose this. Also looking at the mmap2 crate, it chooses some rather opinionated defaults depending on which function you actually call, and it makes accessing things like HUGEPAGE maps somewhat difficult.. and for whatever reason includes the MMAP_STACK flag when you call through this path. I feel like a lot of rust authors put faith in crates that, upon inspection, are generally poorly designed and do not expose the underlying interface properly. It's a bad crutch for the language.
- dbaupp 2y ago
  > That probably should have been the first thing to try. The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!
  - akira2501 2y ago
    I think it's harder for you to write "correct code" here because the crate is hiding most of the actual detail from you. I put that in quotes because there's absolutely nothing incorrect about the code, it's really just suboptimal, and most probably because it can't even use the full syscall interface. Seriously, I hate to be a curmudgeon, but that crate looks like a particularly bad and naive wrapper around mmap. It works very hard to provide you things you don't need when the basic interface is much more flexible. Aside from having to put `unsafe` around the call and re-import the kernel header constants, there's almost no reason to even have this in a crate.
    - dbaupp 2y ago
      I have a feeling we’re talking at cross purposes here: I was actively trying to write incorrect code. This post isn’t about memmap2 crate specifically at all, it just happens to be a convenient way to get the exact (“incorrect”) syscall I wanted from Rust. I see where you’re coming from but… it feels like you’re trying to convince me of something about the post? If you feel like convincing a larger audience of the limitations of the memmap2 crate specifically, I suggest writing your own blog posts and/or getting involved with it. :)
pengaru 2y ago
water is wet
- mwcampbell 2y ago
  Nobody automatically knows everything, and we have limited energy for drawing inferences based on what we do know. So the material covered in this post isn't obvious to everyone in its target audience, especially since Rust has had some success in making systems programming more approachable to inexperienced programmers, which is a good thing.
  - dbaupp 2y ago
    (Author here) Thank you, you've expressed one of my goals with doing this sort of investigation/getting this data far better than I have. :) It's something that's feels obvious once the dots are connected, but I was pretty sure many people wouldn't connect these dots automatically.
malkia 2y ago
WIth mmap you have to be prepared to handle unexpected page fault errors due to corrupted volume: Unlike standard read/write, where one can handle the issue, now it can happen anywhere the memory is mapped - your code, third party library, etc. It gets even unwieldy, and now you have to add additional tracking where access is to be expected. Blindly delegating mmap area to any code path that does not have such handling, and you would have to deal with these failures. Maybe that's not the case on Linux/OSX/BSD, but definitely is on Windows where you would have it. Also in C/C++ land you have to handle this using SEH - e.g. `__try/__except` - standard C++ handling won't cut it (I guess in other systems these would be through some signals (?)). In any case, it might seem like an easy path to achieve glory, yet riddled with complications.
- krilovsky 2y ago
  Yes, on POSIX systems you'd get a SIGBUS if the I/O fails or if there's no available physical memory to back the mapping.
- jcalvinowens 2y ago
  On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. By your logic, passing a file descriptor to a library is also "unwieldy", because the library might not handle -EIO. You can use MAP_POPULATE|MAP_LOCKED to ensure you get an error from mmap() instead of getting killed in the ENOMEM case, if you don't overcommit (if you do, you can still be oom-killed). You still get SIGBUS beyond EOF, but that's the behavior you want: it's equivalent to overrunning a buffer. The behavior when file size isn't a multiple of PAGE_SIZE is legitimately weird (writes to the final page beyond EOF are visible to the entire system in memory but never written back to the file), but it's intuitive if you understand how the page cache works at a high level, and you can avoid it by making the size page aligned. For more complex usecases where you really do want to handle these sorts of errors, userfaultfd() gives you all the tools you need: https://www.man7.org/linux/man-pages/man2/userfaultfd.2.html https://www.man7.org/linux/man-pages/man2/userfaultfd.2.html EDIT: Initially described MAP_POPULATE wrong.
dsp_person 2y ago
> One possible implementation might be to literally have the operating system allocate a chunk of physical memory and load the file into it, byte by byte, right when mmap is called… but this is slow, and defeats half the magic of memory mapped IO: manipulating files without having to pull them into memory This doesn't defeat the purpose necessarily. How about for example, implementing a text editor: I want the best performance by loading the existing file initially (say it is <1MB), and the convenience and robustness of any writes to this memory being efficiently written to disk.
dathinab 2y ago
While the general point the article is making is correct there are some issues. - (minor issue) async example is artificially limited to 1 thread (the article states that). The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core.. So in general you should compare something async with num_cpus threads vs. num_cpus*X OS threads. Through this wouldn't have been that useful in this example without pinning the tokio async threads to CPUs to forcefully highlight the page issue, and doing it is bothersome so I wouldn't have done so either. - (bigger issue) The singled thread async "traditional IO" example is NOT single threaded. Async _file_ IO anything between not a thing or very bad in most OSes hence most async engines including tokio do file IO in worker threads. This means the "single threaded" conventional IO async example is running 8 threads for reading IO and one to "touch the buffer" (i.e. do hardly anything). To be clear the single threaded not being single threaded issue isn't discrediting the article, the benchmarks still show the problem it's that the 8 threaded conventional and 1 threaded async conventional are accidentally basically both 8 thraded.
- dbaupp 2y ago
  (Author here) > The singled thread async "traditional IO" example is NOT single threaded The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls. The key is that there's a single coordinator thread that's issuing all the work, with the user-space runtime multiplexing tasks on that thread. I thought the fact that the "start a request and come back when it is finished" behaviour happens to be implemented via user-space threads rather than kernel-level epoll (or similar) is unlikely to affect behaviour. I considered scaling up the number of files and using a multi-threaded runtime, but I felt that'd make everything more complicated without fundamentally changing behaviour. However, maybe my theory is incorrect, in which case someone else can do their own experiments to provide more concrete information. (This is referenced in a footnote: https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:tokio-single-threaded https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:t... ) > The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core The point is not to benchmark async vs. non-async, but provide a general reference point for "mmap working well" for comparison. As you suggest you agree with the "minor" issue tag, I don't think the parallelism vs. concurrency distinction matters much here... but again, definitely happy to see some concrete data that suggests otherwise!
charleshn 2y ago
See also the classic "Are You Sure You Want to Use MMAP in Your Database Management System?" which mentions this common pitfall of mmap, and others, in the context of DBMS. [0] https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf
colonwqbang 2y ago
Function calls are also blocking IO then because executables and libraries are mmapped.
rmholt 2y ago
While author said that C's mmap suffers the same issue, I would argue C's mmap is fine, because C doesn't have async. The issue arises from the mmap crate not having an async read and the confusion around how does async work.
- rmholt 2y ago
  I feel like author used a lot of words to say "mmaped reads are IO (obvious) but critically, they are usually not awaitable like IO (bad)"
- gpderetta 2y ago
  No, that misses the point. Async doesn't require an async keyword. Evented programming (which as the same issues) has been common in C for decades.
davesque 2y ago
I always thought that one of the use cases of memory mapping was to improve multiprocessing workloads, where a group of processes don't have to duplicate the same region of a working set. In that sense, maybe it's not surprising that single-threaded concurrency can't leverage all of the benefits of memory mapping.
dekhn 2y ago
I used to really like mmap for a wide range of uses (having noticed its performance in the BLAST DNA/protein search command) but over time I've come to consider it a true expert tool with deep subtlety, like a palantir.

106 points by mmastrac 2y ago

No secret. Reading from memory is synchronous and always has been, at least in a normal computer. (Sometimes I think of how you could fit a fancy memory controller in a transport triggered architecture but that’s something different)

01HNNWZ0MV43FF 2y ago
By blocking they mean that it can take ballpark non-volatile storage times instead of ballpark RAM times
- xoranth 2y ago
  I believe they mean that since it bypasses the (Tokio) scheduler, so if you use it in async code you lose the main benefit of async code (namely, the scheduler is able to switch to some other task while waiting for IO to complete.). Basically the same behavior you'd get if you called a blocking syscall directly.
- guerrilla 2y ago
  GP is aware. mmap makes files act like memory. Memory is always synchronous, thus blocking, so mmaped files are always blocking. I'm surprised OP even found this surprising. It should be completely obvious.
  - dataflow 2y ago
    At first I thought the title meant the mmap() call itself blocks, which I figured could be slightly surprising. But it seems they're referring to I/O on the mapped file? I'm also baffled, how could it possibly not block?
    - usefulcat 2y ago
      > how could it possibly not block? When the bytes being read are already in the cache. Hence the later part of the article where the author shows that reading mapped memory can be significantly faster.
      - magicalhippo 2y ago
        It still blocks. It just completes orders of magnitudes faster.
        icedchai 2y ago
        Do you consider reading from a normal array (one not backed by a memory mapped file) to also be blocking?
        okr 2y ago
        Ha. Exactly.
        magicalhippo 2y ago
        In the languages and platforms I use, absolutely yes. Do you have some examples where a normal memory read is async?
        icedchai 2y ago
        Your definition of blocking is a bit different from my own. Synchronous is not always blocking. If the data is there, ready to go, there is no "blocking." If you consider all memory reads to be "blocking", then everything must be "blocking". The executable code must, after all, be read by the processor. In an extreme case, the entire executable could be paged out to disk! This interpretation is not what most people mean by "blocking."
        magicalhippo 2y ago
  - jerf 2y ago
    The term "blocking" has diverged between various communities and it is important to recognize those differences or you'll have dozens of people talking past each other for hundreds of messages as they all say "blocking" and think they mean the same thing, and then get very confused and angry at all the other people who are so obviously wrong (and in their context, they are) but just can't see it. It is obvious that a given "execution context", which is my generalized term for a thread and an async job and anything else of a similar nature, when it reaches for a value from an mmap'd file will be blocked until it is available. However, different communities have different ideas of an "execution context". Threaded language users tend to see them as threads, so while a given thread may be blocked the rest of the program can generally proceed. (Although historically the full story around file operations and what other threads can proceed past has been quite complicated.) Async users on the other hand are surprised here because the operation is blocking their entire executor, even though in principle it ought to be able to proceed with some other context. Because it's invisible to the executor, it isn't able to context-switch. In this case, the threaded world view is reasonably "obvious" but it can be non-obvious that a given async environment may not be able to task switch and it may freeze an entire executor, and since "one executor" is still a fairly common scenario, the entire OS process. (I am expressing no opinion about whether it must block an executor. System calls come with a lot of flags nowadays and for all I know there's some way an async executor could "catch" a mapped access and have an opportunity to switch. I am taking the original article's implicit claim that there isn't one happening in their particular environment at face value.) As long as you do not distinguish how various communities use the term "blocking", you will get very, very deeply nested threads full of arguments about something that, if you just are careful with your terminology, isn't complicated for anyone from any subculture to understand.
    - deleted 2y ago
      [deleted]
  - dbaupp 2y ago
    Yeah, of course a synchronous call that might block the thread is blocking, I agree... but, if I didn't have the context of "we're in a comment thread about a blog post about mmap", I'm pretty sure I wouldn't flag `x[i]` on `x: &[u8]` (or any other access) as a synchronous call that is worth worrying about. Hence the discussion of subtlety in https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a-distance https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a... It's obvious when pointed out, but I don't think it's obvious without the context of "there's an mmap nearby". (Author here, this behaviour wasn't surprising to me, but felt subtle enough to be worth investigating, and get some concrete data.)
- bhawks 2y ago
  Same thing would happen if your memory has been paged to disk too.
xoranth 2y ago
On Linux, you might be able to use userfaultfd to make it async...
- dividuum 2y ago
  I don't see how that would work. The memory access causing the page fault still blocks, but now another thread handles paging in the requested data. So without coordination between those two, nothing really changes. Sounds easier to just use nonblocking reads directly. Thanks for the pointer to userfaultfd. Didn't know that existed.
  - cmrdporcupine 2y ago
    Yeah. Part of the problem is that userfaultfd isn't itself quite flexible enough. What you might want to do is release the faulted thread to do some other work, letting it "know" what it can come back later when the data is available, but there's no mechanism to make that happen. Instead it's going to be entirely blocked until the fault can be resolved.

mjb 2y ago

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap between happy case and worst case performance is so large. That makes benchmarking hard, adds to the risk of stability bugs, and complicates taming tail latency. It's behavior also varies a lot between OSs. It's also nice to see all the data in this post. Too many systems design conversations are just dueling assertions.

a-dub 2y ago
it outsources buffer management and user thread i/o scheduling to the kernel. for some use cases it's a great way to simplify implementation or boost performance. for others it may not perform as well. the blog post points (in my mind) at some more general advice when programming which is not to mix and match paradigms unless you really know what you're doing. if you want to do user space async io, cool. if using kernel features tickles your fancy, also cool. mixing both without a deep understanding of what's going on under the hood will probably give you trouble.
- rbanffy 2y ago
  Making it work asynchronously would require the compiler to split the memory access into two parts, a non-blocking IO dispatch and a blocking access to the mapped address. The OS would need to support that, however, and the language would need to keep track of what is a materialised array and what’s not.
  - a-dub 2y ago
    as i understand, mmap is only efficient because it can leverage hardware support for trapping into the kernel when a page needs to be loaded to satisfy an access attempt. i think adding software indirection to every access in the mapped region would be really slow. i think a better answer would be to impose more structure on the planned memory access, then maybe given some constraints (like say, "this loop is embarrassingly parallel") the system could be smarter about working on the stuff in ram first while the rest is loaded in.

Retr0id 2y ago

> This is thus a worst case, the impact on real code is unlikely to be quite this severe! I think the actual worst-case would be to read the pages in a (pseudo-)random order.

dbaupp 2y ago
(Author here.) Good point. Do you think the difference will be observable? Will it be observable on an SSD (vs. HDD)?
- Retr0id 2y ago
  I think so, yes - only one way to find out, though!

lowbloodsugar 2y ago

- register - shadow register - L1 - L2 - L3 - RAM - GPU/SPU/RSP - SSD - Network - HDD The line is drawn depending on what you are doing and how. Edit: moved Network above HDD. :-)

tux3 2y ago
I would put networks above HDDs (depending on how many miles you need to send your emails)
- mjb 2y ago
  Modern data center networks offer RTTs about 100x lower than hard drive latency, and comparable to local SSD. It depends, of course, how far over the network you're going, and how fast the other side responds, but <100us is very achievable.

correnos 2y ago

IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.

deleted 2y ago
[deleted]
dan-robertson 2y ago
I don’t find this argument super strong, fwiw. It could just mean ‘be wary of doing blocking operations with async, and note map makes reading memory blocking (paging in) and writing memory blocking (CoW pages)’ I think there are reasons to be wary but to me, debugging comes first (this goes two ways though: if you have a single ‘actual’ thread then many races can’t happen) because debuggers/traces/… work better on non-async code. Performance comes second but it’s complicated. The big cost with threads is heavy context switches and per-thread memory. The big cost with async is losing cpu locality (because many syscalls on Linux won’t lead to your thread yielding, and the core your thread is on will likely have more of the relevant information and lots of cache to take advantage of when the syscall returns[1]) and spending more on coordination. Without io_uring, you end up sending out your syscall work (nonblocking fd ops excepted) to some thread pool to eventually pick up (likely via some futex) load into cache, send to the os on some random core, and then send back to you in a way that you will notice such that the next step can be (internally) scheduled. It can be hard to keep a handle on the latency added by all that indirection. The third reason I have to be wary of async is that it can be harder to track resource usage when you have a big bag of async stuff going on at once. With threads there is some sense in which you can limit per-thread cost and then limit the number of threads. I find this third reason quite weak. All that said, it seems pretty clear that async provides a lot of value, especially for ‘single-threaded’ (I use this phrase in a loose sense) contexts like JavaScript or Python where you can reduce some multithreading pain. And I remain excited for io_uring based async to pick up steam. [1] there’s this thing people say about the context switching in and out of kernel space for a syscall being very expensive. See for example the first graph here: https://www.usenix.org/legacy/events/osdi10/tech/full_papers/Soares.pdf https://www.usenix.org/legacy/events/osdi10/tech/full_papers... . But I think it isn’t really very true these days (maybe spectre & co mitigations changed that?) at least on Linux.

akira2501 2y ago

> How do other mmap/madvise options influence this (for instance, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_POPULATE, MADV_POPULATE_READ, mlock)? (Hypothesis: these options will make it more likely that data is pre-cached and thus fall into fast path more often, but without a guarantee.) That probably should have been the first thing to try. Too mad the mmap2 crate does not expose this. Also looking at the mmap2 crate, it chooses some rather opinionated defaults depending on which function you actually call, and it makes accessing things like HUGEPAGE maps somewhat difficult.. and for whatever reason includes the MMAP_STACK flag when you call through this path. I feel like a lot of rust authors put faith in crates that, upon inspection, are generally poorly designed and do not expose the underlying interface properly. It's a bad crutch for the language.

dbaupp 2y ago
> That probably should have been the first thing to try. The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!
- akira2501 2y ago
  I think it's harder for you to write "correct code" here because the crate is hiding most of the actual detail from you. I put that in quotes because there's absolutely nothing incorrect about the code, it's really just suboptimal, and most probably because it can't even use the full syscall interface. Seriously, I hate to be a curmudgeon, but that crate looks like a particularly bad and naive wrapper around mmap. It works very hard to provide you things you don't need when the basic interface is much more flexible. Aside from having to put `unsafe` around the call and re-import the kernel header constants, there's almost no reason to even have this in a crate.
  - dbaupp 2y ago
    I have a feeling we’re talking at cross purposes here: I was actively trying to write incorrect code. This post isn’t about memmap2 crate specifically at all, it just happens to be a convenient way to get the exact (“incorrect”) syscall I wanted from Rust. I see where you’re coming from but… it feels like you’re trying to convince me of something about the post? If you feel like convincing a larger audience of the limitations of the memmap2 crate specifically, I suggest writing your own blog posts and/or getting involved with it. :)

pengaru 2y ago

water is wet

mwcampbell 2y ago
Nobody automatically knows everything, and we have limited energy for drawing inferences based on what we do know. So the material covered in this post isn't obvious to everyone in its target audience, especially since Rust has had some success in making systems programming more approachable to inexperienced programmers, which is a good thing.
- dbaupp 2y ago
  (Author here) Thank you, you've expressed one of my goals with doing this sort of investigation/getting this data far better than I have. :) It's something that's feels obvious once the dots are connected, but I was pretty sure many people wouldn't connect these dots automatically.

malkia 2y ago

WIth mmap you have to be prepared to handle unexpected page fault errors due to corrupted volume: Unlike standard read/write, where one can handle the issue, now it can happen anywhere the memory is mapped - your code, third party library, etc. It gets even unwieldy, and now you have to add additional tracking where access is to be expected. Blindly delegating mmap area to any code path that does not have such handling, and you would have to deal with these failures. Maybe that's not the case on Linux/OSX/BSD, but definitely is on Windows where you would have it. Also in C/C++ land you have to handle this using SEH - e.g. `__try/__except` - standard C++ handling won't cut it (I guess in other systems these would be through some signals (?)). In any case, it might seem like an easy path to achieve glory, yet riddled with complications.

krilovsky 2y ago
Yes, on POSIX systems you'd get a SIGBUS if the I/O fails or if there's no available physical memory to back the mapping.
jcalvinowens 2y ago
On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases. By your logic, passing a file descriptor to a library is also "unwieldy", because the library might not handle -EIO. You can use MAP_POPULATE|MAP_LOCKED to ensure you get an error from mmap() instead of getting killed in the ENOMEM case, if you don't overcommit (if you do, you can still be oom-killed). You still get SIGBUS beyond EOF, but that's the behavior you want: it's equivalent to overrunning a buffer. The behavior when file size isn't a multiple of PAGE_SIZE is legitimately weird (writes to the final page beyond EOF are visible to the entire system in memory but never written back to the file), but it's intuitive if you understand how the page cache works at a high level, and you can avoid it by making the size page aligned. For more complex usecases where you really do want to handle these sorts of errors, userfaultfd() gives you all the tools you need: https://www.man7.org/linux/man-pages/man2/userfaultfd.2.html https://www.man7.org/linux/man-pages/man2/userfaultfd.2.html EDIT: Initially described MAP_POPULATE wrong.

dsp_person 2y ago

> One possible implementation might be to literally have the operating system allocate a chunk of physical memory and load the file into it, byte by byte, right when mmap is called… but this is slow, and defeats half the magic of memory mapped IO: manipulating files without having to pull them into memory This doesn't defeat the purpose necessarily. How about for example, implementing a text editor: I want the best performance by loading the existing file initially (say it is <1MB), and the convenience and robustness of any writes to this memory being efficiently written to disk.

dathinab 2y ago

While the general point the article is making is correct there are some issues. - (minor issue) async example is artificially limited to 1 thread (the article states that). The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core.. So in general you should compare something async with num_cpus threads vs. num_cpus*X OS threads. Through this wouldn't have been that useful in this example without pinning the tokio async threads to CPUs to forcefully highlight the page issue, and doing it is bothersome so I wouldn't have done so either. - (bigger issue) The singled thread async "traditional IO" example is NOT single threaded. Async _file_ IO anything between not a thing or very bad in most OSes hence most async engines including tokio do file IO in worker threads. This means the "single threaded" conventional IO async example is running 8 threads for reading IO and one to "touch the buffer" (i.e. do hardly anything). To be clear the single threaded not being single threaded issue isn't discrediting the article, the benchmarks still show the problem it's that the 8 threaded conventional and 1 threaded async conventional are accidentally basically both 8 thraded.

dbaupp 2y ago
(Author here) > The singled thread async "traditional IO" example is NOT single threaded The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls. The key is that there's a single coordinator thread that's issuing all the work, with the user-space runtime multiplexing tasks on that thread. I thought the fact that the "start a request and come back when it is finished" behaviour happens to be implemented via user-space threads rather than kernel-level epoll (or similar) is unlikely to affect behaviour. I considered scaling up the number of files and using a multi-threaded runtime, but I felt that'd make everything more complicated without fundamentally changing behaviour. However, maybe my theory is incorrect, in which case someone else can do their own experiments to provide more concrete information. (This is referenced in a footnote: https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:tokio-single-threaded https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:t... ) > The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core The point is not to benchmark async vs. non-async, but provide a general reference point for "mmap working well" for comparison. As you suggest you agree with the "minor" issue tag, I don't think the parallelism vs. concurrency distinction matters much here... but again, definitely happy to see some concrete data that suggests otherwise!

charleshn 2y ago

See also the classic "Are You Sure You Want to Use MMAP in Your Database Management System?" which mentions this common pitfall of mmap, and others, in the context of DBMS. [0] https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

colonwqbang 2y ago

Function calls are also blocking IO then because executables and libraries are mmapped.

rmholt 2y ago

While author said that C's mmap suffers the same issue, I would argue C's mmap is fine, because C doesn't have async. The issue arises from the mmap crate not having an async read and the confusion around how does async work.

rmholt 2y ago
I feel like author used a lot of words to say "mmaped reads are IO (obvious) but critically, they are usually not awaitable like IO (bad)"
gpderetta 2y ago
No, that misses the point. Async doesn't require an async keyword. Evented programming (which as the same issues) has been common in C for decades.

davesque 2y ago

I always thought that one of the use cases of memory mapping was to improve multiprocessing workloads, where a group of processes don't have to duplicate the same region of a working set. In that sense, maybe it's not surprising that single-threaded concurrency can't leverage all of the benefits of memory mapping.

dekhn 2y ago

I used to really like mmap for a wide range of uses (having noticed its performance in the BLAST DNA/protein search command) but over time I've come to consider it a true expert tool with deep subtlety, like a palantir.