5 ms·

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't imme

by mjb 2y ago

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux. Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap between happy case and worst case performance is so large. That makes benchmarking hard, adds to the risk of stability bugs, and complicates taming tail latency. It's behavior also varies a lot between OSs. It's also nice to see all the data in this post. Too many systems design conversations are just dueling assertions.

a-dub 2y ago
it outsources buffer management and user thread i/o scheduling to the kernel. for some use cases it's a great way to simplify implementation or boost performance. for others it may not perform as well. the blog post points (in my mind) at some more general advice when programming which is not to mix and match paradigms unless you really know what you're doing. if you want to do user space async io, cool. if using kernel features tickles your fancy, also cool. mixing both without a deep understanding of what's going on under the hood will probably give you trouble.
- rbanffy 2y ago
  Making it work asynchronously would require the compiler to split the memory access into two parts, a non-blocking IO dispatch and a blocking access to the mapped address. The OS would need to support that, however, and the language would need to keep track of what is a materialised array and what’s not.
  - a-dub 2y ago
    as i understand, mmap is only efficient because it can leverage hardware support for trapping into the kernel when a page needs to be loaded to satisfy an access attempt. i think adding software indirection to every access in the mapped region would be really slow. i think a better answer would be to impose more structure on the planned memory access, then maybe given some constraints (like say, "this loop is embarrassingly parallel") the system could be smarter about working on the stuff in ram first while the rest is loaded in.
    - rbanffy 2y ago
      > every access in the mapped region would be really slow. Would certainly be slower. The compiler would need to be aware we want this behaviour and split the access in two parts, one to trigger the page read and yield to the app’s async loop, and another to resolve the read when the page has loaded. This would only need to happen for explicitly marked asynchronous memory reads (doing that without hardware support for all memory reads would be painful).
    - gpderetta 2y ago
      Normal syscalls also "leverage hardware support for trapping into the kernel". Mmap is usually used because it is a simple way to do 0-copy disk I/O.
      - a-dub 2y ago
        i can't think of any other syscall that makes use of tlb caches and page fault machinery to enter the kernel as needed in response to ordinary user space memory access.
  - noctune 2y ago
    I think you could make by with some kind of async memory-touch system call, i.e. "page in this range of memory, notify me when finished". The application would have to call this on blocks of the mmap prior to actually reading it. This of course means you lose some of the benefits of mmap (few system calls, automatic paging), but would maybe still be beneficial from a performance perspective.
    - rbanffy 2y ago
      It would allow a memory read to yield to the async loop, but overall performance of the read itself would always be lower. It’s the kind of thing that would be better implemented as a special “async buffer” where reads are guarded by a page fault handler that returns as soon as the read is scheduled and a read that yields on an unresolved page load.
    - gpderetta 2y ago
      io_uring + madvise is probably the the closest solution. Although if you are using uring, there are other options for async disk Io.
cbsmith 2y ago
I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared libraries are often memory mapped into processes), so page faults are a blocking hazard for cooperative schedulers even without mmap. The MMU in the hardware is aggressively parallel, and the only thread being blocked on the page fault is the one touching the page that needs to be swapped in. In reality, you can get heavily parallelized IO using mmap (indeed, it works quite well when you have a ton of IO you'd like to execute in parallel).
- Moto7451 2y ago
  I think your point here can be more generalized. Why should someone expect reading memory to benefit from async code? The fact that the memory in this case has an access layer with exploitable latency is where the chatter about this stems from, but it misses the fundamental issue at hand. If this was a valid concept we’d have async memcpy interfaces.
  - gpderetta 2y ago
    It is not exactly async memory, but at the turn of the millennium a few unices experimented with scheduler activations: the kernel would upcall back into the application whenever a thread would block for any reason, allowing rescheduling of the user space thread. In the end, the complexity wasn't worth it at the time, bit it is possible that something like that could be brought back in the fitire
- mjb 2y ago
  > I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads Indeed. In practice, though, it's easier to write high performance servers and storage systems with async runtimes (like tokio) than with native threads, at least with the current state of the ecosystem. That's not for some fundamental reason - it's possible to get great threaded performance - just the current reality. So, whoever's fault this is, it's useful to have good evidence of this downside of async runtimes (and worth thinking about ways that OSs could let runtimes know when they were about to block on IO).

by mjb 2y ago

a-dub 2y ago

it outsources buffer management and user thread i/o scheduling to the kernel. for some use cases it's a great way to simplify implementation or boost performance. for others it may not perform as well. the blog post points (in my mind) at some more general advice when programming which is not to mix and match paradigms unless you really know what you're doing. if you want to do user space async io, cool. if using kernel features tickles your fancy, also cool. mixing both without a deep understanding of what's going on under the hood will probably give you trouble.

rbanffy 2y ago
Making it work asynchronously would require the compiler to split the memory access into two parts, a non-blocking IO dispatch and a blocking access to the mapped address. The OS would need to support that, however, and the language would need to keep track of what is a materialised array and what’s not.
- a-dub 2y ago
  as i understand, mmap is only efficient because it can leverage hardware support for trapping into the kernel when a page needs to be loaded to satisfy an access attempt. i think adding software indirection to every access in the mapped region would be really slow. i think a better answer would be to impose more structure on the planned memory access, then maybe given some constraints (like say, "this loop is embarrassingly parallel") the system could be smarter about working on the stuff in ram first while the rest is loaded in.
  - rbanffy 2y ago
    > every access in the mapped region would be really slow. Would certainly be slower. The compiler would need to be aware we want this behaviour and split the access in two parts, one to trigger the page read and yield to the app’s async loop, and another to resolve the read when the page has loaded. This would only need to happen for explicitly marked asynchronous memory reads (doing that without hardware support for all memory reads would be painful).
  - gpderetta 2y ago
    Normal syscalls also "leverage hardware support for trapping into the kernel". Mmap is usually used because it is a simple way to do 0-copy disk I/O.
    - a-dub 2y ago
      i can't think of any other syscall that makes use of tlb caches and page fault machinery to enter the kernel as needed in response to ordinary user space memory access.
- noctune 2y ago
  I think you could make by with some kind of async memory-touch system call, i.e. "page in this range of memory, notify me when finished". The application would have to call this on blocks of the mmap prior to actually reading it. This of course means you lose some of the benefits of mmap (few system calls, automatic paging), but would maybe still be beneficial from a performance perspective.
  - rbanffy 2y ago
    It would allow a memory read to yield to the async loop, but overall performance of the read itself would always be lower. It’s the kind of thing that would be better implemented as a special “async buffer” where reads are guarded by a page fault handler that returns as soon as the read is scheduled and a read that yields on an unresolved page load.
  - gpderetta 2y ago
    io_uring + madvise is probably the the closest solution. Although if you are using uring, there are other options for async disk Io.

cbsmith 2y ago

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared libraries are often memory mapped into processes), so page faults are a blocking hazard for cooperative schedulers even without mmap. The MMU in the hardware is aggressively parallel, and the only thread being blocked on the page fault is the one touching the page that needs to be swapped in. In reality, you can get heavily parallelized IO using mmap (indeed, it works quite well when you have a ton of IO you'd like to execute in parallel).

Moto7451 2y ago
I think your point here can be more generalized. Why should someone expect reading memory to benefit from async code? The fact that the memory in this case has an access layer with exploitable latency is where the chatter about this stems from, but it misses the fundamental issue at hand. If this was a valid concept we’d have async memcpy interfaces.
- gpderetta 2y ago
  It is not exactly async memory, but at the turn of the millennium a few unices experimented with scheduler activations: the kernel would upcall back into the application whenever a thread would block for any reason, allowing rescheduling of the user space thread. In the end, the complexity wasn't worth it at the time, bit it is possible that something like that could be brought back in the fitire
mjb 2y ago
> I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads Indeed. In practice, though, it's easier to write high performance servers and storage systems with async runtimes (like tokio) than with native threads, at least with the current state of the ecosystem. That's not for some fundamental reason - it's possible to get great threaded performance - just the current reality. So, whoever's fault this is, it's useful to have good evidence of this downside of async runtimes (and worth thinking about ways that OSs could let runtimes know when they were about to block on IO).