5 ms·

To me this indicates a limitation of the API. Cause you do want to maintain that the kernel can page out that memory under pressure while userspace accesses tha

To me this indicates a limitation of the API. Cause you do want to maintain that the kernel can page out that memory under pressure while userspace accesses that memory asynchronously while allowing the thread to do other asynchronous things. There’s no good programming model/OS api that can accomplish this today.

im3w1l 1y ago
There are apis that sort of let you do it: mincore, madvise, userfaultfd.
- bcrl 1y ago
  None of those APIs are cheap enough to call in a fast path.
  - gpderetta 1y ago
    no syscall will be cheap to call in a fast path. You would need an hardware instruction that tells you if a load or store would fault.
    - zozbot234 1y ago
      > You would need an hardware instruction that tells you if a load or store would fault. You have MADV_FREE pages/ranges. They get cleared when purged, so reading zeros tells you that the load would have faulted and needs to be populated from storage.
      - vlovich123 1y ago
        MADV_FREE is insufficient - userspace doesn’t get a signal from the OS to know when there’s system wide memory pressure and having userspace try to respond to such a signal would be counter productive and slow in a kernel operation that needs to be a fast path. It’s more that you want MADV (page cache) a memory range and then have some way to have a shared data structure where you are told if it’s still resident and can lock it from being paged out.
        zozbot234 1y ago
        > userspace doesn’t get a signal from the OS to know when there’s system wide memory pressure Memory pressure indicators exist, https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html > have some way to have a shared data structure where you are told if it’s still resident and can lock it from being paged out. What's more efficient than fetching data and comparing it with zero? Any write within the range will then cancel the MADV_FREE property on the written-to page thus "locking" it again, and this is also very efficient.
        bcrl 1y ago
        MADV_FREE is also extremely expensive. CPU vendors have finally simplified TLB shootdown in recent CPUs with both AMD and Intel now having instructions to broadcast TLB flushes in hardware, which gets rid of one of the worst sources of performance degradation in threaded multicore applications (oh the pain of IPIs mixed with TLB flushing!). However, it's still very expensive to walk page tables and free pages. Hardware reference counting of memory allocations would be very interesting. It would be shockingly simple to implement compared to many other features hardware already has to tackle.
        zozbot234 1y ago
        > MADV_FREE is also extremely expensive. It's quite expensive to free pages under memory pressure (though it's not clear that there's any other choice to be made), but if the pages are never freed it should be cheap, AIUI.
    - vlovich123 1y ago
      Rather than a direct syscall, you could imagine something like rseq where you have a shared userspace / kernel data structure where the userspace code gets aborted and restarted if the page was evicted while being processed. But making this work correctly and actually not have a perf overhead and also be an ergonomic API is super hard. In practice people who care probably are satisfied by direct I/O within io_uring with a custom page cache and a truly optimal implementation where the OS can still manage file pages and evict them but the application still new when it happened isn’t worth it.
      - bcrl 1y ago
        Unfortunately, a lot of the shared state with userland became much more difficult to implement securely when the Meltdown and Spectre (and others) exploits became concerns that had to be mitigated. They makes the OS's job a heck of a lot harder. Sometimes I feel modern technology is basically a delicately balanced house of cards that falls over when breathed upon or looked at incorrectly.
avianlyric 1y ago
There is no sensible OS API that could support this, because fundamentally memory access is a hardware API. The OS isn’t involved in normal memory reads, because that would be ludicrously inefficient, effectively requiring a syscall for every memory operation, which effectively means a syscall for any operation involving data I.e. all operations. Memory operations are always synchronous because they’re performed directly as a consequence of CPU instructions. Reading memory that’s been paged out results in the CPU itself detecting that the virtual address isn’t in RAM, and performing a hardware level interrupt. Literally abandoning a CPU instruction mid execution to start executing an entirely separate set of instructions which will hopefully sort out the page fault that just occurred, then kindly ask the CPU to go back and repeat the operation that caused the page fault. OS is only involved only because it’s the thing that provided the handling instructions for the CPU to execute in the event of a page fault. But it’s not in anyway actually capable of changing how the CPU initially handles the page fault. Also the current model does allow other threads to continue executing other work while the page fault is handled. The fault is completely localised to individual thread that triggered the fault. The CPU has no concept of the idea that multiple threads running on different physical core are in anyway related to each other. It also wouldn’t make sense to allow the interrupted thread to someone kick off a separate asynchronous operation, because where is it going to execute? The CPU core where the page fault happened is needed to handle the actual page fault, and copy in the needed memory. So even if you could kick off an async operation, there wouldn’t be any available CPU cycles to carry out the operation. Fundamentally there aren’t any sensible ways to improve on this problem, because the problem only exists due to us pretending that our machines have vastly more memory than they actually do. Which comes with tradeoffs, such as having to pause the CPU and steal CPU time to maintain the illusion. If people don’t like those tradeoffs, there’s a very simple solution. Put enough memory in your machine to keep your entire working set in memory all the time. Then page faults can never happen.
- im3w1l 1y ago
  I think you have a misunderstanding of how disk IO happens. The CPU core sends a command to the disk "I want some this and that data", then the CPU core can go do something else while the disk services that request. From what I read the disk actually puts the data directly into memory by using DMA, without needing to involve the CPU. So far so good, but then the question is to ensure that the CPU core has something more productive to do then just check "did the data arrive yet?" over and over and coordinating that is where good apis come in.
wmf 1y ago
If C had exceptions a page fault could safely unwind the stack up to the main loop which could work on something else until the page arrives. This has the advantage that there's no cost for the common case of accessing resident pages. Exceptions seem to have fallen out of favor so this may trade one problem for another.
- pjmlp 1y ago
  Windows C has exceptions, and no one has ever thought about doing something like this. They are only used for the same purpose as UNIX signals, without their flaws. In any case, page faults are OS specific, how to standardise such behaviour, with the added performance loss switching between both userspace and kernel.
- vlovich123 1y ago
  C++ has exceptions and having seen the vast majority of code and the way it’s written and the understanding of people writing it, exception safety is a foreign concept. Doing it in C without RAII seems particularly masochistic and doomed to fail. And unwinding the stack isn’t what you want to do because you’re basically signaling you want to cancel the operation and you’re throwing all the state when you precisely don’t want to do that - you just want to pause the current task and do other I/O in the meantime.
- gpderetta 1y ago
  you can longjmp, swapcontext or whatever from a signal handler into another lightweight fiber. The problem is that there is no "until the page arrive" notification. You would have to poll mincore which is awful. You could of course imagine an ansychronous "mmap complete notification" syscal, but at that point why not just use io_uring, it will be simpler and it has the benefit of actually existing.
twic 1y ago
There isn't today, but there was in 1991, scheduler activations: https://dl.acm.org/doi/10.1145/121132.121151 https://dl.acm.org/doi/10.1145/121132.121151 The rough idea is that if the kernel blocks a thread on something like a page cache miss, then it notifies the program through something a bit like a signal handler; if the program is doing user-level scheduling, it can then take account of that thread being blocked. The actual mechanism in the paper is more refined than that.
- scottlamb 1y ago
  Nice find. That going nowhere seems like classic consequence of the cyclical nature of these things: user-managed concurrency was cool, then it wasn't, then Go (and others) brought it back. I think the more recent UMCG [1] (kind of a hybrid approach, with threads visible by the kernel but mostly scheduled by userspace) handles this well. Assuming it ever actually lands in upstream, it seems reasonable to guess Go would adopt it, given that both originate within Google. It's worth pointing out that the slow major page fault problem is not unique to programs using mmap(..., fd, ...). The program binary is implicitly mmaped, and if swap is enabled, even anonymous memory can be paged out. I prefer to lock ~everything [2] into RAM to avoid this, but most programs don't do this, and default ulimits prevent programs running within login shells from locking much if anything. [1] https://lwn.net/Articles/879398/ https://lwn.net/Articles/879398/ [2] particularly on (mostly non-Go) programs with many threads, it's good to avoid locking into RAM the guard pages or stack beyond what is likely to be used, so better not to just use mlockall(MCL_CURRENT | MCL_FUTURE) unfortunately.

im3w1l 1y ago

There are apis that sort of let you do it: mincore, madvise, userfaultfd.

bcrl 1y ago
None of those APIs are cheap enough to call in a fast path.
- gpderetta 1y ago
  no syscall will be cheap to call in a fast path. You would need an hardware instruction that tells you if a load or store would fault.
  - zozbot234 1y ago
    > You would need an hardware instruction that tells you if a load or store would fault. You have MADV_FREE pages/ranges. They get cleared when purged, so reading zeros tells you that the load would have faulted and needs to be populated from storage.
    - vlovich123 1y ago
      MADV_FREE is insufficient - userspace doesn’t get a signal from the OS to know when there’s system wide memory pressure and having userspace try to respond to such a signal would be counter productive and slow in a kernel operation that needs to be a fast path. It’s more that you want MADV (page cache) a memory range and then have some way to have a shared data structure where you are told if it’s still resident and can lock it from being paged out.
      - zozbot234 1y ago
        > userspace doesn’t get a signal from the OS to know when there’s system wide memory pressure Memory pressure indicators exist, https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html > have some way to have a shared data structure where you are told if it’s still resident and can lock it from being paged out. What's more efficient than fetching data and comparing it with zero? Any write within the range will then cancel the MADV_FREE property on the written-to page thus "locking" it again, and this is also very efficient.
      - bcrl 1y ago
        MADV_FREE is also extremely expensive. CPU vendors have finally simplified TLB shootdown in recent CPUs with both AMD and Intel now having instructions to broadcast TLB flushes in hardware, which gets rid of one of the worst sources of performance degradation in threaded multicore applications (oh the pain of IPIs mixed with TLB flushing!). However, it's still very expensive to walk page tables and free pages. Hardware reference counting of memory allocations would be very interesting. It would be shockingly simple to implement compared to many other features hardware already has to tackle.
        zozbot234 1y ago
        > MADV_FREE is also extremely expensive. It's quite expensive to free pages under memory pressure (though it's not clear that there's any other choice to be made), but if the pages are never freed it should be cheap, AIUI.
  - vlovich123 1y ago
    Rather than a direct syscall, you could imagine something like rseq where you have a shared userspace / kernel data structure where the userspace code gets aborted and restarted if the page was evicted while being processed. But making this work correctly and actually not have a perf overhead and also be an ergonomic API is super hard. In practice people who care probably are satisfied by direct I/O within io_uring with a custom page cache and a truly optimal implementation where the OS can still manage file pages and evict them but the application still new when it happened isn’t worth it.
    - bcrl 1y ago
      Unfortunately, a lot of the shared state with userland became much more difficult to implement securely when the Meltdown and Spectre (and others) exploits became concerns that had to be mitigated. They makes the OS's job a heck of a lot harder. Sometimes I feel modern technology is basically a delicately balanced house of cards that falls over when breathed upon or looked at incorrectly.

avianlyric 1y ago

There is no sensible OS API that could support this, because fundamentally memory access is a hardware API. The OS isn’t involved in normal memory reads, because that would be ludicrously inefficient, effectively requiring a syscall for every memory operation, which effectively means a syscall for any operation involving data I.e. all operations. Memory operations are always synchronous because they’re performed directly as a consequence of CPU instructions. Reading memory that’s been paged out results in the CPU itself detecting that the virtual address isn’t in RAM, and performing a hardware level interrupt. Literally abandoning a CPU instruction mid execution to start executing an entirely separate set of instructions which will hopefully sort out the page fault that just occurred, then kindly ask the CPU to go back and repeat the operation that caused the page fault. OS is only involved only because it’s the thing that provided the handling instructions for the CPU to execute in the event of a page fault. But it’s not in anyway actually capable of changing how the CPU initially handles the page fault. Also the current model does allow other threads to continue executing other work while the page fault is handled. The fault is completely localised to individual thread that triggered the fault. The CPU has no concept of the idea that multiple threads running on different physical core are in anyway related to each other. It also wouldn’t make sense to allow the interrupted thread to someone kick off a separate asynchronous operation, because where is it going to execute? The CPU core where the page fault happened is needed to handle the actual page fault, and copy in the needed memory. So even if you could kick off an async operation, there wouldn’t be any available CPU cycles to carry out the operation. Fundamentally there aren’t any sensible ways to improve on this problem, because the problem only exists due to us pretending that our machines have vastly more memory than they actually do. Which comes with tradeoffs, such as having to pause the CPU and steal CPU time to maintain the illusion. If people don’t like those tradeoffs, there’s a very simple solution. Put enough memory in your machine to keep your entire working set in memory all the time. Then page faults can never happen.

im3w1l 1y ago
I think you have a misunderstanding of how disk IO happens. The CPU core sends a command to the disk "I want some this and that data", then the CPU core can go do something else while the disk services that request. From what I read the disk actually puts the data directly into memory by using DMA, without needing to involve the CPU. So far so good, but then the question is to ensure that the CPU core has something more productive to do then just check "did the data arrive yet?" over and over and coordinating that is where good apis come in.

wmf 1y ago

If C had exceptions a page fault could safely unwind the stack up to the main loop which could work on something else until the page arrives. This has the advantage that there's no cost for the common case of accessing resident pages. Exceptions seem to have fallen out of favor so this may trade one problem for another.

pjmlp 1y ago
Windows C has exceptions, and no one has ever thought about doing something like this. They are only used for the same purpose as UNIX signals, without their flaws. In any case, page faults are OS specific, how to standardise such behaviour, with the added performance loss switching between both userspace and kernel.
vlovich123 1y ago
C++ has exceptions and having seen the vast majority of code and the way it’s written and the understanding of people writing it, exception safety is a foreign concept. Doing it in C without RAII seems particularly masochistic and doomed to fail. And unwinding the stack isn’t what you want to do because you’re basically signaling you want to cancel the operation and you’re throwing all the state when you precisely don’t want to do that - you just want to pause the current task and do other I/O in the meantime.
gpderetta 1y ago
you can longjmp, swapcontext or whatever from a signal handler into another lightweight fiber. The problem is that there is no "until the page arrive" notification. You would have to poll mincore which is awful. You could of course imagine an ansychronous "mmap complete notification" syscal, but at that point why not just use io_uring, it will be simpler and it has the benefit of actually existing.

twic 1y ago

There isn't today, but there was in 1991, scheduler activations: https://dl.acm.org/doi/10.1145/121132.121151 https://dl.acm.org/doi/10.1145/121132.121151 The rough idea is that if the kernel blocks a thread on something like a page cache miss, then it notifies the program through something a bit like a signal handler; if the program is doing user-level scheduling, it can then take account of that thread being blocked. The actual mechanism in the paper is more refined than that.

scottlamb 1y ago
Nice find. That going nowhere seems like classic consequence of the cyclical nature of these things: user-managed concurrency was cool, then it wasn't, then Go (and others) brought it back. I think the more recent UMCG [1] (kind of a hybrid approach, with threads visible by the kernel but mostly scheduled by userspace) handles this well. Assuming it ever actually lands in upstream, it seems reasonable to guess Go would adopt it, given that both originate within Google. It's worth pointing out that the slow major page fault problem is not unique to programs using mmap(..., fd, ...). The program binary is implicitly mmaped, and if swap is enabled, even anonymous memory can be paged out. I prefer to lock ~everything [2] into RAM to avoid this, but most programs don't do this, and default ulimits prevent programs running within login shells from locking much if anything. [1] https://lwn.net/Articles/879398/ https://lwn.net/Articles/879398/ [2] particularly on (mostly non-Go) programs with many threads, it's good to avoid locking into RAM the guard pages or stack beyond what is likely to be used, so better not to just use mlockall(MCL_CURRENT | MCL_FUTURE) unfortunately.