5 ms·

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old m

There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said, there's nothing wrong with mmap or SIGBUS error reporting in concept. I think I've said it before, but you can think of mmap of a disk file as basically adding a temporary dedicated swap file to the system, with all the pages backed by that "swap file" already "swapped" out unless already cached. Sometimes that's exactly what you want! The author's signal handler registration difficulties come from the POSIX signal API being awful, not from the idea that catching CPU traps is somehow bad. It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. When I ran [1] by the glibc people, though, the response from them was pretty stark and unwarranted hostility toward any use of signals at all, even in perfectly legitimate scenarios, like mmap SIGBUS or certain kinds of important JVM-style pointer check [2] optimizations. It's this "we won't change anything anywhere despite our views being out of step with real user needs" attitude that's responsible for a lot of weird friction in the Unix API surface. Practically every program mmaps disk files has trouble recovering from IO errors. If it were easier for multiple components in a process to share responsibility for handling a signal, we'd more often get subtleties like this right. A better signal API will improve the user experience. Tilting at signal-shaped windmills won't. [1] https://www.facebook.com/notes/daniel-colascione/toward-shareable-posix-signals/10157129032641102/ https://www.facebook.com/notes/daniel-colascione/toward-shar... [2] Relying on CPU traps lets you perform certain checks for free. Why write "if (myptr != nullptr) var = myptr" when you can just write "var = myptr" and handle the SIGSEGV when myptr ends up being nullptr? The latter option is zero-overhead in the non-nullptr case. It's also much more complicated, but if you're a VM, and you generate millions and millions of these "if (myptr != nullptr)" checks in JITed code, the size and speed win of eliminating the check starts to justify the complexity.

Pahr3yah 8y ago
If you roughly know your access patterns in advance you can reduce the page fault costs with madvise.
- quotemstr 8y ago
  Sure, but you still end up with double-caching and frequent entry in the kernel. madvise will never give you as much control and flexibility as just getting the system out of the way and doing the work yourself. (DB systems can also use fun tricks like compressing their cached pages.)
  - gmueckl 8y ago
    Why would mmap lead to double caching? I can't follow.
    - quotemstr 8y ago
      It's not that mmap per se leads to double caching, but that combining the page cache with application-level caching leads to double caching. Say you're reading hugecactus.png into your image processing program. Whether you use mmap(2) or ordinary read(2), the first step in reading hugecactus.png is the kernel DMAing the bytes into the page cache. In the mmap case, the kernel maps the page cache into your application's address space. In the read case, the kernel copies from the page cache to the application read buffer. Now suppose your application PNG-decodes hugecactus.png into RGB raster data. Now, whether you used mmap or read, the kernel has both the decoded RGB data blob and the original PNG data in memory. That's usually wasteful. (Yes, you can reduce the severity of this problem with MADV_DONTNEED and friends.)
      - gmueckl 8y ago
        Well, the kernel side cache is not much of a problem. The kernel is free to evict those pages at any time to respond to memory pressure etc. Linux treats its file system cache almost like unused memory in that it is normally the biggest pool from wich memory allocations for processes are drawn. Essentially, keeping the pages around in the cache is an optimization, because explicitly overwriting them too aggressively is just unnecessary work.
gmueckl 8y ago
Signal handling in POSIX breaks multithreading in practice because it is so crazy hard to get right (I would be surprised if more than 50% of the code out there does it right). Can you link to the libc maintainer responses? I would like to know how they countered you exactly.
- quotemstr 8y ago
  > Signal handling in POSIX breaks multithreading in practice because it is so crazy hard to get right In my experience, it's pretty easy to write async-signal-safe code --- which is almost always also thread-safe --- if you follow a few simple rules. I don't understand why people have so much difficulty writing correct signal handlers.
burntsushi 8y ago
> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this in its man page: > ripgrep may abort unexpectedly when using > default settings if it searches a file that > is simultaneously truncated. This behavior > can be avoided by passing the --no-mmap flag > which will forcefully disable the use of > memory maps in all cases. mmap has its problems. But on Linux for a simple sequential read of a large file, it generally does measurably better than standard `read` calls. ripgrep doesn't even bother with madvise. Changing the workload can dramatically alter these conclusions. For example, on a checkout of the Linux kernel: $ time rg zqzqzqzq --mmap real 1.661 user 1.603 sys 3.128 maxmem 41 MB faults 0 $ time rg zqzqzqzq --no-mmap real 0.126 user 0.702 sys 0.586 maxmem 20 MB faults 0 Performance of mmap can also vary depending on platform as well. FWIW, I do generally disagree with your broader point, but it's important to understand that there's actually good reason to believe that using mmaps can be faster in some circumstances.
- TheCondor 8y ago
  It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing, it’s going to do mostly (maybe only, I assume it backtracks on matches) sequential reads of mostly static files, it’s really the easy case, its not clear that madvise would help and your brand is speed so saving on those copies is worth it. What might be interesting, on certain memory constrained systems you can slide a smaller map space through the file rather than mapping the whole thing; it’s been a while since I looked at it all but mapping the smaller pieces gives huge hints to the vmm and it would probably slow down rg incrementally but speed up overall system performance.
altfredd 8y ago
> It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal. The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too: > It’s occasionally useful to longjmp out of a signal handler. It’s reasonable to want to return non-locally from a shared signal handler too --- that is, to resume program execution after SIGNAL_CONTINUE_EXECUTION in a different state from the state the program had when we entered the shared signal handler. Since the signal system probably wants to maintain some kind of state to track its progress through its shared signal handler list, a plain longjmp out of a shared signal handler will likely leave the system in an unspecified state. Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!! The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. There is a mention of banning longjmp, but individual handlers still can bail via SIGNAL_CONTINUE_EXECUTION. What if I want my handler to always run regardless of registration order? The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API. The proposed API does not address challenges of using signals in multi-threading programs. The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it. Overall the proposed interface brings little to the table, does not work well alongside with existing sigaction() API and creates false illusion, that signal handlers are safe and ok to use. I imagine, that if it had more technical "meat" — more like robust mutexes or FD_CLOEXEC — it would have seen a lot more constructive discussion and less hostility from glibc maintainers.
- quotemstr 8y ago
  > The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safety issues, since a thread causing a fault can, in principle, be anywhere! > Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!! If your handlers play by the rules and are async-signal-safe, there's no problem. > The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. What's half-baked about it? > What if I want my handler to always run regardless of registration order? That's a logical nonsense request. What if two components want to their handlers to be the highest priority? > The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API. The whole point of the facility is to let different components share a signal without stepping on each other. Why would you need to retrieve the list of handlers? > The proposed API does not address challenges of using signals in multi-threading programs. This claim is too vague to rebut. What specific "challenges" are you talking about? > The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it. The proposed API works fine with library unloading: a library can unregister whatever handlers it's installed just before being unloaded (e.g., in a static destructor), and this unregister operation is guaranteed to be safe no matter what the order handlers are unloaded. > Overall the proposed interface brings little to the table It allows multiple components to safely share signals. The objections you've mentioned are based on your misunderstanding my proposal. > does not work well alongside with existing sigaction() Yes it does. The article talks about this interaction specifically. > I imagine, that if it had more technical "meat" The glibc people literally think that nobody should be using signals. That's their objection, not anything you've talked about.
gubbrora 8y ago
Interesting but how will you determine not-null is the hot path? Getting it wrong coughs cost you
- quotemstr 8y ago
  In the case of VMs, null means "throw NullPointerException". How often do you see that happening?

Pahr3yah 8y ago

If you roughly know your access patterns in advance you can reduce the page fault costs with madvise.

quotemstr 8y ago
Sure, but you still end up with double-caching and frequent entry in the kernel. madvise will never give you as much control and flexibility as just getting the system out of the way and doing the work yourself. (DB systems can also use fun tricks like compressing their cached pages.)
- gmueckl 8y ago
  Why would mmap lead to double caching? I can't follow.
  - quotemstr 8y ago
    It's not that mmap per se leads to double caching, but that combining the page cache with application-level caching leads to double caching. Say you're reading hugecactus.png into your image processing program. Whether you use mmap(2) or ordinary read(2), the first step in reading hugecactus.png is the kernel DMAing the bytes into the page cache. In the mmap case, the kernel maps the page cache into your application's address space. In the read case, the kernel copies from the page cache to the application read buffer. Now suppose your application PNG-decodes hugecactus.png into RGB raster data. Now, whether you used mmap or read, the kernel has both the decoded RGB data blob and the original PNG data in memory. That's usually wasteful. (Yes, you can reduce the severity of this problem with MADV_DONTNEED and friends.)
    - gmueckl 8y ago
      Well, the kernel side cache is not much of a problem. The kernel is free to evict those pages at any time to respond to memory pressure etc. Linux treats its file system cache almost like unused memory in that it is normally the biggest pool from wich memory allocations for processes are drawn. Essentially, keeping the pages around in the cache is an optimization, because explicitly overwriting them too aggressively is just unnecessary work.

gmueckl 8y ago

Signal handling in POSIX breaks multithreading in practice because it is so crazy hard to get right (I would be surprised if more than 50% of the code out there does it right). Can you link to the libc maintainer responses? I would like to know how they countered you exactly.

quotemstr 8y ago
> Signal handling in POSIX breaks multithreading in practice because it is so crazy hard to get right In my experience, it's pretty easy to write async-signal-safe code --- which is almost always also thread-safe --- if you follow a few simple rules. I don't understand why people have so much difficulty writing correct signal handlers.

burntsushi 8y ago

> This old myth that mmap is the fast and efficient way to do IO just won't die. Well... because it's not a myth in all cases? $ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap real 1.167 user 0.815 sys 0.349 maxmem 9473 MB faults 0 $ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap real 1.748 user 0.506 sys 1.239 maxmem 9 MB faults 0 The OP's adventures with mmap mirror my own, which is why ripgrep includes this in its man page: > ripgrep may abort unexpectedly when using > default settings if it searches a file that > is simultaneously truncated. This behavior > can be avoided by passing the --no-mmap flag > which will forcefully disable the use of > memory maps in all cases. mmap has its problems. But on Linux for a simple sequential read of a large file, it generally does measurably better than standard `read` calls. ripgrep doesn't even bother with madvise. Changing the workload can dramatically alter these conclusions. For example, on a checkout of the Linux kernel: $ time rg zqzqzqzq --mmap real 1.661 user 1.603 sys 3.128 maxmem 41 MB faults 0 $ time rg zqzqzqzq --no-mmap real 0.126 user 0.702 sys 0.586 maxmem 20 MB faults 0 Performance of mmap can also vary depending on platform as well. FWIW, I do generally disagree with your broader point, but it's important to understand that there's actually good reason to believe that using mmaps can be faster in some circumstances.

TheCondor 8y ago
It’s not a myth at all, mmap is faster, you save on straight copies of data and the sys-calls to do it. It should be faster in nearly all circumstances, faster by at least a copy. In exchange you pick up a lot of complexity dealing with faults and you potentially put stress on the VM system. If you are doing to ‘O’ part of I/O then mmap starts to be really complex, fast. rg is kind of a special case, it’s not writing, it’s going to do mostly (maybe only, I assume it backtracks on matches) sequential reads of mostly static files, it’s really the easy case, its not clear that madvise would help and your brand is speed so saving on those copies is worth it. What might be interesting, on certain memory constrained systems you can slide a smaller map space through the file rather than mapping the whole thing; it’s been a while since I looked at it all but mapping the smaller pieces gives huge hints to the vmm and it would probably slow down rg incrementally but speed up overall system performance.

altfredd 8y ago

> It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal. The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too: > It’s occasionally useful to longjmp out of a signal handler. It’s reasonable to want to return non-locally from a shared signal handler too --- that is, to resume program execution after SIGNAL_CONTINUE_EXECUTION in a different state from the state the program had when we entered the shared signal handler. Since the signal system probably wants to maintain some kind of state to track its progress through its shared signal handler list, a plain longjmp out of a shared signal handler will likely leave the system in an unspecified state. Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!! The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. There is a mention of banning longjmp, but individual handlers still can bail via SIGNAL_CONTINUE_EXECUTION. What if I want my handler to always run regardless of registration order? The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API. The proposed API does not address challenges of using signals in multi-threading programs. The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it. Overall the proposed interface brings little to the table, does not work well alongside with existing sigaction() API and creates false illusion, that signal handlers are safe and ok to use. I imagine, that if it had more technical "meat" — more like robust mutexes or FD_CLOEXEC — it would have seen a lot more constructive discussion and less hostility from glibc maintainers.

quotemstr 8y ago
> The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safety issues, since a thread causing a fault can, in principle, be anywhere! > Are you sure, that we should worry about "signal system maintaining some kind of state"? Not about rest of application being in completely unspecified state?!! If your handlers play by the rules and are async-signal-safe, there's no problem. > The article proposes a primitive system for setting signal priorities, but stops at a half-baked solution. What's half-baked about it? > What if I want my handler to always run regardless of registration order? That's a logical nonsense request. What if two components want to their handlers to be the highest priority? > The proposal does not offer a way to retrieve a list of already installed handlers, which makes that part of it even worse than existing Posix signal API. The whole point of the facility is to let different components share a signal without stepping on each other. Why would you need to retrieve the list of handlers? > The proposed API does not address challenges of using signals in multi-threading programs. This claim is too vague to rebut. What specific "challenges" are you talking about? > The article mentions, that signal handlers can't be reliably unloaded, but proposed API does not address it. The proposed API works fine with library unloading: a library can unregister whatever handlers it's installed just before being unloaded (e.g., in a static destructor), and this unregister operation is guaranteed to be safe no matter what the order handlers are unloaded. > Overall the proposed interface brings little to the table It allows multiple components to safely share signals. The objections you've mentioned are based on your misunderstanding my proposal. > does not work well alongside with existing sigaction() Yes it does. The article talks about this interaction specifically. > I imagine, that if it had more technical "meat" The glibc people literally think that nobody should be using signals. That's their objection, not anything you've talked about.

gubbrora 8y ago

Interesting but how will you determine not-null is the hot path? Getting it wrong coughs cost you

quotemstr 8y ago
In the case of VMs, null means "throw NullPointerException". How often do you see that happening?