3 ms·

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enoug

by _wmd 7y ago

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

gpderetta 7y ago
> Anonymous mappings are backed by swap and may be overcommitted, So is normal memory. Many allocators today even use mmap internally.
- deleted 7y ago
  [deleted]
- quotemstr 7y ago
  All* allocation is mmap, really. All mmap does is dedicate a region of address space to some kind of backing storage. The particular kind of backing storage makes all the difference. The problem is that people colloquially use "mmap" to mean "mmap of a conventional disk file" and don't mean all the other kinds of mmap out there, so discussions can become confusing. * Ther's sbrk too, but it's just a fancy legacy path that amounts to the same thing as anonymous mmap
klodolph 7y ago
> Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances Anonymous mappings won't cause signals, they'll trigger the OOM killer. Remember that malloc() is just a fancy wrapper for mmap() (and sbrk()).
- _wmd 7y ago
  If a process was swapped out and a fault fails to bring a page back due to an IO error, you can at least catch (I think) SIGBUS. But this just reinforces the point: nobody really understands virtual memory, even people like us that think they do
  - klodolph 7y ago
    Okay, but I'd say the correct thing to do is let SIGBUS kill the process. You can at least expect to handle SIGBUS for a file you've mapped.
  - tntn 7y ago
    So should we extend your conclusion above to the following? "There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid virtual memory whenever possible -- it's clear almost nobody fully understands it" I'd suggest that is ludicrous, and for the same reason your original conclusion is also excessive.
    - _wmd 7y ago
      It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :) My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to claim anonymous memory was somehow safer, both instances supporting the notion that most people in this thread don't seem to understand mmap at all. What we're examining is a powerful (and consequently hazardous) OS feature that often provides only marginal performance improvement, yet introduces many exotic error paths into a program that have their own exotic problems (memory access in thread A can raise SEGV in thread B, async-signal safety), that 7 hours' commenting has not been sufficient to fully capture. This thread is full of upvoted miscomprehension, bad advice (spawn a child to deal with SEGV!?), obviously incorrect solutions (signalfd), and yet still manages to completely omit some critical characteristics of mmap, for example, that faults take a VM-global semaphore -- mmap can easily destroy multithreaded app performance in a way read() is immune to, because nobody expects file IO in one thread to cause malloc() latency in another. If this isn't evidence for "avoid this feature wherever possible", I really don't know what is.
      - tntn 7y ago
        As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they also apply to all memory, period. mmap'ed anonymous memory might have th same problems as file-backed memory, but those are the same problems that .text and .bss have. The "powerful (and consequently hazardous) OS feature" here isn't mmap, it's literally virtual memory. At the moment you concede that memory may be backed by something besides physical memory at any point in time, you get the possibility of all those "exotic error paths."

by _wmd 7y ago

gpderetta 7y ago

> Anonymous mappings are backed by swap and may be overcommitted, So is normal memory. Many allocators today even use mmap internally.

deleted 7y ago
[deleted]
quotemstr 7y ago
All* allocation is mmap, really. All mmap does is dedicate a region of address space to some kind of backing storage. The particular kind of backing storage makes all the difference. The problem is that people colloquially use "mmap" to mean "mmap of a conventional disk file" and don't mean all the other kinds of mmap out there, so discussions can become confusing. * Ther's sbrk too, but it's just a fancy legacy path that amounts to the same thing as anonymous mmap

klodolph 7y ago

> Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances Anonymous mappings won't cause signals, they'll trigger the OOM killer. Remember that malloc() is just a fancy wrapper for mmap() (and sbrk()).

_wmd 7y ago
If a process was swapped out and a fault fails to bring a page back due to an IO error, you can at least catch (I think) SIGBUS. But this just reinforces the point: nobody really understands virtual memory, even people like us that think they do
- klodolph 7y ago
  Okay, but I'd say the correct thing to do is let SIGBUS kill the process. You can at least expect to handle SIGBUS for a file you've mapped.
- tntn 7y ago
  So should we extend your conclusion above to the following? "There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid virtual memory whenever possible -- it's clear almost nobody fully understands it" I'd suggest that is ludicrous, and for the same reason your original conclusion is also excessive.
  - _wmd 7y ago
    It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :) My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to claim anonymous memory was somehow safer, both instances supporting the notion that most people in this thread don't seem to understand mmap at all. What we're examining is a powerful (and consequently hazardous) OS feature that often provides only marginal performance improvement, yet introduces many exotic error paths into a program that have their own exotic problems (memory access in thread A can raise SEGV in thread B, async-signal safety), that 7 hours' commenting has not been sufficient to fully capture. This thread is full of upvoted miscomprehension, bad advice (spawn a child to deal with SEGV!?), obviously incorrect solutions (signalfd), and yet still manages to completely omit some critical characteristics of mmap, for example, that faults take a VM-global semaphore -- mmap can easily destroy multithreaded app performance in a way read() is immune to, because nobody expects file IO in one thread to cause malloc() latency in another. If this isn't evidence for "avoid this feature wherever possible", I really don't know what is.
    - tntn 7y ago
      As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they also apply to all memory, period. mmap'ed anonymous memory might have th same problems as file-backed memory, but those are the same problems that .text and .bss have. The "powerful (and consequently hazardous) OS feature" here isn't mmap, it's literally virtual memory. At the moment you concede that memory may be backed by something besides physical memory at any point in time, you get the possibility of all those "exotic error paths."