4 ms·

Early BSD VM pre-allocated swap backing for every anonymous page — you couldn't allocate virtual memory without a swap slot reserved for it, even if the page wa

Early BSD VM pre-allocated swap backing for every anonymous page — you couldn't allocate virtual memory without a swap slot reserved for it, even if the page was never paged out. When a process forks, the child needed swap reservations for the parent's entire address space (before exec replaces it). A large process forking temporarily needs double its swap allocation. If your working set is roughly equal to physical RAM, fork alone gets you to 2x. This was the practical bottleneck people actually hit. Your system had enough RAM, swap wasn't full, but fork() failed because there wasn't enough contiguous swap to reserve. 2x was the number that made fork() stop failing on a reasonably loaded system. The later overcommit/copy-on-write changes made this less relevant, but the rule of thumb outlived the technical reason. Most people repeating "2x RAM" today are running systems where anonymous pages aren't swap-backed until actually paged out. Today swap is no longer about extending your address space, it's about giving the kernel room to page out cold anonymous pages so that RAM can be used for disk cache. A little swap makes the system faster even when you're nowhere near running out of memory, because the kernel can evict pages it hasn't touched in hours and use that RAM for hot file data instead. The exception is hibernation — you need swap >= RAM for that, which is why Ubuntu's recommendations are higher than RedHat's 20% of RAM.

quotemstr 8mo ago
TBF, I think overcommit was and remains an ugliness in how we manage memory. I wish we'd solved the fork commit-charge-spike issue by encouraging vfork (and later, posix_spawn) more heavily, not by making the OS lie about the availability of memory. The ship's long sailed though, so even I run with overcommit enabled and only grumble about what might have been.
- Skunkleton 8mo ago
  It's not just fork. The operating system overcommits memory all over the place. For example, when you map memory, that can/will succeed without actually mapping physical pages. Even "available" memory is put to some use and freed in an asynchronous way behind the scene, a process that is not always successful. Honestly, I think overcommit is a good thing. If you want to give a process an isolated address space, then you have to allow that process to lay out memory as it sees fit, without having to worry too much about what else happens to be on the system. If you immediately "charge" the process for this, you will end up nit-picking every process on the system, even though with overcommit you would have been fine.
- man8alexd 8mo ago
  Overcommit allows much better memory utilization. People disable overcommit all the time and get surprised when their programs start failing mallocs while there are still tons of discardable page cache in the system. https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797841/1027 https://unix.stackexchange.com/q/797841/1027
charcircuit 8mo ago
Today's swap also is not preallocated by the user. It is entirely handled by the OS itself. If it needs swap space to hibernate it will go ahead and allocate it itself.
- Dylan16807 8mo ago
  It does? Last I checked Linux doesn't do dynamic swap sizes, and while Windows has dynamic swap sizes it has a separate big non-dynamic file for hibernation. I have no idea what MacOS does.
  - cr125rider 8mo ago
    You guys are getting Linux to hibernate!?
    - Kaliboy 8mo ago
      I managed to get mine to wake up from sleep, haven't recovered enough to attempt hibernation yet.
      - em-bee 8mo ago
        on my thinkpad with fedora hibernate works fine. i use it frequently. i even use it when for an unknown reason my usb-c headphones stop working. i don't know why. but after hibernate they work again. somehow waking up from hibernation fixes the problem
    - deleted 8mo ago
      [deleted]
man8alexd 8mo ago
The OP here. I think this is the correct answer. I'm aware of the strict swap reservation, even mentioned it in the question, but I didn't realize that it doubles memory on fork even with copy-on-write, so I assumed that it implies a minimum of 1x, not 2x. This is specific to BSD and SunOS, but with them being popular in the 80s, the 2x rule became widespread.

quotemstr 8mo ago

TBF, I think overcommit was and remains an ugliness in how we manage memory. I wish we'd solved the fork commit-charge-spike issue by encouraging vfork (and later, posix_spawn) more heavily, not by making the OS lie about the availability of memory. The ship's long sailed though, so even I run with overcommit enabled and only grumble about what might have been.

Skunkleton 8mo ago
It's not just fork. The operating system overcommits memory all over the place. For example, when you map memory, that can/will succeed without actually mapping physical pages. Even "available" memory is put to some use and freed in an asynchronous way behind the scene, a process that is not always successful. Honestly, I think overcommit is a good thing. If you want to give a process an isolated address space, then you have to allow that process to lay out memory as it sees fit, without having to worry too much about what else happens to be on the system. If you immediately "charge" the process for this, you will end up nit-picking every process on the system, even though with overcommit you would have been fine.
man8alexd 8mo ago
Overcommit allows much better memory utilization. People disable overcommit all the time and get surprised when their programs start failing mallocs while there are still tons of discardable page cache in the system. https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797841/1027 https://unix.stackexchange.com/q/797841/1027

charcircuit 8mo ago

Today's swap also is not preallocated by the user. It is entirely handled by the OS itself. If it needs swap space to hibernate it will go ahead and allocate it itself.

Dylan16807 8mo ago
It does? Last I checked Linux doesn't do dynamic swap sizes, and while Windows has dynamic swap sizes it has a separate big non-dynamic file for hibernation. I have no idea what MacOS does.
- cr125rider 8mo ago
  You guys are getting Linux to hibernate!?
  - Kaliboy 8mo ago
    I managed to get mine to wake up from sleep, haven't recovered enough to attempt hibernation yet.
    - em-bee 8mo ago
      on my thinkpad with fedora hibernate works fine. i use it frequently. i even use it when for an unknown reason my usb-c headphones stop working. i don't know why. but after hibernate they work again. somehow waking up from hibernation fixes the problem
  - deleted 8mo ago
    [deleted]

man8alexd 8mo ago

The OP here. I think this is the correct answer. I'm aware of the strict swap reservation, even mentioned it in the question, but I didn't realize that it doubles memory on fork even with copy-on-write, so I assumed that it implies a minimum of 1x, not 2x. This is specific to BSD and SunOS, but with them being popular in the 80s, the 2x rule became widespread.