42 ms·
I found the currently accepted answer (by user10489) interesting, tying together a lot of concepts I had heard of but didn't properly understand. For example, l
by Aachen 4y ago
I found the currently accepted answer (by user10489) interesting, tying together a lot of concepts I had heard of but didn't properly understand. For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" and either crash the program or let the program handle the failed allocation gracefully? This answer explains how this all works.
The TL;DR answer to the actual question is: processes generally don't get access to each other's memory unless there is some trust relation (like being the parent process, or being allowed to attach a debugger), and being in a container doesn't change that, the same restrictions apply and you always get zeroed-out memory from the kernel. It's when you use a different allocator that you might get nonzeroed memory from elsewhere in your own process (not a random other process).
- dale_glass 4y ago> Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" Blame UNIX for that, and the fork() system call. It's a design quirk. fork() duplicates the process. So suppose your web browser consumes 10GB RAM out of the 16GB total on the system, and wants to run a process for anything. Like it just wants to exec something tiny, like `uname`. 1. 10GB process does fork(). 2. Instantly, you have two 10GB processes 3. A microsecond later, the child calls exec(), completely destroying its own state and replacing it with a 36K binary, freeing 10GB RAM. So there's two ways to go there: 1. You could require step 2 to be a full copy. Which means either you need more RAM, a huge chunk of which would always sit idle, or you need a lot of swap, for the same purpose. 2. We could overlook the memory usage increase and pretend that we have enough memory, and only really panic if the second process truly needs its own 10GB RAM that we don't have. That's what Linux does. The problem with #2 is that dealing with this happens completely in the background, at times completely unpredictable to the code. The OS allocates memory when the child changes memory, like does "a=1" somewhere. A program can't handle memory allocations failures there because as far as it knows, it's not allocating anything. So what you get is this fragile fiction that sometimes breaks and requires the kernel to kill something to maintain the system in some sort of working state. Windows doesn't have this issue at all because it has no fork(). New processes aren't children and start from scratch, so firefox never gets another 10GB sized clone. It just starts a new, 36K sized process.
- peheje 4y agoI find your writing style really pleasant and understandable! Much more so than the StackExchange answer. I really like the breakdown into steps, then what could happen steps and the follow ups. Where can I read more (from you?) in this style about OS and memory management?
- jacquesm 4y ago> 2. Instantly, you have two 10GB processes No, that's not how it works. The process table gets duplicated and copy-on-write takes care of the pages. As long as they are identical they will be shared, there is no way that 10GB of RAM will be allocated to the forked process and that all of the data will be copied.
- worthless-trash 4y agoOP is referring to what would happen in the naive implementation, not what actually happens.
- reisse 4y agoThe data won't be copied, but kernel has to reserve the memory for both processes.
- progbits 4y agoIf you would like to go deeper down the rabbit hole, fasterthanlime recently made a few videos that explore this topic: https://youtu.be/YB6LTaGRQJg https://youtu.be/YB6LTaGRQJg https://youtu.be/c_5Jy_AVDaM https://youtu.be/c_5Jy_AVDaM https://youtu.be/DpnXaNkM9_M https://youtu.be/DpnXaNkM9_M The code is in Rust but that doesn't matter for the explanation.
- skyeto 4y agoWhat I find to be a rather interesting tidbit related to this is that some applications (e.g. certain garbage collectors) map multiple ranges of virtual memory that address the same physical memory.
- yakubin 4y agoThat’s how you can create a gapless ring buffer: <https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc2#remarks https://learn.microsoft.com/en-us/windows/win32/api/memoryap...> (see scenario 1 in examples).
- jeffbee 4y agoIn Linux, a task that tries to allocate more than the allowed amount will be immediately terminated and not notified about anything. The reason out-of-memory systems thrash is merely because people do not always bother setting the limits, and the default assumes users prefer a process that continues through hours of thrashing instead of being promptly terminated.
- Sakos 4y agoWho is "people" in this? If you mean your average user, it's unrealistic to expect them to know or care about details like this. This is the kind of thing that needs sane defaults.
- saagarjha 4y ago> For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Generally browsers allocate large swaths of address space as guard pages (they're unmapped, but accesses into them will trap). Or they'll double map pages, so that the same physical page shows up multiple times in the virtual address space.