10 ms·

Is RAM wiped before use in another LXC container?

309 points by Aachen 4y ago

Aachen 4y ago
I found the currently accepted answer (by user10489) interesting, tying together a lot of concepts I had heard of but didn't properly understand. For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" and either crash the program or let the program handle the failed allocation gracefully? This answer explains how this all works. The TL;DR answer to the actual question is: processes generally don't get access to each other's memory unless there is some trust relation (like being the parent process, or being allowed to attach a debugger), and being in a container doesn't change that, the same restrictions apply and you always get zeroed-out memory from the kernel. It's when you use a different allocator that you might get nonzeroed memory from elsewhere in your own process (not a random other process).
- dale_glass 4y ago
  > Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" Blame UNIX for that, and the fork() system call. It's a design quirk. fork() duplicates the process. So suppose your web browser consumes 10GB RAM out of the 16GB total on the system, and wants to run a process for anything. Like it just wants to exec something tiny, like `uname`. 1. 10GB process does fork(). 2. Instantly, you have two 10GB processes 3. A microsecond later, the child calls exec(), completely destroying its own state and replacing it with a 36K binary, freeing 10GB RAM. So there's two ways to go there: 1. You could require step 2 to be a full copy. Which means either you need more RAM, a huge chunk of which would always sit idle, or you need a lot of swap, for the same purpose. 2. We could overlook the memory usage increase and pretend that we have enough memory, and only really panic if the second process truly needs its own 10GB RAM that we don't have. That's what Linux does. The problem with #2 is that dealing with this happens completely in the background, at times completely unpredictable to the code. The OS allocates memory when the child changes memory, like does "a=1" somewhere. A program can't handle memory allocations failures there because as far as it knows, it's not allocating anything. So what you get is this fragile fiction that sometimes breaks and requires the kernel to kill something to maintain the system in some sort of working state. Windows doesn't have this issue at all because it has no fork(). New processes aren't children and start from scratch, so firefox never gets another 10GB sized clone. It just starts a new, 36K sized process.
  - peheje 4y ago
    I find your writing style really pleasant and understandable! Much more so than the StackExchange answer. I really like the breakdown into steps, then what could happen steps and the follow ups. Where can I read more (from you?) in this style about OS and memory management?
  - jacquesm 4y ago
    > 2. Instantly, you have two 10GB processes No, that's not how it works. The process table gets duplicated and copy-on-write takes care of the pages. As long as they are identical they will be shared, there is no way that 10GB of RAM will be allocated to the forked process and that all of the data will be copied.
    - worthless-trash 4y ago
      OP is referring to what would happen in the naive implementation, not what actually happens.
    - reisse 4y ago
      The data won't be copied, but kernel has to reserve the memory for both processes.
      - l33tman 4y ago
        No, it doesn't. See "overcommit"
        enedil 4y ago
        In a sense it does. Page tables would need to be copied.
        jacquesm 4y ago
        Yes, but they all point to the same pages. The tables take up a small fraction of the memory of the pages themselves.
- progbits 4y ago
  If you would like to go deeper down the rabbit hole, fasterthanlime recently made a few videos that explore this topic: https://youtu.be/YB6LTaGRQJg https://youtu.be/YB6LTaGRQJg https://youtu.be/c_5Jy_AVDaM https://youtu.be/c_5Jy_AVDaM https://youtu.be/DpnXaNkM9_M https://youtu.be/DpnXaNkM9_M The code is in Rust but that doesn't matter for the explanation.
- skyeto 4y ago
  What I find to be a rather interesting tidbit related to this is that some applications (e.g. certain garbage collectors) map multiple ranges of virtual memory that address the same physical memory.
  - yakubin 4y ago
    That’s how you can create a gapless ring buffer: <https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc2#remarks https://learn.microsoft.com/en-us/windows/win32/api/memoryap...> (see scenario 1 in examples).
- jeffbee 4y ago
  In Linux, a task that tries to allocate more than the allowed amount will be immediately terminated and not notified about anything. The reason out-of-memory systems thrash is merely because people do not always bother setting the limits, and the default assumes users prefer a process that continues through hours of thrashing instead of being promptly terminated.
  - Sakos 4y ago
    Who is "people" in this? If you mean your average user, it's unrealistic to expect them to know or care about details like this. This is the kind of thing that needs sane defaults.
- saagarjha 4y ago
  > For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Generally browsers allocate large swaths of address space as guard pages (they're unmapped, but accesses into them will trap). Or they'll double map pages, so that the same physical page shows up multiple times in the virtual address space.
Proven 4y ago
[dead]
vrglvrglvrgl 4y ago
[dead]
adamgordonbell 4y ago
My shortcut for answering these questions is just replace 'container' with process and find the answer for that. A running container is just a process that is using certain kernel features: cgroup, namespaces, pivotroot. LXC has a different DX around containers than OCI containers do, but the same rules apply.
- sgt 4y ago
  DX?
  - adamgordonbell 4y ago
    Developer Experience (Maybe there is a better term, but how you use them) Docker/ OCI containers tend to be a single process and LXC containers have a collection of processes. But in either case they are just processes running on the host in a different namespace. So they feel different to use, but use the same building blocks (to my understanding).
    - sgt 4y ago
      Unrelated, but your name rang a... bell. You're the corecursive guy. Great podcast!
      - adamgordonbell 4y ago
        Thanks!
    - Karellen 4y ago
      Shouldn't "DX" refer to the experience of people hacking on the Docker (or whatever) code itself? Like... how easy the build system is to work with if you're adding a new source file to the docker codebase? The people working with Docker, even if they are developers doing development work, are still users of Docker, aren't they? I mean, the GUI of an IDE is still part of its UX, right? Even though it's for developers doing development work?
worthless-trash 4y ago
One day, the general technical population will understand containers are just processes on a system. No magic under the hood.
- __turbobrew__ 4y ago
  Hey, if people keep thinking it is magic I keep getting paid to be a wizard
gregw2 4y ago
It’s not clear to me why the top rated reply which seemed extremely knowledgeable in discussing Linux kernel auto zeroing of memory mentioned a bunch of caveats including mentioning “some memory management libraries may reuse memory” but didn’t say “including glibc”. That’s a pretty big library to gloss over from a security or working programmer perspective! (Corrections to perspective welcome.)
- hn92726819 4y ago
  I thought the shared memory they were referring to would be read only memory or something. So yes, the memory is shared, but your process doesn't have any access to overwrite any of it. Likewise, when you call library functions, when it mallocs something, it uses memory that your process has write access to, not the shared library memory. In the context of the question, I assume the asker was mostly interested in reading some kind of sensitive data from a previous process, not reading the same librsry-code-only memory or something. Note: all of this could be wrong, it was just my understanding Edit: looks like this answers that: https://stackoverflow.com/questions/20857134/memory-write-protection-against-shared-library https://stackoverflow.com/questions/20857134/memory-write-pr...
- IcePic 4y ago
  Even if your libc IS one of the offenders, this libc code runs in a single process, so even though libc does or does not use evil tricks for your process, it doesn't mean you will get pages from some other process memory map, only that you may get your own old pages back.
  - gregw2 4y ago
    Thanks, makes sense. (Looks like the poster commenting about glibc has corrected themself.)
bluedino 4y ago
Is the confusion coming from when you malloc() memory in C you aren't getting zereoed memory?
- dale_glass 4y ago
  I tried on Linux, and got all zeroes. Of course that's for a trivial program. If you freed something, that probably wasn't returned to the OS, and the next malloc might just recycle it.
  - bluedino 4y ago
    Malloc a bunch of memory, dirty it, free it, then malloc some more. And then check. Your first request in your program, you'll get clean memory from the OS.
    - dale_glass 4y ago
      Yeah, that's what I'm saying.
CodesInChaos 4y ago
Is GPU memory wiped before it's handed to a new process, and is a process prevented from accessing GPU memory of a different process? (assuming low level APIs like Vulkan)
- tryauuum 4y ago
  this is certainly more interesting question than OPs
- NotCamelCase 4y ago
  You can request Windows to zero out every GPU memory allocation, but it's more of a driver thing that's not exposed in APIs. Such option is likely to be off by default in drivers as it might induce additional, unintended overhead. In practice, you are likely to see memory cleared more often than not due to other reasons, though. You can't just peek another process' GPU memory thru UMD app, either. Per-process virtual memory mechanisms similar to CPUs are also present in GPUs, which is the whole reason that resources are explicitly imported/exported across APIs via special API calls.
- johntb86 4y ago
  By the Vulkan spec it is guaranteed that memory from one process can't be seen by another. https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html#fundamentals-validusage https://registry.khronos.org/vulkan/specs/1.3-extensions/htm... says: In particular, any guarantees made by an operating system about whether memory from one process can be visible to another process or not must not be violated by a Vulkan implementation for any memory allocation. In theory you could have some sort of complex per-process scrambling system to avoid leaking information, but I think implementations actually just zero the memory. GPU drivers on different operating systems can be more or less buggy; Windows and Linux generally seem to do the right thing, but MacOS is a bit more haphazard.
- saagarjha 4y ago
  In theory yes. In practice, there have been bugs here.
hansendc 4y ago
Uh... Did I miss the patches that add a pre-zeroed page pool to Linux? Wouldn't be the first time I missed something like that getting added, but 6.3-rc5 definitely zeroes _some_ pages at allocation time, and I don't see any indiciation of it consulting a prezeroed page pool: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_alloc.c?h=v6.3-rc5&id=7e364e56293bb98cae1b55fd835f5991c4e96e7d#n2509 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
pmontra 4y ago
I thought that everybody knows how the CPU and the kernel manage memory. Everything else follows. However as a very good developer told me once, "I don't know how networking works, only that I have to type a URL." "How is that possible?" I replied, "They teach it at a school." And he told me "yes, but I studied graphic design, then discovered that I like programming." Replace and repeat with SQL, machine language, everything below modern programming languages.
ritcgab 4y ago
I think the main insight is that user-level memory allocation does not necessarily involve the kernel. If the application uses `malloc` to get memory allocated, the real "application" that will (possibly) request free pages from the kernel is the memory allocator. If the space requested can be served by the pages already assigned to the allocator, it can just hand them out. So when you `malloc`, it is not zeroed, unless you know it is freshly from kernel (but you don't know the internal state of the allocator). That is why one needs to manually zero out space returned from `malloc`, or use `calloc` instead.
jcarrano 4y ago
I'm curious, when did OSes first begin zeroing out pages. Did the first operating systems already zero the memory?
- saagarjha 4y ago
  OSes that have paging (and generally memory protection) have been zeroing them out from very early on, because part of the reason you have this is to avoid processes being able to mess with or spy on each other.
jk_i_am_a_robot 4y ago
If I could answer it would be a solid "maybe". If you're willing to consider the Translation Lookaside Buffer as part of the contents of RAM then, if the claims are to be believed, SPECTRE and Meltdown could read the contents of other processes' (and containers) RAM.
coppsilgold 4y ago
It is if you tell the kernel. zero memory on free (3-5% average system performance impact, due to touching cold memory) init_on_free=1 zero memory on alloc (<1% average system performance impact) init_on_alloc=1

309 points by Aachen 4y ago

I found the currently accepted answer (by user10489) interesting, tying together a lot of concepts I had heard of but didn't properly understand. For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" and either crash the program or let the program handle the failed allocation gracefully? This answer explains how this all works. The TL;DR answer to the actual question is: processes generally don't get access to each other's memory unless there is some trust relation (like being the parent process, or being allowed to attach a debugger), and being in a container doesn't change that, the same restrictions apply and you always get zeroed-out memory from the kernel. It's when you use a different allocator that you might get nonzeroed memory from elsewhere in your own process (not a random other process).

dale_glass 4y ago
> Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" Blame UNIX for that, and the fork() system call. It's a design quirk. fork() duplicates the process. So suppose your web browser consumes 10GB RAM out of the 16GB total on the system, and wants to run a process for anything. Like it just wants to exec something tiny, like `uname`. 1. 10GB process does fork(). 2. Instantly, you have two 10GB processes 3. A microsecond later, the child calls exec(), completely destroying its own state and replacing it with a 36K binary, freeing 10GB RAM. So there's two ways to go there: 1. You could require step 2 to be a full copy. Which means either you need more RAM, a huge chunk of which would always sit idle, or you need a lot of swap, for the same purpose. 2. We could overlook the memory usage increase and pretend that we have enough memory, and only really panic if the second process truly needs its own 10GB RAM that we don't have. That's what Linux does. The problem with #2 is that dealing with this happens completely in the background, at times completely unpredictable to the code. The OS allocates memory when the child changes memory, like does "a=1" somewhere. A program can't handle memory allocations failures there because as far as it knows, it's not allocating anything. So what you get is this fragile fiction that sometimes breaks and requires the kernel to kill something to maintain the system in some sort of working state. Windows doesn't have this issue at all because it has no fork(). New processes aren't children and start from scratch, so firefox never gets another 10GB sized clone. It just starts a new, 36K sized process.
- peheje 4y ago
  I find your writing style really pleasant and understandable! Much more so than the StackExchange answer. I really like the breakdown into steps, then what could happen steps and the follow ups. Where can I read more (from you?) in this style about OS and memory management?
- jacquesm 4y ago
  > 2. Instantly, you have two 10GB processes No, that's not how it works. The process table gets duplicated and copy-on-write takes care of the pages. As long as they are identical they will be shared, there is no way that 10GB of RAM will be allocated to the forked process and that all of the data will be copied.
  - worthless-trash 4y ago
    OP is referring to what would happen in the naive implementation, not what actually happens.
  - reisse 4y ago
    The data won't be copied, but kernel has to reserve the memory for both processes.
    - l33tman 4y ago
      No, it doesn't. See "overcommit"
      - enedil 4y ago
        In a sense it does. Page tables would need to be copied.
        jacquesm 4y ago
        Yes, but they all point to the same pages. The tables take up a small fraction of the memory of the pages themselves.
progbits 4y ago
If you would like to go deeper down the rabbit hole, fasterthanlime recently made a few videos that explore this topic: https://youtu.be/YB6LTaGRQJg https://youtu.be/YB6LTaGRQJg https://youtu.be/c_5Jy_AVDaM https://youtu.be/c_5Jy_AVDaM https://youtu.be/DpnXaNkM9_M https://youtu.be/DpnXaNkM9_M The code is in Rust but that doesn't matter for the explanation.
skyeto 4y ago
What I find to be a rather interesting tidbit related to this is that some applications (e.g. certain garbage collectors) map multiple ranges of virtual memory that address the same physical memory.
- yakubin 4y ago
  That’s how you can create a gapless ring buffer: <https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc2#remarks https://learn.microsoft.com/en-us/windows/win32/api/memoryap...> (see scenario 1 in examples).
jeffbee 4y ago
In Linux, a task that tries to allocate more than the allowed amount will be immediately terminated and not notified about anything. The reason out-of-memory systems thrash is merely because people do not always bother setting the limits, and the default assumes users prefer a process that continues through hours of thrashing instead of being promptly terminated.
- Sakos 4y ago
  Who is "people" in this? If you mean your average user, it's unrealistic to expect them to know or care about details like this. This is the kind of thing that needs sane defaults.
saagarjha 4y ago
> For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Generally browsers allocate large swaths of address space as guard pages (they're unmapped, but accesses into them will trap). Or they'll double map pages, so that the same physical page shows up multiple times in the virtual address space.

Proven 4y ago

[dead]

vrglvrglvrgl 4y ago

[dead]

adamgordonbell 4y ago

My shortcut for answering these questions is just replace 'container' with process and find the answer for that. A running container is just a process that is using certain kernel features: cgroup, namespaces, pivotroot. LXC has a different DX around containers than OCI containers do, but the same rules apply.

sgt 4y ago
DX?
- adamgordonbell 4y ago
  Developer Experience (Maybe there is a better term, but how you use them) Docker/ OCI containers tend to be a single process and LXC containers have a collection of processes. But in either case they are just processes running on the host in a different namespace. So they feel different to use, but use the same building blocks (to my understanding).
  - sgt 4y ago
    Unrelated, but your name rang a... bell. You're the corecursive guy. Great podcast!
    - adamgordonbell 4y ago
      Thanks!
  - Karellen 4y ago
    Shouldn't "DX" refer to the experience of people hacking on the Docker (or whatever) code itself? Like... how easy the build system is to work with if you're adding a new source file to the docker codebase? The people working with Docker, even if they are developers doing development work, are still users of Docker, aren't they? I mean, the GUI of an IDE is still part of its UX, right? Even though it's for developers doing development work?

worthless-trash 4y ago

One day, the general technical population will understand containers are just processes on a system. No magic under the hood.

__turbobrew__ 4y ago
Hey, if people keep thinking it is magic I keep getting paid to be a wizard

gregw2 4y ago

It’s not clear to me why the top rated reply which seemed extremely knowledgeable in discussing Linux kernel auto zeroing of memory mentioned a bunch of caveats including mentioning “some memory management libraries may reuse memory” but didn’t say “including glibc”. That’s a pretty big library to gloss over from a security or working programmer perspective! (Corrections to perspective welcome.)

hn92726819 4y ago
I thought the shared memory they were referring to would be read only memory or something. So yes, the memory is shared, but your process doesn't have any access to overwrite any of it. Likewise, when you call library functions, when it mallocs something, it uses memory that your process has write access to, not the shared library memory. In the context of the question, I assume the asker was mostly interested in reading some kind of sensitive data from a previous process, not reading the same librsry-code-only memory or something. Note: all of this could be wrong, it was just my understanding Edit: looks like this answers that: https://stackoverflow.com/questions/20857134/memory-write-protection-against-shared-library https://stackoverflow.com/questions/20857134/memory-write-pr...
IcePic 4y ago
Even if your libc IS one of the offenders, this libc code runs in a single process, so even though libc does or does not use evil tricks for your process, it doesn't mean you will get pages from some other process memory map, only that you may get your own old pages back.
- gregw2 4y ago
  Thanks, makes sense. (Looks like the poster commenting about glibc has corrected themself.)

bluedino 4y ago

Is the confusion coming from when you malloc() memory in C you aren't getting zereoed memory?

dale_glass 4y ago
I tried on Linux, and got all zeroes. Of course that's for a trivial program. If you freed something, that probably wasn't returned to the OS, and the next malloc might just recycle it.
- bluedino 4y ago
  Malloc a bunch of memory, dirty it, free it, then malloc some more. And then check. Your first request in your program, you'll get clean memory from the OS.
  - dale_glass 4y ago
    Yeah, that's what I'm saying.

CodesInChaos 4y ago

Is GPU memory wiped before it's handed to a new process, and is a process prevented from accessing GPU memory of a different process? (assuming low level APIs like Vulkan)

tryauuum 4y ago
this is certainly more interesting question than OPs
NotCamelCase 4y ago
You can request Windows to zero out every GPU memory allocation, but it's more of a driver thing that's not exposed in APIs. Such option is likely to be off by default in drivers as it might induce additional, unintended overhead. In practice, you are likely to see memory cleared more often than not due to other reasons, though. You can't just peek another process' GPU memory thru UMD app, either. Per-process virtual memory mechanisms similar to CPUs are also present in GPUs, which is the whole reason that resources are explicitly imported/exported across APIs via special API calls.
johntb86 4y ago
By the Vulkan spec it is guaranteed that memory from one process can't be seen by another. https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html#fundamentals-validusage https://registry.khronos.org/vulkan/specs/1.3-extensions/htm... says: In particular, any guarantees made by an operating system about whether memory from one process can be visible to another process or not must not be violated by a Vulkan implementation for any memory allocation. In theory you could have some sort of complex per-process scrambling system to avoid leaking information, but I think implementations actually just zero the memory. GPU drivers on different operating systems can be more or less buggy; Windows and Linux generally seem to do the right thing, but MacOS is a bit more haphazard.
saagarjha 4y ago
In theory yes. In practice, there have been bugs here.

hansendc 4y ago

Uh... Did I miss the patches that add a pre-zeroed page pool to Linux? Wouldn't be the first time I missed something like that getting added, but 6.3-rc5 definitely zeroes _some_ pages at allocation time, and I don't see any indiciation of it consulting a prezeroed page pool: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/page_alloc.c?h=v6.3-rc5&id=7e364e56293bb98cae1b55fd835f5991c4e96e7d#n2509 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

pmontra 4y ago

I thought that everybody knows how the CPU and the kernel manage memory. Everything else follows. However as a very good developer told me once, "I don't know how networking works, only that I have to type a URL." "How is that possible?" I replied, "They teach it at a school." And he told me "yes, but I studied graphic design, then discovered that I like programming." Replace and repeat with SQL, machine language, everything below modern programming languages.

ritcgab 4y ago

I think the main insight is that user-level memory allocation does not necessarily involve the kernel. If the application uses `malloc` to get memory allocated, the real "application" that will (possibly) request free pages from the kernel is the memory allocator. If the space requested can be served by the pages already assigned to the allocator, it can just hand them out. So when you `malloc`, it is not zeroed, unless you know it is freshly from kernel (but you don't know the internal state of the allocator). That is why one needs to manually zero out space returned from `malloc`, or use `calloc` instead.

jcarrano 4y ago

I'm curious, when did OSes first begin zeroing out pages. Did the first operating systems already zero the memory?

saagarjha 4y ago
OSes that have paging (and generally memory protection) have been zeroing them out from very early on, because part of the reason you have this is to avoid processes being able to mess with or spy on each other.

jk_i_am_a_robot 4y ago

If I could answer it would be a solid "maybe". If you're willing to consider the Translation Lookaside Buffer as part of the contents of RAM then, if the claims are to be believed, SPECTRE and Meltdown could read the contents of other processes' (and containers) RAM.

coppsilgold 4y ago

It is if you tell the kernel. zero memory on free (3-5% average system performance impact, due to touching cold memory) init_on_free=1 zero memory on alloc (<1% average system performance impact) init_on_alloc=1