4 ms·

This is not about memory allocation. OOM occurs when too much of the allocated memory is being used. For example, forking happens with copy-on-write memory, so

This is not about memory allocation. OOM occurs when too much of the allocated memory is being used. For example, forking happens with copy-on-write memory, so if I have a 3G process on a 4G machine, I can fork it 10 times without problems. It's only when those ten processes each start to write to their memory that the physical memory usage will quickly become too much to handle. In the scenario in the blog post, the OOM killer is disabled, and instead Linux can only prevent the memory from being accessed to avoid things getting worse. I would agree that this setup does sound like it needs fixing, but it's what we got in exchange for having cheap process forks.

JoshTriplett 12y ago
Ah, thanks for the clarification; you're right, there's no way around that, short of blocking forks and speculative allocations unless there's enough memory to back them. That's theoretically possible, but extremely harsh, given the common case of fork/exec for instance.
- andreasvc 12y ago
  I think it would be worthwhile to find a way around it, but it would require all applications to be conservative about their allocations. For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested. It's probably never going to happen, given that these primitives such as fork and malloc are so ingrained, but when you consider what Linux is being used for nowadays, from mission critical servers to satellites, it might make sense to make things more robust at the expense of compatibility.
  - EdiX 12y ago
    >For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested A fine grained version of fork, clone, clone already exists. But it's not just fork, call stacks are also part of this problem.
    - andreasvc 12y ago
      How are stacks part of the problem? Given tail-call optimization stacks shouldn't grow unbounded, right?
      - JoshTriplett 12y ago
        They shouldn't, but they do tend to have a large quantity of memory backing them that gets added to when grown into. The kernel uses 4k, 8k, or 16k stacks; a quick test on a Linux x86-64 system suggests that userspace has 8M stacks by default. Hackish test code to recurse infinitely and print the stack pointer until a segfault: #include <stdio.h> static inline unsigned long get_sp(void) { unsigned long ret; __asm__("mov %%rsp, %0" : "=r" (ret)); return ret; } void main(void) { printf("%lu\n", get_sp()); main(); } Running this and comparing the first and last values shows a stack size of about 8M.
      - EdiX 12y ago
        I didn't see this until now, the problem is that when you spawn a new thread space for the stack needs to be allocated. Since the stack can't be reallocated easily in C/C++ it also needs to be "big enough". The current implementation is to allocate several megabytes of stack for each thread, since memory accounting isn't strict this doesn't create problems because unused stack doesn't really take up space. If you start accounting memory you will also have to be stricter with allocating stacks for new threads, limiting yourself to just a page or two wherever possible. This is not an easy task.
WallWextra 12y ago
I do not understand. Where is the memory backing /proc/pid/cmdline and who in the container has a COW copy of it? If nobody, how can reading it cause more allocations in the container?
- andreasvc 12y ago
  COW due to forking was just an example of how use not allocation leads to problems. I got the impression that if the container does not solve its own OOM condition, the kernel simply resorts to blocking all memory access. Perhaps blocking reads is not strictly necessary, as they do not have to increase memory use, but on the other hand it's possible that some of the memory already had to be discarded due to the OOM condition so it could have become inaccessible.

JoshTriplett 12y ago

Ah, thanks for the clarification; you're right, there's no way around that, short of blocking forks and speculative allocations unless there's enough memory to back them. That's theoretically possible, but extremely harsh, given the common case of fork/exec for instance.

andreasvc 12y ago
I think it would be worthwhile to find a way around it, but it would require all applications to be conservative about their allocations. For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested. It's probably never going to happen, given that these primitives such as fork and malloc are so ingrained, but when you consider what Linux is being used for nowadays, from mission critical servers to satellites, it might make sense to make things more robust at the expense of compatibility.
- EdiX 12y ago
  >For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested A fine grained version of fork, clone, clone already exists. But it's not just fork, call stacks are also part of this problem.
  - andreasvc 12y ago
    How are stacks part of the problem? Given tail-call optimization stacks shouldn't grow unbounded, right?
    - JoshTriplett 12y ago
      They shouldn't, but they do tend to have a large quantity of memory backing them that gets added to when grown into. The kernel uses 4k, 8k, or 16k stacks; a quick test on a Linux x86-64 system suggests that userspace has 8M stacks by default. Hackish test code to recurse infinitely and print the stack pointer until a segfault: #include <stdio.h> static inline unsigned long get_sp(void) { unsigned long ret; __asm__("mov %%rsp, %0" : "=r" (ret)); return ret; } void main(void) { printf("%lu\n", get_sp()); main(); } Running this and comparing the first and last values shows a stack size of about 8M.
    - EdiX 12y ago
      I didn't see this until now, the problem is that when you spawn a new thread space for the stack needs to be allocated. Since the stack can't be reallocated easily in C/C++ it also needs to be "big enough". The current implementation is to allocate several megabytes of stack for each thread, since memory accounting isn't strict this doesn't create problems because unused stack doesn't really take up space. If you start accounting memory you will also have to be stricter with allocating stacks for new threads, limiting yourself to just a page or two wherever possible. This is not an easy task.

WallWextra 12y ago

I do not understand. Where is the memory backing /proc/pid/cmdline and who in the container has a COW copy of it? If nobody, how can reading it cause more allocations in the container?

andreasvc 12y ago
COW due to forking was just an example of how use not allocation leads to problems. I got the impression that if the container does not solve its own OOM condition, the kernel simply resorts to blocking all memory access. Perhaps blocking reads is not strictly necessary, as they do not have to increase memory use, but on the other hand it's possible that some of the memory already had to be discarded due to the OOM condition so it could have become inaccessible.