6 ms·

"Memory allocation "doesn't fail"" That's not quite it. But it does deserve a double-take. When you request memory, it generally works out fine. But it doesn'

by cturner 10y ago

"Memory allocation "doesn't fail"" That's not quite it. But it does deserve a double-take. When you request memory, it generally works out fine. But it doesn't actually give you a lock on the pages you need until you request those pages. This leads to weird situations where memory is oversubscribed. When this happens the OS casts one of your processes into the rift. Google for "oom killer". To me this sounds insane - a basic breach in the contract of malloc. However, it's possible that this is one of those situations that involves a complex tradeoff that you won't grok until you've been the guy building it. It would be interesting to know what other OSs do. Something that is annoying: as far as I know there is no signal that fires when this happens. Instead, you scrape logs to learn about it. Whereupon you will reboot the box, because you no longer trust its state. You can get a callback when this happens if you use systemtap. In practice it's less of a problem than you'd think. If you care about the thing you're building, you will have made it host-independent. From the perspective of production software, there's no difference between a rifted process and a motherboard death. If you're using sockets as your IPC mechanism, it doesn't matter why the process died, just that it did.

mikeash 10y ago
To elaborate a bit, there are basically three ways to do this stuff. One is to commit memory on request. If all of your RAM and swap is committed, then malloc will fail. This is probably how we expect it to work, but it can be really wasteful. Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. Another way is to overcommit, where malloc doesn't actually reserve memory initially but only on first use, but only pause processes when memory fills up. If a process needs more memory and there isn't any more available, that process blocks until more memory becomes available. This means nobody gets killed, but it's not uncommon for the system to just deadlock like this forever once memory fills up. And the third way is as you describe, to overcommit and kill processes when you run out of memory. Note that there's no right answer, just different tradeoffs. Linux lets you configure this to an extent as well, so if you don't like the default, you can change it!
- gcr 10y ago
  After a fork, does the amount of committed memory increase by the size of the child's allocated virtual memory space? I'm curious how committed memory interacts with COW fork semantics. After all, a child can just overwrite the COW/copy of the memory it inherited from its parent without allocating anything else.
  - haldean 10y ago
    No, all of the child's memory mappings are shared with its parent. Upon write, though, that becomes a new committed mapping; I believe the entire page is copied, so you could end up committing another 4k with a one-byte write from the child.
    - gcr 10y ago
      Hm. Then suppose I turn off the OOM killer. What happens to a process that writes to its parent's page if the system has no memory left to commit?
  - tedunangst 10y ago
    COW is usually a kind of loophole. Most systems only count the pages once (otherwise you'd have to reject the fork, but that becomes inconvenient fast).
- tedunangst 10y ago
  > It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. But at least then it's the program that doesn't handle failure that dies, not one that does!
- paulddraper 10y ago
  > Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. ...and it would be absurd for bad programs to eat memory and crash? We must have other programs pay for its sins?
gcr 10y ago
There's a switch you can toggle that will turn off the OOM (vm.overcommit_memory). The downside is memory allocations that would oversubscribe memory will immediately fail. That means processes won't be able to allocate tens of gigabytes of virtual memory anymore. If I understand the limitation correctly, this also means that all process's allocated virtual memory can no longer exceed the amount of physical+swap memory in the machine. I think this probably interacts badly with COW memory semantics for child processes: even though children and parent reuse address space, the kernel can't know whether the child will write to the memory it inherited from its parent's memory map. I think if the kernel is being too careful, all "shared" memory would have to be accounted separately, which would severely limit the number of processes you could have open. (Is that really what happens? That seems awfully limiting. I hope I'm wrong.)
- deathanatos 10y ago
  You can still allocate more virtual memory; it just has to be backed by a file, e.g., through mmap(). So, for example, what mongod does — mmap() in its entire DB, which can be terabytes of RAM — would still work. You can almost think of file-backed mmaps as swap, in fact.
SomeStupidPoint 10y ago
I appreciate your reply (and replies to your reply), but I must be missing something about why this is a good idea. Is it just that software is (badly) coded to request lots of memory it doesn't intend to use? The only sensible usecase I can think of for that is oversubscribing VMs, but even there... Doesn't this just crash a VM when there's contention for the memory that's oversubscribed? I guess that would be okay for the VM case, but why would I want this in a normal usecase? It seems like it's just encouraging/supporting bad memory management in software. Have I just been lucky that this has never killed a long-running job of mine when I use 90%+ of RAM to do a big job? Or shit, maybe it has and the specific error just was obscured. Ugh, I hate when the system lies by default. How does that help me as a user?
- deleted 10y ago
  [deleted]
- Asooka 10y ago
  It does help you as a user, since the OOM killer tries to be somewhat sensible in what it kills. Imagine you have 10 well-behaved programs and 1 program that just leaks 1GB/h. After a few hours your entire RAM has been committed and the next process to request memory will fail. In this scenario, the process that crashes is random - it might be the leaky program, or it might be your desktop shell. The OOM killer on the other hand will definitely kill the leaky program once there is literally no more memory available. It's a combination of programs not using up all pages they mmap and there being no good ordering of which memory requests are more important that leads to overcommitting and the OOM killer being generally good ideas. In my experience that's exactly how it works out - the moment you start something that gobbles up memory due to a bug, it gets killed promptly, rather than some other random program crashing due to being unable to allocate memory.
fhars 10y ago
The real drawback of disabling memory overcommitting is not that processes can no longer allocate huge amounts of unused memory, but that processes that use more memory that the remaining availabe RAM + swap can no longer call fork (even if the forked process like most forked processes would immediately discard all the shared memory by execing some small program).
deathanatos 10y ago
> Something that is annoying: as far as I know there is no signal that fires when this happens. Instead, you scrape logs to learn about it. Whereupon you will reboot the box, because you no longer trust its state. You can get a callback when this happens if you use systemtap. It's a SIGKILL, which isn't catchable/handleable by the process (how could it be?). The machine should still be stable after an OOM, just minus a few process. (In my production setups, usually a supervising daemon can restart the now-dead child, and begin the memory leaking cycle anew.)

by cturner 10y ago

mikeash 10y ago

To elaborate a bit, there are basically three ways to do this stuff. One is to commit memory on request. If all of your RAM and swap is committed, then malloc will fail. This is probably how we expect it to work, but it can be really wasteful. Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. Another way is to overcommit, where malloc doesn't actually reserve memory initially but only on first use, but only pause processes when memory fills up. If a process needs more memory and there isn't any more available, that process blocks until more memory becomes available. This means nobody gets killed, but it's not uncommon for the system to just deadlock like this forever once memory fills up. And the third way is as you describe, to overcommit and kill processes when you run out of memory. Note that there's no right answer, just different tradeoffs. Linux lets you configure this to an extent as well, so if you don't like the default, you can change it!

gcr 10y ago
After a fork, does the amount of committed memory increase by the size of the child's allocated virtual memory space? I'm curious how committed memory interacts with COW fork semantics. After all, a child can just overwrite the COW/copy of the memory it inherited from its parent without allocating anything else.
- haldean 10y ago
  No, all of the child's memory mappings are shared with its parent. Upon write, though, that becomes a new committed mapping; I believe the entire page is copied, so you could end up committing another 4k with a one-byte write from the child.
  - gcr 10y ago
    Hm. Then suppose I turn off the OOM killer. What happens to a process that writes to its parent's page if the system has no memory left to commit?
- tedunangst 10y ago
  COW is usually a kind of loophole. Most systems only count the pages once (otherwise you'd have to reject the fork, but that becomes inconvenient fast).
tedunangst 10y ago
> It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. But at least then it's the program that doesn't handle failure that dies, not one that does!
paulddraper 10y ago
> Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway. ...and it would be absurd for bad programs to eat memory and crash? We must have other programs pay for its sins?

gcr 10y ago

There's a switch you can toggle that will turn off the OOM (vm.overcommit_memory). The downside is memory allocations that would oversubscribe memory will immediately fail. That means processes won't be able to allocate tens of gigabytes of virtual memory anymore. If I understand the limitation correctly, this also means that all process's allocated virtual memory can no longer exceed the amount of physical+swap memory in the machine. I think this probably interacts badly with COW memory semantics for child processes: even though children and parent reuse address space, the kernel can't know whether the child will write to the memory it inherited from its parent's memory map. I think if the kernel is being too careful, all "shared" memory would have to be accounted separately, which would severely limit the number of processes you could have open. (Is that really what happens? That seems awfully limiting. I hope I'm wrong.)

deathanatos 10y ago
You can still allocate more virtual memory; it just has to be backed by a file, e.g., through mmap(). So, for example, what mongod does — mmap() in its entire DB, which can be terabytes of RAM — would still work. You can almost think of file-backed mmaps as swap, in fact.

SomeStupidPoint 10y ago

I appreciate your reply (and replies to your reply), but I must be missing something about why this is a good idea. Is it just that software is (badly) coded to request lots of memory it doesn't intend to use? The only sensible usecase I can think of for that is oversubscribing VMs, but even there... Doesn't this just crash a VM when there's contention for the memory that's oversubscribed? I guess that would be okay for the VM case, but why would I want this in a normal usecase? It seems like it's just encouraging/supporting bad memory management in software. Have I just been lucky that this has never killed a long-running job of mine when I use 90%+ of RAM to do a big job? Or shit, maybe it has and the specific error just was obscured. Ugh, I hate when the system lies by default. How does that help me as a user?

deleted 10y ago
[deleted]
Asooka 10y ago
It does help you as a user, since the OOM killer tries to be somewhat sensible in what it kills. Imagine you have 10 well-behaved programs and 1 program that just leaks 1GB/h. After a few hours your entire RAM has been committed and the next process to request memory will fail. In this scenario, the process that crashes is random - it might be the leaky program, or it might be your desktop shell. The OOM killer on the other hand will definitely kill the leaky program once there is literally no more memory available. It's a combination of programs not using up all pages they mmap and there being no good ordering of which memory requests are more important that leads to overcommitting and the OOM killer being generally good ideas. In my experience that's exactly how it works out - the moment you start something that gobbles up memory due to a bug, it gets killed promptly, rather than some other random program crashing due to being unable to allocate memory.

fhars 10y ago

The real drawback of disabling memory overcommitting is not that processes can no longer allocate huge amounts of unused memory, but that processes that use more memory that the remaining availabe RAM + swap can no longer call fork (even if the forked process like most forked processes would immediately discard all the shared memory by execing some small program).

deathanatos 10y ago

> Something that is annoying: as far as I know there is no signal that fires when this happens. Instead, you scrape logs to learn about it. Whereupon you will reboot the box, because you no longer trust its state. You can get a callback when this happens if you use systemtap. It's a SIGKILL, which isn't catchable/handleable by the process (how could it be?). The machine should still be stable after an OOM, just minus a few process. (In my production setups, usually a supervising daemon can restart the now-dead child, and begin the memory leaking cycle anew.)