4 ms·

So once you've allocate more memory than you have, what happens when you to to write to any of it?

by afhof 14y ago

scvalex1 14y ago
Everything's ok until you try to write to more memory than is available. At that point, either your program segfaults, or Linux's Out-Of-Memory (OOM) Killer is invoked and kills a random process (not necessarily the offending process).
- rosser 14y ago
  (not necessarily the offending process) It's terrible about this with PostgreSQL. The OOM Killer tends to thump the postmaster, not the offending backend.
  - EdiX 14y ago
    You should instruct the OOM Killer to not kill PostgreSQL. Memory is a global resource, even if you didn't overcommit the process getting the allocation failure is not guaranteed to be the misbehaving one, it's just the unlucky process that did the first allocation after all the memory was gone (this is also why handling allocation errors in your application is usually impossible, even without an overcommitting kernel).
  - pyre 14y ago
    It's not something that's easy to develop a general solution to. Maybe the process that's using a lot of RAM is not runaway, and is mission-critical. Maybe it is a runaway process and something else is mission-critical. The most 'fair' way to go about it is to just kill a random process. That skits the whole issue.
jws 14y ago
I altered the program to first allocate 64G then go through and write on each page of the allocation on my 16G machine, dropped my swap partition. After suitable screwing around to keep gcc from optimizing away the writes, it goes a bit like this… jim@gvdev:~$ ./fail Before Allocated 1 GB Allocated 2 GB ... Allocated 64 GB After Writing in 0 GB Writing in 1 GB ... Writing in 15 GB ... rather long delay here... Write failed: Broken pipe upstairs:~ jim$ <----- bad news here, notice the machine. upstairs:~ jim$ slogin gvdev.XXXXX.net ssh: connect to host gvdev.XXXXX.net port 22: Network is unreachable … so now I have to get back in "going out" clothes, go downstairs, and drive over to gvdev and reboot it.
- mileswu 14y ago
  Kinda OT, but out of interest, why do you use 'slogin' instead of 'ssh'? Just old habits dieing hard from 'rlogin'?
  - jws 14y ago
    Yes.
- xentronium 14y ago
  So, when there is no memory and no swap space available, instead of crashing, it zombifies whole system? This is weird.
  - keeperofdakeys 14y ago
    The linux Out Of Memory killer would have activated (if not disabled). This program derives a 'score' for each program, based on a number of statistics, and kills the process with the highest score. Although it usually makes the right choice, sometimes it can kill innocent programs.
    - jws 14y ago
      It was worse than a dead sshd. Looks like something kernel-ish died badly. There was one more cron message in the syslog after this, but then nothing until I rebooted. Console unresponsive, no ping answering. Nov 2 21:29:33 gvdev kernel: [15115318.546376] fail[17150]: segfault at 7fffe9cbc1e0 ip 00000000004006fe sp 00007fffe9cbb1d0 error 4 in fail[400000+1000]
- antirez 14y ago
  This is a problem with virtual machines vendors such as Linode where the configured swap space is usually tiny by default... It's much better to provide more swap (At least as much as RAM size) and see the performances start to deteriorate as a sign of alarm of overcommit, instead of having random processes killed.
- 14y ago
mjg59 14y ago
That depends on what you mean "more memory than you have". If it's more physical RAM than is currently available, not necessarily a problem - some other processes can be pushed out of RAM into swap. If it's more than that, some pages can be discarded because they're backed by binaries on disk. If it's more than that, some processes can be killed. If it's more than that, really bad things happen. The problem is that there's no good way to differentiate between a process that merely wants more VM that it'll never touch and a process that wants actual RAM. In theory, every fork() should double your VM requirements - in practice, almost all are immediately followed by an exec() and the previous allocations are irrelevant. Do you fail fork() because you can't guarantee you can back every modified page in the new process? If not, why should you fail malloc() because you can't guarantee you can back it? There's no correct answer, and Linux lets you control which answer you want via /proc/sys/vm/overcommit_memory .
- pootch 14y ago
  Actually on solaris, the fork does OOM a java VM. So while this is a pragmatic solution on linux, its also a false promise which will paper over some transient memory requirements while leaving a huge gap in regards to realistic resource expectations. Unfortunately this problem is magnified when using virtual OS instances, and shitty hardware.
tjdetwiler 14y ago
Zero-Fill on Demand - All new pages map to a single zero "frame" in physical memory. In this case, you don't get any memory until you write to the page, at which time the OS steps in and allocates you a page of all zeros EDIT: this means if the kernel can't allocate more memory when you write then the kernel has a decision to make; kill you, or kill someone else to get more memory.

So once you've allocate more memory than you have, what happens when you to to write to any of it?

by afhof 14y ago

So once you've allocate more memory than you have, what happens when you to to write to any of it?

Everything's ok until you try to write to more memory than is available. At that point, either your program segfaults, or Linux's Out-Of-Memory (OOM) Killer is invoked and kills a random process (not necessarily the offending process).

rosser 14y ago
(not necessarily the offending process) It's terrible about this with PostgreSQL. The OOM Killer tends to thump the postmaster, not the offending backend.
- EdiX 14y ago
  You should instruct the OOM Killer to not kill PostgreSQL. Memory is a global resource, even if you didn't overcommit the process getting the allocation failure is not guaranteed to be the misbehaving one, it's just the unlucky process that did the first allocation after all the memory was gone (this is also why handling allocation errors in your application is usually impossible, even without an overcommitting kernel).
- pyre 14y ago
  It's not something that's easy to develop a general solution to. Maybe the process that's using a lot of RAM is not runaway, and is mission-critical. Maybe it is a runaway process and something else is mission-critical. The most 'fair' way to go about it is to just kill a random process. That skits the whole issue.

jws 14y ago

I altered the program to first allocate 64G then go through and write on each page of the allocation on my 16G machine, dropped my swap partition. After suitable screwing around to keep gcc from optimizing away the writes, it goes a bit like this… jim@gvdev:~$ ./fail Before Allocated 1 GB Allocated 2 GB ... Allocated 64 GB After Writing in 0 GB Writing in 1 GB ... Writing in 15 GB ... rather long delay here... Write failed: Broken pipe upstairs:~ jim$ <----- bad news here, notice the machine. upstairs:~ jim$ slogin gvdev.XXXXX.net ssh: connect to host gvdev.XXXXX.net port 22: Network is unreachable … so now I have to get back in "going out" clothes, go downstairs, and drive over to gvdev and reboot it.

mileswu 14y ago
Kinda OT, but out of interest, why do you use 'slogin' instead of 'ssh'? Just old habits dieing hard from 'rlogin'?
- jws 14y ago
  Yes.
xentronium 14y ago
So, when there is no memory and no swap space available, instead of crashing, it zombifies whole system? This is weird.
- keeperofdakeys 14y ago
  The linux Out Of Memory killer would have activated (if not disabled). This program derives a 'score' for each program, based on a number of statistics, and kills the process with the highest score. Although it usually makes the right choice, sometimes it can kill innocent programs.
  - jws 14y ago
    It was worse than a dead sshd. Looks like something kernel-ish died badly. There was one more cron message in the syslog after this, but then nothing until I rebooted. Console unresponsive, no ping answering. Nov 2 21:29:33 gvdev kernel: [15115318.546376] fail[17150]: segfault at 7fffe9cbc1e0 ip 00000000004006fe sp 00007fffe9cbb1d0 error 4 in fail[400000+1000]
antirez 14y ago
This is a problem with virtual machines vendors such as Linode where the configured swap space is usually tiny by default... It's much better to provide more swap (At least as much as RAM size) and see the performances start to deteriorate as a sign of alarm of overcommit, instead of having random processes killed.
14y ago

mjg59 14y ago

That depends on what you mean "more memory than you have". If it's more physical RAM than is currently available, not necessarily a problem - some other processes can be pushed out of RAM into swap. If it's more than that, some pages can be discarded because they're backed by binaries on disk. If it's more than that, some processes can be killed. If it's more than that, really bad things happen. The problem is that there's no good way to differentiate between a process that merely wants more VM that it'll never touch and a process that wants actual RAM. In theory, every fork() should double your VM requirements - in practice, almost all are immediately followed by an exec() and the previous allocations are irrelevant. Do you fail fork() because you can't guarantee you can back every modified page in the new process? If not, why should you fail malloc() because you can't guarantee you can back it? There's no correct answer, and Linux lets you control which answer you want via /proc/sys/vm/overcommit_memory .

pootch 14y ago
Actually on solaris, the fork does OOM a java VM. So while this is a pragmatic solution on linux, its also a false promise which will paper over some transient memory requirements while leaving a huge gap in regards to realistic resource expectations. Unfortunately this problem is magnified when using virtual OS instances, and shitty hardware.

tjdetwiler 14y ago

Zero-Fill on Demand - All new pages map to a single zero "frame" in physical memory. In this case, you don't get any memory until you write to the page, at which time the OS steps in and allocates you a page of all zeros EDIT: this means if the kernel can't allocate more memory when you write then the kernel has a decision to make; kill you, or kill someone else to get more memory.