8 ms·

Glad to see this issue raised! My system hangs for minutes sometimes and is very frustrating compared to Windows and OSX which seem to handle out of memory in a

Glad to see this issue raised! My system hangs for minutes sometimes and is very frustrating compared to Windows and OSX which seem to handle out of memory in a much more user-friendly way. Which seems to be: suspending the offending program and letting the user decide what to do from there. I'm sure there's a reason the Linux kernel doesn't do something similar, but can anyone enlighten me? :)

deleted 7y ago
[deleted]
adamnemecek 7y ago
I'm not sure but the assumption might be that there's generally no user to ask as the computer might be a server.
- IshKebab 7y ago
  Right but if there is a user to ask then it should ask!!
noncoml 7y ago
Ehm, and how does the OS know which is the “offending” process? I think you are confusing the issue raised here with your desktop experience.
- brianpgordon 7y ago
  Currently the Linux kernel computes a score for each process based on some heuristics. There's a good introductory article on LWN: https://lwn.net/Articles/317814/ https://lwn.net/Articles/317814/
  - lazyguy 7y ago
    Yep and it's about as good as just picking a random process and killing it. It's awesome when you run out of memory and you try to log in only to have it kill sshd.
    - Dylan16807 7y ago
      I blame the specific algorithm for that, not the basic concept. Nothing with less than 10MB of memory use should ever get killed unless you're in some kind of fork bomb.
    - NikkiA 7y ago
      Running sshd as an on-demand (Type=socket) service would probably work better, since then the sshd process would be new and thus have a better heuristic score - also not be tying up memory sitting unused in the meantime. systemd still seems to run it (Type=notify) with the -D option all the time though, at least on the systems I can check. Dropbear is configured by default as a Type=socket service though.
      - bch 7y ago
        This is sort of just kicking the problem down the road. Your idea actually might work for (presumably) low-volume use ssh, but what about the next important service? When does the work-around to a papered-over work-around to a virtual problem that is supposed to just be RAM-backed or handled at ptr = malloc(42); if(!ptr) exit_error(); end?
        NikkiA 7y ago
        Well, there probably needs to be a way to override the heuristic at least, sort of a 'this process is important, don't auto-kill it if trying to find memory'. As for ssh specifically, I rarely ssh into my desktop machine, but I keep sshd running for just this kind of situation where I might need to try and rescue a swamped machine. So in most cases low-volume sshd use is exactly what is called for. If you're running into the memory purge of doom on a server that's probably a whole different nightmare scenario. malloc returning NULL has been a broken assumption for a long time though, and that isn't going to change afaik.
    - wyldfire 7y ago
      A classic from [1]: > An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused. [1] https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
  - silon42 7y ago
    Probably it would be good to include the amount of overcommit in the heuristic. Processes that overcommit should be killed first. What is a good way to measure/inspect overcommit of the process?
mrtweetyhack 7y ago
And you've turned swap off on Windows?
- anticensor 7y ago
  Windows will not BSOD due to memory pressure.
  - StreamBright 7y ago
    I think it happens only if you have a broken device driver, applications cannot cause BSOD with memory allocation.
  - 1-6 7y ago
    Does it BSOD if it runs out of swap space though?
  - EmpirePhoenix 7y ago
    Well I beg to differ, using a unlimited swap file can quickly reach hard issues after 64gb of swap use. At that point mallocs in the windows ui fail (timouts or something?), that apparently are not meant to, eg fonts from shutdown menu missing, the system being unable to shutdown ect.
yjftsjthsd-h 7y ago
Probably lack of integration; if NT hits a memory issue, it can just pass notice to the tightly-coupled userland and GUI. If Linux runs out of memory, even if it internally knows what process to blame... What would it do that makes sense for a headless server, TiVo, and Android phone? Keeping in mind that the kernel folks don't even work that closely with many userspace vendors.
- useerup 7y ago
  Windows (server and desktop versions) will throw up a message dialog on the screen. It will also start to kill off processes just enough to resolve the low memory situation. During this - unlike Linux - you can actually use the mouse, CLI and close programs yourself. On top of that server applications like IIS has built-in watchdogs. If an IIS process grows to use too much memory (60% IIRC) or excessive CPU, the watchdog will recycle the process.
  - StreamBright 7y ago
    I think Windows kernels do not use overcommit, so memory allocation will fail if you run out of memory.
- chacham15 7y ago
  Throw a signal like it would do if the process were out of memory completely and about to be killed? (for clarification, no snark intended, actual question)
  - saagarjha 7y ago
    What signal is sent when a process is out of memory? I thought malloc would either start returning NULL or you’d fault when trying to access overcommitted memory.
    - joosters 7y ago
      Yes, but ideally you want to be throwing some ‘memory pressure’ signals before absolutely running out of memory, so that programs can take simple actions like emptying caches, etc. Catching an otherwise-fatal out-of-memory fault and recovering would be too complicated / bug-prone.
ieoei9jjd 7y ago
Linux use cases tend to be servers where user interaction is unexpected at 3AM? No one around to make a choice, so automate a choice. IMO despite the standout behavior, I prefer my software to deal with itself. Systems designed to wait for user input end up having design choices intent on keeping a user using them. Software is just a tool. Not a lifestyle. Set and forget this shit as much as possible
- hvidgaard 7y ago
  If the alternative is simply killing a process or crashing the kernel, then surely a better approach would be to suspend the offending process and call a handle that does something. If you want that something to restart the machine, fine. You want it to notify the administrator, fine.
- pmontra 7y ago
  There is the use case of android phones. One of the answers to the OP is about that case. It sends that Google developed a user space process to monitor those events https://lkml.org/lkml/2019/8/5/1121 https://lkml.org/lkml/2019/8/5/1121 From that reply it seems that Facebook implemented something similar, I guess for their servers.
- jcelerier 7y ago
  > Linux use cases tend to be servers where user interaction is unexpected at 3AM? No one around to make a choice, so automate a choice. Even if it's a small percentage of the overall "computing" population, there are still millions of people running Linux on the desktop (roughly 2% out of 3.2 billion people using internet makes for 64 million - a large european country). It's 64 million of people for which this behaviour is a pain in the arse.
LifeIsBio 7y ago
I used to run into this problem all the time in grad school. Once a month or so I'd load a data set, do some dumb Python operation on it that took significantly more memory than I predicted, and BAM! I'd have to restart my laptop. I just kinda assumed that's how computers worked until I got a Mac a couple of months ago... The link suggests that there might be some default parameters you could change to protect against this behavior. Does anyone have any suggestions on what settings to change?
- kccqzy 7y ago
  A Mac is certainly better at handling these kinds of issues but it's by no means totally safe. It tries to compress memory and dynamically allocate more swap, but there's still a limit and you can see that if you accidentally run programs with way higher RAM requirement than you have. I've had multiple occasions where my program used so much RAM that even moving the cursor is an exercise in patience, never mind switching to a terminal window and typing commands to kill the process.
  - Avamander 7y ago
    But at least you can recover rather cleanly from the issue.
  - savoytruffle 7y ago
    A Mac will keep creating virtual memory swap up to some limit (some multiple of the amount of physical RAM — can't quite remember, possibly 5x) and then it will produce a kind of vague dialog box saying "You've run out of application memory" with a list of applications to force quit.
1-6 7y ago
Does FreeBSD handle this issue better?
caf 7y ago
If you've foolishly decided to run without swap (like the original post), then suspending the offending program does nothing. This is because the offending program has allocated a lot of private dirty pages, which can't be dropped from memory because without swap space, there is nowhere for it to go.

deleted 7y ago

[deleted]

adamnemecek 7y ago

I'm not sure but the assumption might be that there's generally no user to ask as the computer might be a server.

IshKebab 7y ago
Right but if there is a user to ask then it should ask!!

noncoml 7y ago

Ehm, and how does the OS know which is the “offending” process? I think you are confusing the issue raised here with your desktop experience.

brianpgordon 7y ago
Currently the Linux kernel computes a score for each process based on some heuristics. There's a good introductory article on LWN: https://lwn.net/Articles/317814/ https://lwn.net/Articles/317814/
- lazyguy 7y ago
  Yep and it's about as good as just picking a random process and killing it. It's awesome when you run out of memory and you try to log in only to have it kill sshd.
  - Dylan16807 7y ago
    I blame the specific algorithm for that, not the basic concept. Nothing with less than 10MB of memory use should ever get killed unless you're in some kind of fork bomb.
  - NikkiA 7y ago
    Running sshd as an on-demand (Type=socket) service would probably work better, since then the sshd process would be new and thus have a better heuristic score - also not be tying up memory sitting unused in the meantime. systemd still seems to run it (Type=notify) with the -D option all the time though, at least on the systems I can check. Dropbear is configured by default as a Type=socket service though.
    - bch 7y ago
      This is sort of just kicking the problem down the road. Your idea actually might work for (presumably) low-volume use ssh, but what about the next important service? When does the work-around to a papered-over work-around to a virtual problem that is supposed to just be RAM-backed or handled at ptr = malloc(42); if(!ptr) exit_error(); end?
      - NikkiA 7y ago
        Well, there probably needs to be a way to override the heuristic at least, sort of a 'this process is important, don't auto-kill it if trying to find memory'. As for ssh specifically, I rarely ssh into my desktop machine, but I keep sshd running for just this kind of situation where I might need to try and rescue a swamped machine. So in most cases low-volume sshd use is exactly what is called for. If you're running into the memory purge of doom on a server that's probably a whole different nightmare scenario. malloc returning NULL has been a broken assumption for a long time though, and that isn't going to change afaik.
  - wyldfire 7y ago
    A classic from [1]: > An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused. [1] https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
- silon42 7y ago
  Probably it would be good to include the amount of overcommit in the heuristic. Processes that overcommit should be killed first. What is a good way to measure/inspect overcommit of the process?

mrtweetyhack 7y ago

And you've turned swap off on Windows?

anticensor 7y ago
Windows will not BSOD due to memory pressure.
- StreamBright 7y ago
  I think it happens only if you have a broken device driver, applications cannot cause BSOD with memory allocation.
- 1-6 7y ago
  Does it BSOD if it runs out of swap space though?
- EmpirePhoenix 7y ago
  Well I beg to differ, using a unlimited swap file can quickly reach hard issues after 64gb of swap use. At that point mallocs in the windows ui fail (timouts or something?), that apparently are not meant to, eg fonts from shutdown menu missing, the system being unable to shutdown ect.

yjftsjthsd-h 7y ago

Probably lack of integration; if NT hits a memory issue, it can just pass notice to the tightly-coupled userland and GUI. If Linux runs out of memory, even if it internally knows what process to blame... What would it do that makes sense for a headless server, TiVo, and Android phone? Keeping in mind that the kernel folks don't even work that closely with many userspace vendors.

useerup 7y ago
Windows (server and desktop versions) will throw up a message dialog on the screen. It will also start to kill off processes just enough to resolve the low memory situation. During this - unlike Linux - you can actually use the mouse, CLI and close programs yourself. On top of that server applications like IIS has built-in watchdogs. If an IIS process grows to use too much memory (60% IIRC) or excessive CPU, the watchdog will recycle the process.
- StreamBright 7y ago
  I think Windows kernels do not use overcommit, so memory allocation will fail if you run out of memory.
chacham15 7y ago
Throw a signal like it would do if the process were out of memory completely and about to be killed? (for clarification, no snark intended, actual question)
- saagarjha 7y ago
  What signal is sent when a process is out of memory? I thought malloc would either start returning NULL or you’d fault when trying to access overcommitted memory.
  - joosters 7y ago
    Yes, but ideally you want to be throwing some ‘memory pressure’ signals before absolutely running out of memory, so that programs can take simple actions like emptying caches, etc. Catching an otherwise-fatal out-of-memory fault and recovering would be too complicated / bug-prone.

ieoei9jjd 7y ago

Linux use cases tend to be servers where user interaction is unexpected at 3AM? No one around to make a choice, so automate a choice. IMO despite the standout behavior, I prefer my software to deal with itself. Systems designed to wait for user input end up having design choices intent on keeping a user using them. Software is just a tool. Not a lifestyle. Set and forget this shit as much as possible

hvidgaard 7y ago
If the alternative is simply killing a process or crashing the kernel, then surely a better approach would be to suspend the offending process and call a handle that does something. If you want that something to restart the machine, fine. You want it to notify the administrator, fine.
pmontra 7y ago
There is the use case of android phones. One of the answers to the OP is about that case. It sends that Google developed a user space process to monitor those events https://lkml.org/lkml/2019/8/5/1121 https://lkml.org/lkml/2019/8/5/1121 From that reply it seems that Facebook implemented something similar, I guess for their servers.
jcelerier 7y ago
> Linux use cases tend to be servers where user interaction is unexpected at 3AM? No one around to make a choice, so automate a choice. Even if it's a small percentage of the overall "computing" population, there are still millions of people running Linux on the desktop (roughly 2% out of 3.2 billion people using internet makes for 64 million - a large european country). It's 64 million of people for which this behaviour is a pain in the arse.

LifeIsBio 7y ago

I used to run into this problem all the time in grad school. Once a month or so I'd load a data set, do some dumb Python operation on it that took significantly more memory than I predicted, and BAM! I'd have to restart my laptop. I just kinda assumed that's how computers worked until I got a Mac a couple of months ago... The link suggests that there might be some default parameters you could change to protect against this behavior. Does anyone have any suggestions on what settings to change?

kccqzy 7y ago
A Mac is certainly better at handling these kinds of issues but it's by no means totally safe. It tries to compress memory and dynamically allocate more swap, but there's still a limit and you can see that if you accidentally run programs with way higher RAM requirement than you have. I've had multiple occasions where my program used so much RAM that even moving the cursor is an exercise in patience, never mind switching to a terminal window and typing commands to kill the process.
- Avamander 7y ago
  But at least you can recover rather cleanly from the issue.
- savoytruffle 7y ago
  A Mac will keep creating virtual memory swap up to some limit (some multiple of the amount of physical RAM — can't quite remember, possibly 5x) and then it will produce a kind of vague dialog box saying "You've run out of application memory" with a list of applications to force quit.

1-6 7y ago

Does FreeBSD handle this issue better?

caf 7y ago

If you've foolishly decided to run without swap (like the original post), then suspending the offending program does nothing. This is because the offending program has allocated a lot of private dirty pages, which can't be dropped from memory because without swap space, there is nowhere for it to go.