6 ms·

> Warning signs of a genuine low memory situation that you may want to look into: > available memory (or "free + buffers/cache") is close to zero > swap used

> Warning signs of a genuine low memory situation that you may want to look into: > available memory (or "free + buffers/cache") is close to zero > swap used increases or fluctuates > dmesg | grep oom-killer shows the OutOfMemory-killer at work How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox. The workaround was installing earlyoom [1] (after which the tab would crash instead), although I assume there's some kind of sysfs parameter that I might have tweaked instead. [1] https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom

justsomehnguy 5y ago
How about when you can't ssh to the server because ... oom-killer killed sshd? Thankfully that system wasn't mine so I just got a laugh of this situation. Still, 20 years later I still can't wrap my head around why Linux still needs oom-killer when Windows can run just fine without such functionality.
- oshiar53-0 5y ago
  Hint: disable or minimize overcommit and you're good to go.
  - justsomehnguy 5y ago
    That works for the systems I control. Most of the time I encounter OOM shenanigans on the systems which aren't mine.
  - rrss 5y ago
    I dearly wish overcommit could be disabled per-cgroup, but AFAIK this is not possible. malloc failing seems like a much simpler indication of memory pressure than the alternatives like PSI.
    - zozbot234 5y ago
      AIUI, you can always use plain old ulimit to restrict virtual memory usage per-process, which has the same outcome as disabling overcommit.
      - rrss 5y ago
        yeah, for a single process. but I’m mostly interested in doing this for a process tree (like a browser), and I don’t think this would work well in that case, since the limit would apply to each process individually. this would work if there was a cgroup virtual address space controller, but I think those have been proposed but never merged.
  - kasabali 5y ago
    In my short experience you can't disable overcommit if you're using Linux on desktop because programs using gtk/glib freak out when they can't malloc.
- cozzyd 5y ago
  Here is where something like systemd is really nice because sshd will get restarted! As for Windows, the big difference is no fork()
  - zinekeller 5y ago
    > As for Windows, the big difference is no fork() Missing the whole picture. fork() isn't the real reason, it's that Windows guarantees memory or no memory while Linux by default overcommits (both of which isn't necessarily a bad thing - it's just different ways of managing a limited resource). NT could implement (and does, it's just not exposed in Win32 API) a non-overcommiting fork() and its memory algorithms will be the same afterwards.
    - ayende 5y ago
      The problem with `fork()` is that its usage requires over commit. Consider the scenario of a process with 51% of memory usage, what happens when it forks? The system pretty much _has_ to allow overcommit at this point, otherwise fork() will fail. Now, you can change the accounting to say that when the process modify a memory page (thus creating a copy), we'll check that there is enough memory _then_. That would move the "danger" of memory being yanked from underneath you. But that means that a process that calls fork() may then run into a problem some indeterministic time later. That is kind of a killer.
      - zinekeller 5y ago
        ERRORS The fork() function will fail if: [EAGAIN] The system lacked the necessary resources to create another process, or the system-imposed limit on the total number of processes under execution system-wide or by a single user {CHILD_MAX} would be exceeded. The fork() function may fail if: [ENOMEM] Insufficient storage space is available. The POSIX specification is clear on this: you can hard fail. Also, as an aside malloc() can also fail and the program must be able to handle that. Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail. (Also, most rely on fork-exec but there's posix_spawn() for de novo execution). Again, NT does indeed have a fork function, however the Windows API doesn't expose that (and this is the hack Cygwin used when forking), but it's designed to fail instead of overcommit when all memory is exhausted.
- rcxdude 5y ago
  Windows for me frequenctly needs a pagefile 8x the size of my memory in order to service the committed memory, even though it's never touched in practice.
- kasabali 5y ago
  > why Linux still needs oom-killer when Windows can run just fine without such functionality My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled. I'd much prefer the OOM killer.
  - archerx 5y ago
    Chrome just crashes, it seems like the fault may lie with Firefox.
  - gruez 5y ago
    >My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled. I never had that happen. What actually happens is firefox just crashes, or the screen goes blank for a few seconds while the graphics driver resets.
    - kasabali 5y ago
      I get all of those, too. BSOD occurs less often when the system was already pressured with several cycles of the above. But even the graphics driver crashing is already bad enough. If there's not enough memory, offending programs should be crashing, not drivers. (and yes, ram sticks are good and verifed with memtest86 and testmem5)
  - ferongr 5y ago
    Running with the page file disabled is bad. The system runs out of address space, hence the crashes.
  - justsomehnguy 5y ago
    > if the page file is disabled Ouch. Never run Windows without the swap. It can be as small as the system allows (AFAIR 16Mb for Win7), but it should be enabled. While this is the obvious footgun the problem is nobody ever tests with the swap disabled. I've seen very weird behaviour without the swap, the most funny one was when some 3D game (sadly don't remember the name) internal timer went completely nuts without the swap and game was working just fine with the swap enabled.
- weare138 5y ago
  Windows uses a dynamic page file for swapping so it will just keep increasing the page file size until it hits the limits of the OS or config settings. You can do the same thing in OSs like Linux and BSD but most distros still default to fixed swap partitions. For example macOS uses page/swap files instead of a swap partition by default. It's just a design choice. Both have advantages and disadvantages. Linux and BSD distros usually are more aggressive about kicking out applications in favor of keeping the system stable but the trade off with OSs like Windows is misbehaving applications can more easily take down the OS.
remram 5y ago
Do you have swap?
- 0xcde4c3db 5y ago
  It's been a while, but I think it had 2 GB of swap on a 16 GB SSD (this was one of the Linux-friendly Chromebook models, hence the small storage). In any case, reports seem to vary considerably on how this behavior varies with swap size or presence in practice.
rrss 5y ago
I think this happens when the kernel starts swapping executable images / shared libraries in and out and keeps going for a long time because according to the kernel metrics it is making great progress, even though user space is completely stalled because running code is paged out and has to be paged back in to do anything. earlyoom kills stuff sooner instead of trying this for a long time. I’ve also heard that some people have had some success by locking base system libraries and executables in memory. (Please let me know if this explanation is not correct or there are other solutions worth trying)
- jleahy 5y ago
  I normally run Linux without any swap partition and this still happens without swap. Realistically is this kind of oom condition (where oom killer fails to trigger) I just end up pulling the plug. I’ve never been able to track down the exact cause.
  - rcxdude 5y ago
    Code can almost always be paged out, even without a swap file, because it's memory mapped from files on disk. In fact running without swap makes this behaviour worse (i.e. it happens sooner and more suddenly) because then the kernel can't prioritise keeping currently running code in memory over keeping rarely-accessed data in memory. In general you should have some swap and run with earlyoom if you want decent behaviour in a low-memory condition on linux.
    - jleahy 5y ago
      Good point.
2Gkashmiri 5y ago
I've been using auto tab discard on Firefox. Its a small annoyance for users but if you are like me and have more than a few tabs open, this saves a lot of ram anxiety
Nextgrid 5y ago
The reason this happens is because due to memory pressure, the IO cache is emptied and thus every IO request (even if reading just one byte) is passed through to the actual storage device. This quickly fills up the device’s queue and IO performance can no longer keep up with the system’s demands even for normally-benign stuff such as writing logs.
Nasreddin_Hodja 5y ago
> How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox. It looks like 12309 https://web.archive.org/web/20140816132654/https://bugzilla.kernel.org/show_bug.cgi?id=12309 https://web.archive.org/web/20140816132654/https://bugzilla....

justsomehnguy 5y ago

How about when you can't ssh to the server because ... oom-killer killed sshd? Thankfully that system wasn't mine so I just got a laugh of this situation. Still, 20 years later I still can't wrap my head around why Linux still needs oom-killer when Windows can run just fine without such functionality.

oshiar53-0 5y ago
Hint: disable or minimize overcommit and you're good to go.
- justsomehnguy 5y ago
  That works for the systems I control. Most of the time I encounter OOM shenanigans on the systems which aren't mine.
- rrss 5y ago
  I dearly wish overcommit could be disabled per-cgroup, but AFAIK this is not possible. malloc failing seems like a much simpler indication of memory pressure than the alternatives like PSI.
  - zozbot234 5y ago
    AIUI, you can always use plain old ulimit to restrict virtual memory usage per-process, which has the same outcome as disabling overcommit.
    - rrss 5y ago
      yeah, for a single process. but I’m mostly interested in doing this for a process tree (like a browser), and I don’t think this would work well in that case, since the limit would apply to each process individually. this would work if there was a cgroup virtual address space controller, but I think those have been proposed but never merged.
- kasabali 5y ago
  In my short experience you can't disable overcommit if you're using Linux on desktop because programs using gtk/glib freak out when they can't malloc.
cozzyd 5y ago
Here is where something like systemd is really nice because sshd will get restarted! As for Windows, the big difference is no fork()
- zinekeller 5y ago
  > As for Windows, the big difference is no fork() Missing the whole picture. fork() isn't the real reason, it's that Windows guarantees memory or no memory while Linux by default overcommits (both of which isn't necessarily a bad thing - it's just different ways of managing a limited resource). NT could implement (and does, it's just not exposed in Win32 API) a non-overcommiting fork() and its memory algorithms will be the same afterwards.
  - ayende 5y ago
    The problem with `fork()` is that its usage requires over commit. Consider the scenario of a process with 51% of memory usage, what happens when it forks? The system pretty much _has_ to allow overcommit at this point, otherwise fork() will fail. Now, you can change the accounting to say that when the process modify a memory page (thus creating a copy), we'll check that there is enough memory _then_. That would move the "danger" of memory being yanked from underneath you. But that means that a process that calls fork() may then run into a problem some indeterministic time later. That is kind of a killer.
    - zinekeller 5y ago
      ERRORS The fork() function will fail if: [EAGAIN] The system lacked the necessary resources to create another process, or the system-imposed limit on the total number of processes under execution system-wide or by a single user {CHILD_MAX} would be exceeded. The fork() function may fail if: [ENOMEM] Insufficient storage space is available. The POSIX specification is clear on this: you can hard fail. Also, as an aside malloc() can also fail and the program must be able to handle that. Unfortunately for some programmers (mainly coming from GC-aware languages), they just assume that it won't fail. (Also, most rely on fork-exec but there's posix_spawn() for de novo execution). Again, NT does indeed have a fork function, however the Windows API doesn't expose that (and this is the hack Cygwin used when forking), but it's designed to fail instead of overcommit when all memory is exhausted.
rcxdude 5y ago
Windows for me frequenctly needs a pagefile 8x the size of my memory in order to service the committed memory, even though it's never touched in practice.
kasabali 5y ago
> why Linux still needs oom-killer when Windows can run just fine without such functionality My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled. I'd much prefer the OOM killer.
- archerx 5y ago
  Chrome just crashes, it seems like the fault may lie with Firefox.
- gruez 5y ago
  >My Windows 10 installation just BSODs when Firefox fills up the RAM if the page file is disabled. I never had that happen. What actually happens is firefox just crashes, or the screen goes blank for a few seconds while the graphics driver resets.
  - kasabali 5y ago
    I get all of those, too. BSOD occurs less often when the system was already pressured with several cycles of the above. But even the graphics driver crashing is already bad enough. If there's not enough memory, offending programs should be crashing, not drivers. (and yes, ram sticks are good and verifed with memtest86 and testmem5)
- ferongr 5y ago
  Running with the page file disabled is bad. The system runs out of address space, hence the crashes.
- justsomehnguy 5y ago
  > if the page file is disabled Ouch. Never run Windows without the swap. It can be as small as the system allows (AFAIR 16Mb for Win7), but it should be enabled. While this is the obvious footgun the problem is nobody ever tests with the swap disabled. I've seen very weird behaviour without the swap, the most funny one was when some 3D game (sadly don't remember the name) internal timer went completely nuts without the swap and game was working just fine with the swap enabled.
weare138 5y ago
Windows uses a dynamic page file for swapping so it will just keep increasing the page file size until it hits the limits of the OS or config settings. You can do the same thing in OSs like Linux and BSD but most distros still default to fixed swap partitions. For example macOS uses page/swap files instead of a swap partition by default. It's just a design choice. Both have advantages and disadvantages. Linux and BSD distros usually are more aggressive about kicking out applications in favor of keeping the system stable but the trade off with OSs like Windows is misbehaving applications can more easily take down the OS.

remram 5y ago

Do you have swap?

0xcde4c3db 5y ago
It's been a while, but I think it had 2 GB of swap on a 16 GB SSD (this was one of the Linux-friendly Chromebook models, hence the small storage). In any case, reports seem to vary considerably on how this behavior varies with swap size or presence in practice.

rrss 5y ago

I think this happens when the kernel starts swapping executable images / shared libraries in and out and keeps going for a long time because according to the kernel metrics it is making great progress, even though user space is completely stalled because running code is paged out and has to be paged back in to do anything. earlyoom kills stuff sooner instead of trying this for a long time. I’ve also heard that some people have had some success by locking base system libraries and executables in memory. (Please let me know if this explanation is not correct or there are other solutions worth trying)

jleahy 5y ago
I normally run Linux without any swap partition and this still happens without swap. Realistically is this kind of oom condition (where oom killer fails to trigger) I just end up pulling the plug. I’ve never been able to track down the exact cause.
- rcxdude 5y ago
  Code can almost always be paged out, even without a swap file, because it's memory mapped from files on disk. In fact running without swap makes this behaviour worse (i.e. it happens sooner and more suddenly) because then the kernel can't prioritise keeping currently running code in memory over keeping rarely-accessed data in memory. In general you should have some swap and run with earlyoom if you want decent behaviour in a low-memory condition on linux.
  - jleahy 5y ago
    Good point.

2Gkashmiri 5y ago

I've been using auto tab discard on Firefox. Its a small annoyance for users but if you are like me and have more than a few tabs open, this saves a lot of ram anxiety

Nextgrid 5y ago

The reason this happens is because due to memory pressure, the IO cache is emptied and thus every IO request (even if reading just one byte) is passed through to the actual storage device. This quickly fills up the device’s queue and IO performance can no longer keep up with the system’s demands even for normally-benign stuff such as writing logs.

Nasreddin_Hodja 5y ago

> How about this one: the entire system becomes so unresponsive that even a VT switch takes multiple minutes to process, never mind actually logging in and running diagnostics. I don't know exactly why, but I used to get this pretty regularly on a 2 GB laptop if I did the wrong thing in Firefox. It looks like 12309 https://web.archive.org/web/20140816132654/https://bugzilla.kernel.org/show_bug.cgi?id=12309 https://web.archive.org/web/20140816132654/https://bugzilla....