6 ms·

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most program

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges that would protect them from this. It would be preferable for the system to die, rather than permit this debugging nonsense, considering it would perhaps actually be fixed if the machine died in this case. This is an asinine failure case mired in 1970s malpractice. Don't you agree this is damning and unacceptable? No one has any right to be proud of this mess. There's millions of lines of code and yet basic failure cases aren't truly accounted for or are handled in the most asinine of ways, such as with the ``Out of Memory Killer''.

theamk 7y ago
Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glaring design omission.
- Avamander 7y ago
  The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
  - lulouie 7y ago
    Yep, when Chromium ate up all the memory, it just hang the whole OS, waiting for OOM killer about 10+ min, then the cursor can be moved again, then freeze again...
    - olliej 7y ago
      If you want hilarious fun: make a gl shader than takes ~30 seconds to run. GPUs are only very recently preemptable (if they are at all yet? I lose track of what is “planned” vs released). Make it run that shader in a loop. See how well your system appears to respond. IIRC macOS has a 60s or something watchdog the hard resets the GPU, while the gpu is hung the screen is not updated. Everything is running fine, cpu isn’t pinned or anything, but the gpu is blocked so no compositing, and so no screen updating. I’m not sure what Linux does in that case, and I think windows may be able to paint because the directx driver interfaces let it do ... something? I’ve always assume some way to dma straight to the framebuffer, but no real idea :)
  - viraptor 7y ago
    Sounds like you're starting to swap heavily. Adjusting swappiness to 0 may help there.
    - Avamander 7y ago
      This also happens without swap.
  - AnIdiotOnTheNet 7y ago
    The OOM killer is a terrible hack that no self respecting system should have ever employed.
    - olliej 7y ago
      The OOM killer is a “solution” to a very real, and sensible design choice: not committing physical memory and swap whenever address space is mapped - there are very good (and noticeable) reasons for not eagerly committing, but fundamentally if you have done so you have to decide what to do when you end up needing more physical space than is available. Linux went down the “if a process is trying to do this, it must be important so I’ll prioritise it and kill something else”, and alternative is to kill that process when the commit fails. Either is a valid option, the OOM killer ran against a regular desktop user’s idea of what is the correct course of action, but for a server it might not have been.
      - pjmlp 7y ago
        Problem is that something else might be actually performing a critical task.
        olliej 7y ago
        The assumption being made is that the app going mad for memory is the critical one. Something has to die, and deciding which is hard.
  - segfaultbuserr 7y ago
    Pro-Tip: You can use Alt + SysRq + F to trigger the OOM-killer action immediately. It helps me avoiding pulling the plug of my desktops on multiple occasions over the years when I accidentally start RAM-eating programs. Just make sure SysRq is enabled in sysctl.
    - emmelaich 7y ago
      When the machine is busy -- and even when it's not, it's very uncertain thing to try sys-rq combinations.
      - segfaultbuserr 7y ago
        Yes, it's risky. But when the machine is swapping to death and pulling the plug is the only option, SysRq is a better alternative.
  - eikenberry 7y ago
    That is probably being cause by your use of swap space, not an OOM issue. I've had multiple cases of the OOM killer kicking off on my system, all without it slowing way down.
    - rcxdude 7y ago
      Lacking swap space causes more severe symptoms in an OOM situation, not less, from my experience. I think this is because everything that can get evicted from RAM is before the OOM killer gets invoked, which means every disk access slows to a crawl.
    - Avamander 7y ago
      No. This happens even without swap, OOM killer might as well not exist.
  - colechristensen 7y ago
    In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years. (spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM app could have a 12 GB heap on a machine with 12 GB of memory)
    - mshook 7y ago
      That's pretty much why we use vm.panic_on_oom = 1 as you have no idea about what's going to be killed...
      - colechristensen 7y ago
        I am in the camp of having systemd or your service manager of choice restart any killed service or service failing health check and emitting an ERROR or higher log message. I want my systems to heal themselves. More often than not these memory problems end up being slow leaks which can be effectively permanently resolved with periodic restarts, and the engineering time to fix them is appropriately not prioritized. I want to know that there has been a problem but I would rather not be forced to do anything about it unless absolutely necessary.
  - pas 7y ago
    https://lwn.net/Articles/759781/ https://lwn.net/Articles/759781/ already merged in 4.20. https://github.com/facebookincubator/oomd/blob/master/README.md https://github.com/facebookincubator/oomd/blob/master/README... and here's a less complicated, but similar proactive daemon: https://github.com/rfjakob/earlyoom/blob/master/README.md https://github.com/rfjakob/earlyoom/blob/master/README.md
    - Avamander 7y ago
      I'm on 5.0.0 and I legit haven't noticed a difference. If I run out of RAM without swap the system freezes and if I have swap then the system freezes when both are full. The only reliable solution is having a RAM+swap usage graph on my screen at all times and then closing stuff manually.
      - pas 7y ago
        You probably won't until the distro you use (or you manually) set up something other than the default OOM killer. You probably have too much swap. More than ~10 sec * your I/O speed (so let's say 512M-1G) is probably the max for the reasons you mentioned.
        Avamander 7y ago
        The system freezes also without any swap enabled but much more suddenly (there's no slowdown before dying). It's really just that the OOM killer triggers way way way too late.
  - Symbiote 7y ago
    It triggered on one of my systems yesterday, and killed the runaway process. When we were running tests of a new distributed system on our development (slightly underspecced) cluster, it would kill the distributed system processes when they took too much RAM. As other write, having slow or "too much" swap can delay the OOM killer from running in reasonable time.
- deleted 7y ago
  [deleted]
chmod775 7y ago
I fill my tiny 250GB SSD on my thinkpad all the time (well it's mostly logs of a certain terrible database topping it up now and then). Everything including my usual desktop environment keeps working. I have /home on a separate partition though. Most of the time I only notice that it filled up again when pacman -Syu fails because it can't save anything. It probably really depends on your setup and how stuff is configured. It appears the various systemd tools keep working just fine though, otherwise my system wouldn't even boot. If I could make a wish, I'd want my system to just buffer writes (transparently) to RAM once the SSD filled up. Then fire some events that can be intercepted by a GUI, so the user could be informed that he'll lose data if he shuts the system down now. Then as a last resort, abort any shutdown the user initiated and drop him into a shell so he can fix the mess and allow the system to save his data.
- dsfyu404ed 7y ago
  Same here only I don't have /home on a separate partition. I don't usually notice it's full until I go to download something and it fails.
simonjgreen 7y ago
We have a legacy platform running on VMWare with an NFS SAN. If/when connectivity is lost between the cluster and the SAN we get to experience resolving disk issues across several hundred VMs. Yanking access to the underlying storage device creates the most frustrating array of issues in Linux, depending really on the workload on the server. By far the most common is the mount becomes read only, forcing a reboot, which halts in order to fsck. Windows on the other hand just resumes from where it left off, or sometimes reboots but is usually clean. This platform is currently being retrofitted with vsan.

theamk 7y ago

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glaring design omission.

Avamander 7y ago
The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
- lulouie 7y ago
  Yep, when Chromium ate up all the memory, it just hang the whole OS, waiting for OOM killer about 10+ min, then the cursor can be moved again, then freeze again...
  - olliej 7y ago
    If you want hilarious fun: make a gl shader than takes ~30 seconds to run. GPUs are only very recently preemptable (if they are at all yet? I lose track of what is “planned” vs released). Make it run that shader in a loop. See how well your system appears to respond. IIRC macOS has a 60s or something watchdog the hard resets the GPU, while the gpu is hung the screen is not updated. Everything is running fine, cpu isn’t pinned or anything, but the gpu is blocked so no compositing, and so no screen updating. I’m not sure what Linux does in that case, and I think windows may be able to paint because the directx driver interfaces let it do ... something? I’ve always assume some way to dma straight to the framebuffer, but no real idea :)
- viraptor 7y ago
  Sounds like you're starting to swap heavily. Adjusting swappiness to 0 may help there.
  - Avamander 7y ago
    This also happens without swap.
- AnIdiotOnTheNet 7y ago
  The OOM killer is a terrible hack that no self respecting system should have ever employed.
  - olliej 7y ago
    The OOM killer is a “solution” to a very real, and sensible design choice: not committing physical memory and swap whenever address space is mapped - there are very good (and noticeable) reasons for not eagerly committing, but fundamentally if you have done so you have to decide what to do when you end up needing more physical space than is available. Linux went down the “if a process is trying to do this, it must be important so I’ll prioritise it and kill something else”, and alternative is to kill that process when the commit fails. Either is a valid option, the OOM killer ran against a regular desktop user’s idea of what is the correct course of action, but for a server it might not have been.
    - pjmlp 7y ago
      Problem is that something else might be actually performing a critical task.
      - olliej 7y ago
        The assumption being made is that the app going mad for memory is the critical one. Something has to die, and deciding which is hard.
- segfaultbuserr 7y ago
  Pro-Tip: You can use Alt + SysRq + F to trigger the OOM-killer action immediately. It helps me avoiding pulling the plug of my desktops on multiple occasions over the years when I accidentally start RAM-eating programs. Just make sure SysRq is enabled in sysctl.
  - emmelaich 7y ago
    When the machine is busy -- and even when it's not, it's very uncertain thing to try sys-rq combinations.
    - segfaultbuserr 7y ago
      Yes, it's risky. But when the machine is swapping to death and pulling the plug is the only option, SysRq is a better alternative.
- eikenberry 7y ago
  That is probably being cause by your use of swap space, not an OOM issue. I've had multiple cases of the OOM killer kicking off on my system, all without it slowing way down.
  - rcxdude 7y ago
    Lacking swap space causes more severe symptoms in an OOM situation, not less, from my experience. I think this is because everything that can get evicted from RAM is before the OOM killer gets invoked, which means every disk access slows to a crawl.
  - Avamander 7y ago
    No. This happens even without swap, OOM killer might as well not exist.
- colechristensen 7y ago
  In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years. (spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM app could have a 12 GB heap on a machine with 12 GB of memory)
  - mshook 7y ago
    That's pretty much why we use vm.panic_on_oom = 1 as you have no idea about what's going to be killed...
    - colechristensen 7y ago
      I am in the camp of having systemd or your service manager of choice restart any killed service or service failing health check and emitting an ERROR or higher log message. I want my systems to heal themselves. More often than not these memory problems end up being slow leaks which can be effectively permanently resolved with periodic restarts, and the engineering time to fix them is appropriately not prioritized. I want to know that there has been a problem but I would rather not be forced to do anything about it unless absolutely necessary.
- pas 7y ago
  https://lwn.net/Articles/759781/ https://lwn.net/Articles/759781/ already merged in 4.20. https://github.com/facebookincubator/oomd/blob/master/README.md https://github.com/facebookincubator/oomd/blob/master/README... and here's a less complicated, but similar proactive daemon: https://github.com/rfjakob/earlyoom/blob/master/README.md https://github.com/rfjakob/earlyoom/blob/master/README.md
  - Avamander 7y ago
    I'm on 5.0.0 and I legit haven't noticed a difference. If I run out of RAM without swap the system freezes and if I have swap then the system freezes when both are full. The only reliable solution is having a RAM+swap usage graph on my screen at all times and then closing stuff manually.
    - pas 7y ago
      You probably won't until the distro you use (or you manually) set up something other than the default OOM killer. You probably have too much swap. More than ~10 sec * your I/O speed (so let's say 512M-1G) is probably the max for the reasons you mentioned.
      - Avamander 7y ago
        The system freezes also without any swap enabled but much more suddenly (there's no slowdown before dying). It's really just that the OOM killer triggers way way way too late.
- Symbiote 7y ago
  It triggered on one of my systems yesterday, and killed the runaway process. When we were running tests of a new distributed system on our development (slightly underspecced) cluster, it would kill the distributed system processes when they took too much RAM. As other write, having slow or "too much" swap can delay the OOM killer from running in reasonable time.
deleted 7y ago
[deleted]

chmod775 7y ago

I fill my tiny 250GB SSD on my thinkpad all the time (well it's mostly logs of a certain terrible database topping it up now and then). Everything including my usual desktop environment keeps working. I have /home on a separate partition though. Most of the time I only notice that it filled up again when pacman -Syu fails because it can't save anything. It probably really depends on your setup and how stuff is configured. It appears the various systemd tools keep working just fine though, otherwise my system wouldn't even boot. If I could make a wish, I'd want my system to just buffer writes (transparently) to RAM once the SSD filled up. Then fire some events that can be intercepted by a GUI, so the user could be informed that he'll lose data if he shuts the system down now. Then as a last resort, abort any shutdown the user initiated and drop him into a shell so he can fix the mess and allow the system to save his data.

dsfyu404ed 7y ago
Same here only I don't have /home on a separate partition. I don't usually notice it's full until I go to download something and it fails.

simonjgreen 7y ago

We have a legacy platform running on VMWare with an NFS SAN. If/when connectivity is lost between the cluster and the SAN we get to experience resolving disk issues across several hundred VMs. Yanking access to the underlying storage device creates the most frustrating array of issues in Linux, depending really on the workload on the server. By far the most common is the mount becomes read only, forcing a reboot, which halts in order to fsck. Windows on the other hand just resumes from where it left off, or sometimes reboots but is usually clean. This platform is currently being retrofitted with vsan.