15 ms·

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as m

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man swapon(8): > The swap file implementation in the kernel expects to be able to write to the file directly, without the assistance of the filesystem. This is a problem on files with holes or on copy-on-write files on filesystems like Btrfs. > Commands like cp(1) or truncate(1) create files with holes. These files will be rejected by swapon. I just verified on Linux 5.8.0-48-generic (Ubuntu 20.10) / ext4 that trying to swapon a sparse file fails with "skipping - it appears to have holes". Now, swap is horribly slow, particularly on spinning rust rather than SSD. I run my systems without any swap for that reason. But swapping shouldn't fail on a full filesystem, unless you're trying to create & swapon a new swapfile after the filesystem is filled.

bostonsre 6y ago
Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.
- franga2000 6y ago
  I've experienced far more full disks than I'd want to admit, on many different hardware and software configurations, and I've never seen sudo break. Is this something you've experienced recently? I definitely agree with your advice and will go double check all my servers if /filler is 777 (not in /tmp since it's sometimes mounted tmpfs), but if sudo does break in that situation, that sounds like a pretty severe and most likely fixable bug.
  - bostonsre 6y ago
    Yea, have seen it recently on centos 7. I think it's due to having sudo logging enabled. It won't let you run sudo without it working (at least I think that's the case, but haven't spent the time to investigate too thoroughly).
- jeroenhd 6y ago
  I've never had sudo break on my full disks. However, that doesn't mean recovery is easy... Working in a terminal to find out what on earth has just filled up your disk is a real pain when your shell complains about failing to write to your $HISTFILE and such. And, of course, the problem always shows up on that one server that doesn't have ncdu installed... I'm sure sudo can theoretically break with 0 free disk space, but that's not the usual mode of failure in my experience. At most sudo need to touch a dotfile or two, so deleting _any_ temporary file or old log archive will do for it to recover. The balloon file is not a bad idea. I think I will apply it on my own servers just for good measure, although 8GiB is a bit much for my tastes.
- reph2097 6y ago
  In that case, use "su".
  - bostonsre 6y ago
    No luck when I've tried that. I think it's tied to sudo logging not working when the disk is full.
Yizahi 6y ago
I've seen Linux systems simply crash when root partition was 100% full, though it was an embedded system, not representative to a big servers.
- Blikkentrekker 6y ago
  Define crash of a "system"? Kernel panic? some user process you deem essential stopping?
  - Yizahi 6y ago
    One of essential services crashed and then whole unit rebooted automatically. I don't remember details now since it was a year ago, but that's approximately what has happened. Again, that's not a regular x86 fat server with linux, but embedded arm device running barebones debian plus some custom software, so it may behave differently.
MayeulC 6y ago
IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap. These days I always at least configure an in-memory compressed swap (zram).
- zamadatix 6y ago
  Depends if you want things to gracefully degrade because you know you don't have enough RAM or if you'd rather things just straight up die. E.g. for the things I work on my laptop with if whatever I do isn't going to work with 128 GB of RAM (80% of which was meant to be cached data not actually used) then it's because it went horribly wrong and needs to be halted not because I needed some swap which is just going to try to hide that things have gone horribly wrong for a minute and then die anyways. Now if I were doing the same things on a machine with 8 GB or 16 GB of RAM then yeah I want to gracefully handle running out of physical memory because things are probably working correctly it's just a heavier load and it can be better to swap pages to disk than drop them from a small amount of cache completely.
  - MayeulC 6y ago
    Personally, I prefer things that use a huge amount of memory to die instead of having the machine become unresponsive, especially if I have latency-sensitive things at the same time, like a conference call. Now, my main issue is that disabling swap doesn't cause this. With swap, OOM ends up being detected and corrected by the kernel. On some machines, it doesn't happen without swap, and often neither with swap. A sweet spot seems to be current-to-last CPU gen with 8 GB of memory. There's probably a lot of factors at play: what makes a task "hung"? How memory pressure is measured. Stuff probably changed with time, SSDs are now pretty quick, yet storage is a lot slower than RAM compared to what it used to be. Anyway, I use zram and have upgraded my memory, I'll also increase a bit my swap partition size in light of all of this. I'll probably have a look at userspace OOM daemons too.
- scottlamb 6y ago
  You recall incorrectly; swap is not needed. It's not just me who runs without it; Google production machines did for many years. "The behavior is often worse without swap" is more vague / subjective. I prefer that a process die cleanly than everything slow to a crawl semi-permanently. I've previously written about swap causing the latter: https://news.ycombinator.com/item?id=13715917 https://news.ycombinator.com/item?id=13715917 To some extent the bad behavior can happen even without swap because non-locked file-backed pages get paged in slowly also, but it seems significantly worse with swap. zram is a decent idea though. I use it on my memory-limited Raspberry Pi machines.

bostonsre 6y ago

Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.

franga2000 6y ago
I've experienced far more full disks than I'd want to admit, on many different hardware and software configurations, and I've never seen sudo break. Is this something you've experienced recently? I definitely agree with your advice and will go double check all my servers if /filler is 777 (not in /tmp since it's sometimes mounted tmpfs), but if sudo does break in that situation, that sounds like a pretty severe and most likely fixable bug.
- bostonsre 6y ago
  Yea, have seen it recently on centos 7. I think it's due to having sudo logging enabled. It won't let you run sudo without it working (at least I think that's the case, but haven't spent the time to investigate too thoroughly).
jeroenhd 6y ago
I've never had sudo break on my full disks. However, that doesn't mean recovery is easy... Working in a terminal to find out what on earth has just filled up your disk is a real pain when your shell complains about failing to write to your $HISTFILE and such. And, of course, the problem always shows up on that one server that doesn't have ncdu installed... I'm sure sudo can theoretically break with 0 free disk space, but that's not the usual mode of failure in my experience. At most sudo need to touch a dotfile or two, so deleting _any_ temporary file or old log archive will do for it to recover. The balloon file is not a bad idea. I think I will apply it on my own servers just for good measure, although 8GiB is a bit much for my tastes.
reph2097 6y ago
In that case, use "su".
- bostonsre 6y ago
  No luck when I've tried that. I think it's tied to sudo logging not working when the disk is full.

Yizahi 6y ago

I've seen Linux systems simply crash when root partition was 100% full, though it was an embedded system, not representative to a big servers.

Blikkentrekker 6y ago
Define crash of a "system"? Kernel panic? some user process you deem essential stopping?
- Yizahi 6y ago
  One of essential services crashed and then whole unit rebooted automatically. I don't remember details now since it was a year ago, but that's approximately what has happened. Again, that's not a regular x86 fat server with linux, but embedded arm device running barebones debian plus some custom software, so it may behave differently.

MayeulC 6y ago

IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap. These days I always at least configure an in-memory compressed swap (zram).

zamadatix 6y ago
Depends if you want things to gracefully degrade because you know you don't have enough RAM or if you'd rather things just straight up die. E.g. for the things I work on my laptop with if whatever I do isn't going to work with 128 GB of RAM (80% of which was meant to be cached data not actually used) then it's because it went horribly wrong and needs to be halted not because I needed some swap which is just going to try to hide that things have gone horribly wrong for a minute and then die anyways. Now if I were doing the same things on a machine with 8 GB or 16 GB of RAM then yeah I want to gracefully handle running out of physical memory because things are probably working correctly it's just a heavier load and it can be better to swap pages to disk than drop them from a small amount of cache completely.
- MayeulC 6y ago
  Personally, I prefer things that use a huge amount of memory to die instead of having the machine become unresponsive, especially if I have latency-sensitive things at the same time, like a conference call. Now, my main issue is that disabling swap doesn't cause this. With swap, OOM ends up being detected and corrected by the kernel. On some machines, it doesn't happen without swap, and often neither with swap. A sweet spot seems to be current-to-last CPU gen with 8 GB of memory. There's probably a lot of factors at play: what makes a task "hung"? How memory pressure is measured. Stuff probably changed with time, SSDs are now pretty quick, yet storage is a lot slower than RAM compared to what it used to be. Anyway, I use zram and have upgraded my memory, I'll also increase a bit my swap partition size in light of all of this. I'll probably have a look at userspace OOM daemons too.
scottlamb 6y ago
You recall incorrectly; swap is not needed. It's not just me who runs without it; Google production machines did for many years. "The behavior is often worse without swap" is more vague / subjective. I prefer that a process die cleanly than everything slow to a crawl semi-permanently. I've previously written about swap causing the latter: https://news.ycombinator.com/item?id=13715917 https://news.ycombinator.com/item?id=13715917 To some extent the bad behavior can happen even without swap because non-locked file-backed pages get paged in slowly also, but it seems significantly worse with swap. zram is a decent idea though. I use it on my memory-limited Raspberry Pi machines.