4 ms·

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That mean

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of disk space, you actually have 5% left for root. So log in as root and fix the issue. Simple. Also: > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. Why would memory swap to disk when the disk is full? I feel like the author is conflating potential memory pressure issues with disk issues. How many serious production-grade servers even use swap, which usually just causes everything to grind to a halt if memory becomes full? [0] https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesystem-linux/ https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys...

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
cellularmitosis 6y ago
> So log in as root and fix the issue. Simple. "just fix the issue" often takes longer than 'rm spacer.img', which allows non-root processes to begin working immediately. In practice, the 5% root reserve is not really an effective substitute for OP's technique.
- herpderperator 6y ago
  Running `tune2fs -m 0 /dev/sdaX` as root instantly makes the reserved blocks available. You could even just lower it by 1% which would be more than sufficient in the meantime: `tune2fs -m 4 /dev/sdaX`. When you've fixed the issue and freed up space, you can increase it to 5% again. So why would you bother making a blank file? :)
  - cellularmitosis 6y ago
    because 6 months from now I won't have 'tune2fs -m 0 /dev/sdaX' memorized, especially in a panic'ed state. but I'll absolutely have 'rm spacer.img' memorized. (edit: also, if you truly set -m to zero, and you fill up again, now you can't even ssh in as root...)
rabuse 6y ago
I didn't know this. Thanks!
nottorp 6y ago
> So log in as root and fix the issue. Simple. Except you can't ssh in as root any more. You have to ssh as a normal user then sudo in. Which may not be possible. A paid on call admin may not mind driving to the data center too much. Everyone else would.
- herpderperator 6y ago
  sshd runs as root, so yes, it will be possible to login as any user -- the sshd daemon will be able to function. :-) The only files that would be appended to on user login (by sshd, which is running as root) would be /var/log/{utmp,wtmp,btmp} to record the login (in practice I've only seen wtmp). After that you have sshd logging (e.g. /var/log/{messages,syslog,auth.log}) which is picked up by syslog-ng or its equivalents, which also run as root. Regarding not being able to login via SSH as root by default: the sshd_config default is actually `PermitRootLogin prohibit-password` which means you can login if you use public-key cryptography, it just won't allow you to login with the root password (even if it's correct.) It's good practice these days to use public keys for SSH anyway, so I wouldn't say this is much of a setback.
  - nottorp 6y ago
    Logging in as the user loads the user's shell, shell config files etc. As that user. Only then you can go to being root... if you can. As for keys, they're fine if you only ever use the same machine. Which may not happen in an emergency.
- indigomm 6y ago
  You can login as root at the console on many providers. Even where not allowed (eg. AWS) you can mount the volume on another instance and clean it up from there.
Stevvo 6y ago
Sure, but one of the most common causes of disks being filled on Linux is either the kernel or some process running as root filling the disk with endless repeating log entries.
- herpderperator 6y ago
  It's a good point that the process that is flooding the disk could be running as root and therefore bypasses the reserved blocks restriction. It's also worth noting that the kernel is not a process and it does not write directly to files in the conventional sense. syslog-ng or its equivalents (which do run as root) will pick up messages from the kernel's circular ring buffer at /dev/kmsg and write them to a text file such as /var/log/kern.log, so it's possible that the kernel or one of its modules are verbose enough that syslog-ng causes that text file to get big. However, these log files can be limited to a certain size and/or logrotated[0] daily which will also remove log files older than a configured amount of time. In other words, there are better ways to manage servers than creating 8GB files as a bandaid. [0] https://linux.die.net/man/8/logrotate https://linux.die.net/man/8/logrotate
  - airhead969 6y ago
    Log files are awful, log rotation is even worse. Logs are a continuous, persistent stream of messages that sometimes gets expunged when too old. Log files try to flatten and serialize them, but the coordination and management of them is a hassle. An FS like Reiser where each log entry is a file arranged in directory trees by minute, hour, day, month, and year, or a sqlite database would make more sense than spewing blindly into a giant text file and praying that it works when another process comes along and moves things around underneath a running program.
- airhead969 6y ago
  This is why on *nix systems, each directory of the vfs that is intended to accumulate files OR have different mount options is broken-off into its own volume. / /home /usr /usr/local /var /var/log /var/lib/docker /var/lib/postgresql and so on.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

cellularmitosis 6y ago

> So log in as root and fix the issue. Simple. "just fix the issue" often takes longer than 'rm spacer.img', which allows non-root processes to begin working immediately. In practice, the 5% root reserve is not really an effective substitute for OP's technique.

herpderperator 6y ago
Running `tune2fs -m 0 /dev/sdaX` as root instantly makes the reserved blocks available. You could even just lower it by 1% which would be more than sufficient in the meantime: `tune2fs -m 4 /dev/sdaX`. When you've fixed the issue and freed up space, you can increase it to 5% again. So why would you bother making a blank file? :)
- cellularmitosis 6y ago
  because 6 months from now I won't have 'tune2fs -m 0 /dev/sdaX' memorized, especially in a panic'ed state. but I'll absolutely have 'rm spacer.img' memorized. (edit: also, if you truly set -m to zero, and you fill up again, now you can't even ssh in as root...)

rabuse 6y ago

I didn't know this. Thanks!

nottorp 6y ago

> So log in as root and fix the issue. Simple. Except you can't ssh in as root any more. You have to ssh as a normal user then sudo in. Which may not be possible. A paid on call admin may not mind driving to the data center too much. Everyone else would.

herpderperator 6y ago
sshd runs as root, so yes, it will be possible to login as any user -- the sshd daemon will be able to function. :-) The only files that would be appended to on user login (by sshd, which is running as root) would be /var/log/{utmp,wtmp,btmp} to record the login (in practice I've only seen wtmp). After that you have sshd logging (e.g. /var/log/{messages,syslog,auth.log}) which is picked up by syslog-ng or its equivalents, which also run as root. Regarding not being able to login via SSH as root by default: the sshd_config default is actually `PermitRootLogin prohibit-password` which means you can login if you use public-key cryptography, it just won't allow you to login with the root password (even if it's correct.) It's good practice these days to use public keys for SSH anyway, so I wouldn't say this is much of a setback.
- nottorp 6y ago
  Logging in as the user loads the user's shell, shell config files etc. As that user. Only then you can go to being root... if you can. As for keys, they're fine if you only ever use the same machine. Which may not happen in an emergency.
indigomm 6y ago
You can login as root at the console on many providers. Even where not allowed (eg. AWS) you can mount the volume on another instance and clean it up from there.

Stevvo 6y ago

Sure, but one of the most common causes of disks being filled on Linux is either the kernel or some process running as root filling the disk with endless repeating log entries.

herpderperator 6y ago
It's a good point that the process that is flooding the disk could be running as root and therefore bypasses the reserved blocks restriction. It's also worth noting that the kernel is not a process and it does not write directly to files in the conventional sense. syslog-ng or its equivalents (which do run as root) will pick up messages from the kernel's circular ring buffer at /dev/kmsg and write them to a text file such as /var/log/kern.log, so it's possible that the kernel or one of its modules are verbose enough that syslog-ng causes that text file to get big. However, these log files can be limited to a certain size and/or logrotated[0] daily which will also remove log files older than a configured amount of time. In other words, there are better ways to manage servers than creating 8GB files as a bandaid. [0] https://linux.die.net/man/8/logrotate https://linux.die.net/man/8/logrotate
- airhead969 6y ago
  Log files are awful, log rotation is even worse. Logs are a continuous, persistent stream of messages that sometimes gets expunged when too old. Log files try to flatten and serialize them, but the coordination and management of them is a hassle. An FS like Reiser where each log entry is a file arranged in directory trees by minute, hour, day, month, and year, or a sqlite database would make more sense than spewing blindly into a giant text file and praying that it works when another process comes along and moves things around underneath a running program.
airhead969 6y ago
This is why on *nix systems, each directory of the vfs that is intended to accumulate files OR have different mount options is broken-off into its own volume. / /home /usr /usr/local /var /var/log /var/lib/docker /var/lib/postgresql and so on.