4 ms·

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesystem-linux/ https://ma.ttias.be/change-rese

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesystem-linux/ https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

stonesweep 6y ago
I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).
- birdyrooster 6y ago
  Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
  - stonesweep 6y ago
    Fun trivia: you can actually set per-device settings in /etc/mke2fs.conf with all sorts of alternate defaults. https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html
    - anonymousiam 6y ago
      More fun trivia. The original(?) man page for tunefs had an "easter egg": https://unixhistory.livejournal.com/1808.html?nojs=1 https://unixhistory.livejournal.com/1808.html?nojs=1
  - tytso 6y ago
    If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense. Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performance, and then have to pay $$$ to switch to an SSD, maybe it would have been much cheaper to use a HDD with a larger free space reserve....
    - axiolite 6y ago
      > If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zero fragmentation. You can create plenty of 10MB files on a multi-terabyte volume that has under 1% free space, as that's still tens of gigabytes to work with. Obviously at some point you'll hit a severe problem and it's best to avoid taking a chance, but a fixed percentage really isn't the best measurement to tell you where that horizon will be.
      - ericbarrett 6y ago
        It's true, but it's more true for some file systems than others. When you write a file larger than the contiguous available space after its starting point, a file system must break the file into "extents" (chunks). The less space available, the smaller the extents tend to be, and the more fragmentation you will impose for continued writes. It's just math. Different file systems have wildly different strategies and data structures behind this process, however. Some drop to their knees over 92-93%. Some can write to the last byte with reasonable efficiency—but it'll never be as fast as when it was empty. Copy-on-write systems like ZFS tend to do poorly under near-full conditions.
        axiolite 6y ago
  - boring_twenties 6y ago
    It's reserved for the superuser, not the kernel
  - tryauuum 6y ago
    Had anyone succeed in looking up the motivation behind the reserved space idea in ext filesystems, e.g a commit message from the time this feature was introduced? I've tried but failed miserably, got lost in many different git repositories. One fun fact I learned though - one of the reasons is "for quota file". Since ext stores quota information in an ordinary file, there could be an issue where user fills up a disk but the information about it isn't written in quota file due to lack of space.
sonicrocketman 6y ago
I did not know this. Good solution.
GekkePrutser 6y ago
I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution. The way I found out about it originally was because I was using external storage drives and I was never able to fit as much as I expected :D Luckily you can easily change this without reformatting.
- chousuke 6y ago
  What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd. Nothing that actually does the "serving" or accesses data should be running as root.
  - GekkePrutser 6y ago
    No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.
    - edoceo 6y ago
      Mine don't run as root.
      - vntok 6y ago
        You should fix it then, takes no time at all.
        repiret 6y ago
        Thats not really something that needs fixing.
    - derefr 6y ago
throw0101a 6y ago
ZFS has explicit reservations: > The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations. * https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html These are done on a per dataset basis (basically a directory delineated boundary).
- rincebrain 6y ago
  ZFS, like ext[34], also has a reserved space allocation for the entire pool to allow you to still do certain operations when it's reporting "0" free space (...like deleting files or snapshots, to free space). Unlike ext[34], that reservation is not available for root's general use, but it's there.

stonesweep 6y ago

I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).

birdyrooster 6y ago
Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
- stonesweep 6y ago
  Fun trivia: you can actually set per-device settings in /etc/mke2fs.conf with all sorts of alternate defaults. https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html
  - anonymousiam 6y ago
    More fun trivia. The original(?) man page for tunefs had an "easter egg": https://unixhistory.livejournal.com/1808.html?nojs=1 https://unixhistory.livejournal.com/1808.html?nojs=1
- tytso 6y ago
  If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense. Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performance, and then have to pay $$$ to switch to an SSD, maybe it would have been much cheaper to use a HDD with a larger free space reserve....
  - axiolite 6y ago
    > If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zero fragmentation. You can create plenty of 10MB files on a multi-terabyte volume that has under 1% free space, as that's still tens of gigabytes to work with. Obviously at some point you'll hit a severe problem and it's best to avoid taking a chance, but a fixed percentage really isn't the best measurement to tell you where that horizon will be.
    - ericbarrett 6y ago
      It's true, but it's more true for some file systems than others. When you write a file larger than the contiguous available space after its starting point, a file system must break the file into "extents" (chunks). The less space available, the smaller the extents tend to be, and the more fragmentation you will impose for continued writes. It's just math. Different file systems have wildly different strategies and data structures behind this process, however. Some drop to their knees over 92-93%. Some can write to the last byte with reasonable efficiency—but it'll never be as fast as when it was empty. Copy-on-write systems like ZFS tend to do poorly under near-full conditions.
      - axiolite 6y ago
- boring_twenties 6y ago
  It's reserved for the superuser, not the kernel
- tryauuum 6y ago
  Had anyone succeed in looking up the motivation behind the reserved space idea in ext filesystems, e.g a commit message from the time this feature was introduced? I've tried but failed miserably, got lost in many different git repositories. One fun fact I learned though - one of the reasons is "for quota file". Since ext stores quota information in an ordinary file, there could be an issue where user fills up a disk but the information about it isn't written in quota file due to lack of space.

sonicrocketman 6y ago

I did not know this. Good solution.

GekkePrutser 6y ago

I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution. The way I found out about it originally was because I was using external storage drives and I was never able to fit as much as I expected :D Luckily you can easily change this without reformatting.

chousuke 6y ago
What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd. Nothing that actually does the "serving" or accesses data should be running as root.
- GekkePrutser 6y ago
  No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.
  - edoceo 6y ago
    Mine don't run as root.
    - vntok 6y ago
      You should fix it then, takes no time at all.
      - repiret 6y ago
        Thats not really something that needs fixing.
  - derefr 6y ago

throw0101a 6y ago

ZFS has explicit reservations: > The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations. * https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html These are done on a per dataset basis (basically a directory delineated boundary).

rincebrain 6y ago
ZFS, like ext[34], also has a reserved space allocation for the entire pool to allow you to still do certain operations when it's reporting "0" free space (...like deleting files or snapshots, to free space). Unlike ext[34], that reservation is not available for root's general use, but it's there.