8 ms·

Scripts for Btrfs Maintenance

89 points by pantalaimon 3y ago

EscapeFromNY 3y ago
Was I supposed to be maintaining my btrfs partition all this time? I just formatted my disk as btrfs when I bought my laptop and haven't thought about it since.
- sp332 3y ago
  Since you have checkums available, you might as well do a periodic "scrub" and see if any of your data has been corrupted. Trim is useful for SSDs because it can make your drive's firmware more effective at wear leveling. Balance is pretty workload-specific and I don't have a heuristic for when it might make a difference. But being super unbalanced might cause performance issues.
  - kevincox 3y ago
    IIUC the filesystem should balance itself. So unless you have added a new device to a mostly full filesystem you should be fine.
    - sp332 3y ago
      If you delete a bunch of files it might get unbalanced? That wouldn't degrade performance though I think. Or it is one of those FS's that makes sure everything is balanced on disk anyway?
      - kevincox 3y ago
        In theory yes. But it is unlikely. The files are very likely balanced before you delete them so it should be mostly balanced after. It will also self-correct the difference after a bit more use. Unbalanced drives can affect performance as one drive will have to do more of the work often bottlenecking. But except for extreme cases it is probably a rounding error.
      - foobarqux 3y ago
        I don't understand the details but I know one case it can fix problems is after deleting files to free up space on a nearly full disk.
        tremon 3y ago
        Btrfs has separate allocation pools for data and metadata. If you delete files, the freed-up space is returned to the data pool but that does not make it generally-available for use in the metadata pool. So in the case where the entire disk space is already allocated to one of the allocation pools, none of the pools can grow beyond their current size. One of the effects of balancing the tree is to release unused space from the allocation pools to general availability, solving that problem.
  - EscapeFromNY 3y ago
    Scrubbing sounds useful. I'll start doing that every so often. I thought trim was enabled by default (https://wiki.archlinux.org/title/Btrfs#SSD_TRIM https://wiki.archlinux.org/title/Btrfs#SSD_TRIM). Does fstrim do something more than that or is it the same thing?
    - sweettea 3y ago
      If there is no discard work to do, a fstrim on btrfs will do nothing. Discard=async, the new default, should be enough to trim deleted data without using fstrim. But nothing bad happens by using both.
- kevincox 3y ago
  Running an occasional scrub is probably a good idea. I don't think any of the other commands mentioned here should be required. Naturally the disk should be balanced as it is written so you shouldn't need explicit balances unless you added a new drive when they others were quite full.
  - Gigachad 3y ago
    If it was a good idea, I assume it should already be happening for me. What would be the point of mindlessly running some command periodically?
    - kevincox 3y ago
      Because different use cases have different requirements. So having the maintenance command be separate from the filesystem gives ultimate flexibility.
      - Gigachad 3y ago
        The OP comment didn't give any context for what you should be thinking to make a judgement call on. Just blindly "run this command periodically". If I'm not making any actual decision, it should just be run for me by default.
    - bravetraveler 3y ago
      Does your car wash itself? /s It very well may be! Depending on your distribution... they may already have similar services/timers for periodic scrubbing On Fedora they aren't provided; but are with Arch Unless you run an array it's fairly meaningless, beyond being an administrative-reporting tool. It can't fix failed checksums without mirrors or parity I think the checksums are validated on read anyway, so it's mostly to mitigate bitrot - very specific to 'glacial' storage
    - lmm 3y ago
      > If it was a good idea, I assume it should already be happening for me. It would, if you were running a well-designed/maintained operating system. Unfortunately many Linux distributions make poor decisions.
- bravetraveler 3y ago
  You're not benefiting from it as much as you could be; others have mentioned scrubbing That will read all of the data from the storage device(s) checking for coherency. In the case of redundancy (ie: RAID1) and corruption/rot, would use the other copy to make things whole Beyond that and trim, most of what's in here is fairly specific. ie: avoiding ENOSPC or dealing with array changes edit: I think running this more than ~monthly is overzealous... and only really meaningful if you have redundancy If memory serves, the checksums are validated when you read things anyway - so I question doing passes too aggressively. I'll accept a bit for bitrot
  - londons_explore 3y ago
    Thing is, the drive itself typically is in a much better position to protect against bitrot. Both SSD's and spinning metal drives have ECC bits stored with the data, and the drive can detect when the data was read easily or required lots of error correction applied. And then based on that they can make the optimal call of how often to read data to see it it is 'nearly rotted' and needs a rewrite. The filesystem has no knowledge of any of that, so has to do dumb periodic scans.
- xtracto 3y ago
  Just recently I had a worst-case nightmare scenario with a single BTRFS partition 5TB disk (no RAID or anything crazy). It was a single BTRFS disk, single partition external USB disk running under Linux Mint. At some point after I restarted the computer and it wouldn't mount the partition saying that the path for mounting was already being used (even after restarting??), but the disk was not mounted. Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1]. At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. Which I did... and it supposedly changed successfully. Except that after trying to mount the partition again, it will tell me that there was an error in the partition: the tree had mismatching UUIDs :-/. I tried updating the UUID several times, and the command ended in success, but the error remained. My BTRFS disk was officially broken... After spending several hours trying to fix the issue following the BTRFS documentation (pretty shitty TBH), in the end, I ended up having to do `btrfs restore -iv ...` to extract the data from the disk into some other external disk (formatted NTFS this time!!). The command is still going, about 2 weeks later and I am slowly recovering my 5TB of data. But the one thing clear to me is that I don't trust BTRFS or it's admin commands AT ALL after this experience. Once I finish recovering my data, I'll nuke the partition, format the disk in NTFS and forget about this sour experience. [1] https://unix.stackexchange.com/questions/603528/why-am-i-getting-this-file-exists-error-trying-to-mount-unmounted-filesystem https://unix.stackexchange.com/questions/603528/why-am-i-get...
  - bravetraveler 3y ago
    I can echo/support this claim - I wandered down the same rabbit hole. It doesn't even get much better with many drives and avoiding the docs/tooling BTRFS arrays will consistently corrupt with my reset button -- RAID10 on gen4 NVMe drives... while LVM/dm-raid + traditional file systems are absolutely fine
    - xtracto 3y ago
      >BTRFS arrays will consistently corrupt with my reset button YES!!! Sorry to beat a dead horse, but it has been so frustrating for me. I thought I did something wrong, like, how could the partition break for doing nothing! Where did I fucked up? I cannot imagine the experience with a RAIDed BTRFS array shudders
- heavyset_go 3y ago
  Do a scrub every once in a while. Mount with discard=async or turn on systemd's fstrim service if you're using SSDs. If it would actually affect performance, turn on autodefrag. Be aware that running a manual defrag, instead of using the mount option, will break reflinks.
  - quantumfissure 3y ago
    As of kernel 6.2, discard=async isn't needed anymore. It got a fix in 6.3[1] [1]:https://www.phoronix.com/news/Btrfs-Discard-Tuning-Linux-6.3 https://www.phoronix.com/news/Btrfs-Discard-Tuning-Linux-6.3
    - heavyset_go 3y ago
      Nice, thanks for the heads up.
chasil 3y ago
Why is nobody mentioning?: btrfs fi defrag -r / This is a recovery that ZFS cannot make. https://www.usenix.org/system/files/login/articles/login_summer17_02_conway.pdf https://www.usenix.org/system/files/login/articles/login_sum...
- nubinetwork 3y ago
  ZFS doesn't need it, if the file is accessed enough, it stays in cache. If the file isn't important enough, who cares that it takes a little bit longer to load into memory?
  - chasil 3y ago
    If a ZFS dataset goes over 80% utilization, I understand it will switch from "first fit" to "best fit" which can permanently and severely impact performance. https://serverfault.com/questions/733817/does-the-max-80-use-target-suggested-for-zfs-for-performance-reasons-apply-to-s https://serverfault.com/questions/733817/does-the-max-80-use...
    - LeoPanthera 3y ago
      I actually tested this, and could not measure any performance hit at all until I hit 95% full. It sped right back up again when I deleted some stuff. I wouldn't worry about it.
      - chasil 3y ago
        Interesting test. Thanks for finding this out.
    - seized 3y ago
      You can ZFS send back and forth between pools or to a new dataset to defrag/fix that, so it's not "permanent" as in can't ever be fixed. It also depends on workload, write once/read many is different from VM disk files.
seized 3y ago
I just scrub my ZFS pools once in a while and that's it... OpenIndiana autosnap is the rest of it which is built in. Then Monit to check pool health and alert me automatically.
DiabloD3 3y ago
There is no reason to ever use btrfs, IMO. btrfs is the result of Oracle trying to clone, badly, ZFS. When they bought Sun, they discontinued development on btrfs completely, as they (thought they) owned ZFS. Nobody sponsors btrfs development anymore, and its development has completely stagnated; while ZFS, under the OpenZFS project, continues to accelerate and absolutely dominates the enterprise mission critical file system sort of market. Just use the real deal: use ZFS. Due to the massive data loss issues with btrfs, for example, Redhat completely removed btrfs support in 8 entirely, after the preview feature in 7 bombed. They default to, and highly recommend, XFS (as do I, its a good file system).
- candiddevmike 3y ago
  Anecdotally, I've had so much trouble running Linux on a ZFS root. Failing DKMS builds or installing a kernel that is too new or old for the ZFS components has rendered my system unbootable enough that I switched to BTRFS.
  - DiabloD3 3y ago
    The only time I've heard this happening is on Ubuntu. Don't use Ubuntu, and switch to Debian or SUSE, and your problems go away. The only times I've ever heard of this not being Ubuntu pathology is people running builds of the kernel from git, and not keeping track of the extremely rare breaking changes that effect ZFS (OpenZFS's git repo for the kernel modules are updated quickly, and breaking changes are often handled ahead of time; all of this is fixed before stable kernels get released nowadays).
- akvadrako 3y ago
  A good reason to use Btrfs over ZFS is if, like me, you use Fedora Silverblue. It's supported by default while ZFS is essentially impossible to use – I use VMs for the occasional need. Another good reason is simplified tooling – ZFS has a lot of special ways of doing things different from all other filesystems. RedHat seemed to remove Btrfs because that want to move towards Stratis, not because ZFS or XFS is good enough. None of them are perfect yet and my best hope for the future is bcachefs.
Locrin 3y ago
I have happily used BTRFS in Raid1 on my Plex server for a good while now. I originally had a chucked WD 8TB drive. I started dreading to rebuild my library if the drive failed. So I got a 8TB Seagate disk and created a Raid-1 setup with one drive missing. I then copied over the data and added the old disk to the new raid. It took a good while to balance itself but it's been problemfree since. I might use ZFS if I switch to Ubuntu for home server, but I am using Fedora now, and want to stick to what is natively supported.
taacc3 3y ago
[dead]
eternityforest 3y ago
This does not exactly inspired confidence in BTRFS, if I need a script to maintain it. I only use it for one thing, read only compressed lower layers. It's great for that because it can be written to at setup time with standard tools, as opposed to some extra image compile step like true compressed read only FSes, but I wouldn't use it for anything that gets written in normal use.

89 points by pantalaimon 3y ago

Was I supposed to be maintaining my btrfs partition all this time? I just formatted my disk as btrfs when I bought my laptop and haven't thought about it since.

sp332 3y ago
Since you have checkums available, you might as well do a periodic "scrub" and see if any of your data has been corrupted. Trim is useful for SSDs because it can make your drive's firmware more effective at wear leveling. Balance is pretty workload-specific and I don't have a heuristic for when it might make a difference. But being super unbalanced might cause performance issues.
- kevincox 3y ago
  IIUC the filesystem should balance itself. So unless you have added a new device to a mostly full filesystem you should be fine.
  - sp332 3y ago
    If you delete a bunch of files it might get unbalanced? That wouldn't degrade performance though I think. Or it is one of those FS's that makes sure everything is balanced on disk anyway?
    - kevincox 3y ago
      In theory yes. But it is unlikely. The files are very likely balanced before you delete them so it should be mostly balanced after. It will also self-correct the difference after a bit more use. Unbalanced drives can affect performance as one drive will have to do more of the work often bottlenecking. But except for extreme cases it is probably a rounding error.
    - foobarqux 3y ago
      I don't understand the details but I know one case it can fix problems is after deleting files to free up space on a nearly full disk.
      - tremon 3y ago
        Btrfs has separate allocation pools for data and metadata. If you delete files, the freed-up space is returned to the data pool but that does not make it generally-available for use in the metadata pool. So in the case where the entire disk space is already allocated to one of the allocation pools, none of the pools can grow beyond their current size. One of the effects of balancing the tree is to release unused space from the allocation pools to general availability, solving that problem.
- EscapeFromNY 3y ago
  Scrubbing sounds useful. I'll start doing that every so often. I thought trim was enabled by default (https://wiki.archlinux.org/title/Btrfs#SSD_TRIM https://wiki.archlinux.org/title/Btrfs#SSD_TRIM). Does fstrim do something more than that or is it the same thing?
  - sweettea 3y ago
    If there is no discard work to do, a fstrim on btrfs will do nothing. Discard=async, the new default, should be enough to trim deleted data without using fstrim. But nothing bad happens by using both.
kevincox 3y ago
Running an occasional scrub is probably a good idea. I don't think any of the other commands mentioned here should be required. Naturally the disk should be balanced as it is written so you shouldn't need explicit balances unless you added a new drive when they others were quite full.
- Gigachad 3y ago
  If it was a good idea, I assume it should already be happening for me. What would be the point of mindlessly running some command periodically?
  - kevincox 3y ago
    Because different use cases have different requirements. So having the maintenance command be separate from the filesystem gives ultimate flexibility.
    - Gigachad 3y ago
      The OP comment didn't give any context for what you should be thinking to make a judgement call on. Just blindly "run this command periodically". If I'm not making any actual decision, it should just be run for me by default.
  - bravetraveler 3y ago
    Does your car wash itself? /s It very well may be! Depending on your distribution... they may already have similar services/timers for periodic scrubbing On Fedora they aren't provided; but are with Arch Unless you run an array it's fairly meaningless, beyond being an administrative-reporting tool. It can't fix failed checksums without mirrors or parity I think the checksums are validated on read anyway, so it's mostly to mitigate bitrot - very specific to 'glacial' storage
  - lmm 3y ago
    > If it was a good idea, I assume it should already be happening for me. It would, if you were running a well-designed/maintained operating system. Unfortunately many Linux distributions make poor decisions.
bravetraveler 3y ago
You're not benefiting from it as much as you could be; others have mentioned scrubbing That will read all of the data from the storage device(s) checking for coherency. In the case of redundancy (ie: RAID1) and corruption/rot, would use the other copy to make things whole Beyond that and trim, most of what's in here is fairly specific. ie: avoiding ENOSPC or dealing with array changes edit: I think running this more than ~monthly is overzealous... and only really meaningful if you have redundancy If memory serves, the checksums are validated when you read things anyway - so I question doing passes too aggressively. I'll accept a bit for bitrot
- londons_explore 3y ago
  Thing is, the drive itself typically is in a much better position to protect against bitrot. Both SSD's and spinning metal drives have ECC bits stored with the data, and the drive can detect when the data was read easily or required lots of error correction applied. And then based on that they can make the optimal call of how often to read data to see it it is 'nearly rotted' and needs a rewrite. The filesystem has no knowledge of any of that, so has to do dumb periodic scans.
xtracto 3y ago
Just recently I had a worst-case nightmare scenario with a single BTRFS partition 5TB disk (no RAID or anything crazy). It was a single BTRFS disk, single partition external USB disk running under Linux Mint. At some point after I restarted the computer and it wouldn't mount the partition saying that the path for mounting was already being used (even after restarting??), but the disk was not mounted. Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1]. At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. Which I did... and it supposedly changed successfully. Except that after trying to mount the partition again, it will tell me that there was an error in the partition: the tree had mismatching UUIDs :-/. I tried updating the UUID several times, and the command ended in success, but the error remained. My BTRFS disk was officially broken... After spending several hours trying to fix the issue following the BTRFS documentation (pretty shitty TBH), in the end, I ended up having to do `btrfs restore -iv ...` to extract the data from the disk into some other external disk (formatted NTFS this time!!). The command is still going, about 2 weeks later and I am slowly recovering my 5TB of data. But the one thing clear to me is that I don't trust BTRFS or it's admin commands AT ALL after this experience. Once I finish recovering my data, I'll nuke the partition, format the disk in NTFS and forget about this sour experience. [1] https://unix.stackexchange.com/questions/603528/why-am-i-getting-this-file-exists-error-trying-to-mount-unmounted-filesystem https://unix.stackexchange.com/questions/603528/why-am-i-get...
- bravetraveler 3y ago
  I can echo/support this claim - I wandered down the same rabbit hole. It doesn't even get much better with many drives and avoiding the docs/tooling BTRFS arrays will consistently corrupt with my reset button -- RAID10 on gen4 NVMe drives... while LVM/dm-raid + traditional file systems are absolutely fine
  - xtracto 3y ago
    >BTRFS arrays will consistently corrupt with my reset button YES!!! Sorry to beat a dead horse, but it has been so frustrating for me. I thought I did something wrong, like, how could the partition break for doing nothing! Where did I fucked up? I cannot imagine the experience with a RAIDed BTRFS array shudders
heavyset_go 3y ago
Do a scrub every once in a while. Mount with discard=async or turn on systemd's fstrim service if you're using SSDs. If it would actually affect performance, turn on autodefrag. Be aware that running a manual defrag, instead of using the mount option, will break reflinks.
- quantumfissure 3y ago
  As of kernel 6.2, discard=async isn't needed anymore. It got a fix in 6.3[1] [1]:https://www.phoronix.com/news/Btrfs-Discard-Tuning-Linux-6.3 https://www.phoronix.com/news/Btrfs-Discard-Tuning-Linux-6.3
  - heavyset_go 3y ago
    Nice, thanks for the heads up.

chasil 3y ago

Why is nobody mentioning?: btrfs fi defrag -r / This is a recovery that ZFS cannot make. https://www.usenix.org/system/files/login/articles/login_summer17_02_conway.pdf https://www.usenix.org/system/files/login/articles/login_sum...

nubinetwork 3y ago
ZFS doesn't need it, if the file is accessed enough, it stays in cache. If the file isn't important enough, who cares that it takes a little bit longer to load into memory?
- chasil 3y ago
  If a ZFS dataset goes over 80% utilization, I understand it will switch from "first fit" to "best fit" which can permanently and severely impact performance. https://serverfault.com/questions/733817/does-the-max-80-use-target-suggested-for-zfs-for-performance-reasons-apply-to-s https://serverfault.com/questions/733817/does-the-max-80-use...
  - LeoPanthera 3y ago
    I actually tested this, and could not measure any performance hit at all until I hit 95% full. It sped right back up again when I deleted some stuff. I wouldn't worry about it.
    - chasil 3y ago
      Interesting test. Thanks for finding this out.
  - seized 3y ago
    You can ZFS send back and forth between pools or to a new dataset to defrag/fix that, so it's not "permanent" as in can't ever be fixed. It also depends on workload, write once/read many is different from VM disk files.

seized 3y ago

I just scrub my ZFS pools once in a while and that's it... OpenIndiana autosnap is the rest of it which is built in. Then Monit to check pool health and alert me automatically.

DiabloD3 3y ago

There is no reason to ever use btrfs, IMO. btrfs is the result of Oracle trying to clone, badly, ZFS. When they bought Sun, they discontinued development on btrfs completely, as they (thought they) owned ZFS. Nobody sponsors btrfs development anymore, and its development has completely stagnated; while ZFS, under the OpenZFS project, continues to accelerate and absolutely dominates the enterprise mission critical file system sort of market. Just use the real deal: use ZFS. Due to the massive data loss issues with btrfs, for example, Redhat completely removed btrfs support in 8 entirely, after the preview feature in 7 bombed. They default to, and highly recommend, XFS (as do I, its a good file system).

candiddevmike 3y ago
Anecdotally, I've had so much trouble running Linux on a ZFS root. Failing DKMS builds or installing a kernel that is too new or old for the ZFS components has rendered my system unbootable enough that I switched to BTRFS.
- DiabloD3 3y ago
  The only time I've heard this happening is on Ubuntu. Don't use Ubuntu, and switch to Debian or SUSE, and your problems go away. The only times I've ever heard of this not being Ubuntu pathology is people running builds of the kernel from git, and not keeping track of the extremely rare breaking changes that effect ZFS (OpenZFS's git repo for the kernel modules are updated quickly, and breaking changes are often handled ahead of time; all of this is fixed before stable kernels get released nowadays).
akvadrako 3y ago
A good reason to use Btrfs over ZFS is if, like me, you use Fedora Silverblue. It's supported by default while ZFS is essentially impossible to use – I use VMs for the occasional need. Another good reason is simplified tooling – ZFS has a lot of special ways of doing things different from all other filesystems. RedHat seemed to remove Btrfs because that want to move towards Stratis, not because ZFS or XFS is good enough. None of them are perfect yet and my best hope for the future is bcachefs.

Locrin 3y ago

I have happily used BTRFS in Raid1 on my Plex server for a good while now. I originally had a chucked WD 8TB drive. I started dreading to rebuild my library if the drive failed. So I got a 8TB Seagate disk and created a Raid-1 setup with one drive missing. I then copied over the data and added the old disk to the new raid. It took a good while to balance itself but it's been problemfree since. I might use ZFS if I switch to Ubuntu for home server, but I am using Fedora now, and want to stick to what is natively supported.

taacc3 3y ago

[dead]

eternityforest 3y ago

This does not exactly inspired confidence in BTRFS, if I need a script to maintain it. I only use it for one thing, read only compressed lower layers. It's great for that because it can be written to at setup time with standard tools, as opposed to some extra image compile step like true compressed read only FSes, but I wouldn't use it for anything that gets written in normal use.