8 ms·

I was very excited about btrfs' advanced features, but that meant that btrfs would bite me multiple times when I expected it to 'just work™': - RAID5/6 are sti

I was very excited about btrfs' advanced features, but that meant that btrfs would bite me multiple times when I expected it to 'just work™': - RAID5/6 are still not stable - it will not mount a RAID in degraded mode automatically, failing the high availability promise that might tempt you towards RAID. - swapfile support exists, but it breaks snapshots (and I don't want to snapshot the swapfile) - Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists)

ndsipa_pomu 3y ago
> Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists) Thanks - I did not know about that. I agree with the stance of not mounting a degraded RAID automatically as then the danger is that someone might not notice it and be subjected to total data loss later on. The best option would be to allow over-riding that choice if the RAID is otherwise monitored.
webstrand 3y ago
Can you not just place the swapfile inside of another subvolume? Since subvolumes are not included in snapshots. There's generally a bunch of stuff in /var that you don't want to include in snapshots, too, so it's not like putting the swapfile inside of a subvolume is an exotic task.
- pantalaimon 3y ago
  I guess so, I just didn't know when I created the swapfile and was surprised later when I tried to create a snapshot.
  - xorcist 3y ago
    Be careful, just like disk images, you probably don't want to place swapfiles on a CoW filesystem.
    - Bu9818 3y ago
      You can mark individual files as No_COW in btrfs, and No_COW + preallocation is a requirement for swapfiles anyway due to how the swap subsystem works.
rini17 3y ago
- RAID5/6 are still not stable Why does everyone insist on RAID5? I am completely fine with btrfs RAID1, survived few disk crashes as advertised. - it will not mount a RAID in degraded mode automatically it will if using -o degraded, it's the tooling which generally sucks and won't support this. I even had problems booting at all with / on multi-device btrfs using recommended tools (dracut, grub-mkconfig), the bugs are known and unfixed, ended up rolling out own initramfs. - swapfile support exists, but it breaks snapshots you are supposed to have all your stuff in a subvolume, and snapshot that, not whole toplevel root filesystem...and yes it should be documented better that it interferes with snapshots - but snapshots are not integrated.. yep tooling sucks
- dinosaurdynasty 3y ago
  RAID1 is a lot more expensive and/or stores a lot less data than RAID5/6.
  - rini17 3y ago
    How much is the "a lot"?
    - pantalaimon 3y ago
      You can try yourself: https://carfax.org.uk/btrfs-usage/ https://carfax.org.uk/btrfs-usage/ But generally, RAID1 gets you 50% of storage space, whereas RAID5 gets you 66% (and any odd combination of disks).
      - bmicraft 3y ago
        RAID5 should actually get you n-1 disks of space (if they're equally large), while raid1 only gets you n/2
        __david__ 3y ago
        Raid1 actually gets you less than that. If you mirror 5 drives that only gives you 1 drive worth of data (with 4 redundancies). But… you have to weigh it against losing all your data. With raid5 if any 2 of your disks die at the same time then you've lost your volume and most likely 100% of your data (no matter how many disks you have). With raid1 you have to lose _all_ your disks before that happens. Typically that's also just 2 but you can mirror 3 or more drives if you need some data to be _really_ resistant to disk failures and you don't care about "wasting" n-1 times the space. So you end up trading off efficiency of storage space and resiliency to data loss. Myself, disks got cheap enough that I always just buy 2 disks and mirror them. I find it easier to reason about overall, especially in the face of a degraded array.
keep_reading 3y ago
> it will not mount a RAID in degraded mode automatically, I remember when btrfs was very young and they announced the ability to create mirrors. I tested this out and was pleased, and then I tested the failure scenario: pull a drive, try to boot. It wouldn't boot! I jump into IRC and ask if it's expected that you can't boot from a degraded mirror and the answer was "not supported" which means the mirror is pointless. Obviously it has improved since then as there's a way to force it to work, but I returned to ZFS on FreeBSD and never looked back
- nwmcsween 3y ago
  Yeah this is the probably the worst default for a fs that supports raid.
  - wtallis 3y ago
    It's the safest default. If you want a filesystem that automatically downgrades the safety of your data when things go wrong, you should have to opt-in to such behavior (and btrfs does make that possible).
    - keep_reading 3y ago
      There's less risk of damage from running off a single disk in a faulted mirror than to rebuild the mirror. The stress the drive is under to rebuild can and will kill drives at the end of their life.
      - wtallis 3y ago
        If one half of a mirrored pair dies, you're left with one copy of the data on a drive that's at risk of also dying, plus whatever portion of the data you have backed up. If you then write new data to that drive, it isn't mirrored anywhere and is also guaranteed to not be backed up yet. So the newly written data is at equal or higher risk than the data that was already on the surviving drive. It's not hard to see how silently accepting new writes that cannot be mirrored could be an unacceptable risk to some end users. With btrfs, you can easily add a new drive so that new writes can be accepted and mirrored immediately, then start rebuilding the old data with lost redundancy (possibly after running another incremental backup, which will be less stressful to the drive than the full rebuild). But the "add a new drive" step happens outside the kernel, in userspace and possibly in meatspace, so it can't be a default action for the kernel to take.

ndsipa_pomu 3y ago

> Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists) Thanks - I did not know about that. I agree with the stance of not mounting a degraded RAID automatically as then the danger is that someone might not notice it and be subjected to total data loss later on. The best option would be to allow over-riding that choice if the RAID is otherwise monitored.

webstrand 3y ago

Can you not just place the swapfile inside of another subvolume? Since subvolumes are not included in snapshots. There's generally a bunch of stuff in /var that you don't want to include in snapshots, too, so it's not like putting the swapfile inside of a subvolume is an exotic task.

pantalaimon 3y ago
I guess so, I just didn't know when I created the swapfile and was surprised later when I tried to create a snapshot.
- xorcist 3y ago
  Be careful, just like disk images, you probably don't want to place swapfiles on a CoW filesystem.
  - Bu9818 3y ago
    You can mark individual files as No_COW in btrfs, and No_COW + preallocation is a requirement for swapfiles anyway due to how the swap subsystem works.

rini17 3y ago

- RAID5/6 are still not stable Why does everyone insist on RAID5? I am completely fine with btrfs RAID1, survived few disk crashes as advertised. - it will not mount a RAID in degraded mode automatically it will if using -o degraded, it's the tooling which generally sucks and won't support this. I even had problems booting at all with / on multi-device btrfs using recommended tools (dracut, grub-mkconfig), the bugs are known and unfixed, ended up rolling out own initramfs. - swapfile support exists, but it breaks snapshots you are supposed to have all your stuff in a subvolume, and snapshot that, not whole toplevel root filesystem...and yes it should be documented better that it interferes with snapshots - but snapshots are not integrated.. yep tooling sucks

dinosaurdynasty 3y ago
RAID1 is a lot more expensive and/or stores a lot less data than RAID5/6.
- rini17 3y ago
  How much is the "a lot"?
  - pantalaimon 3y ago
    You can try yourself: https://carfax.org.uk/btrfs-usage/ https://carfax.org.uk/btrfs-usage/ But generally, RAID1 gets you 50% of storage space, whereas RAID5 gets you 66% (and any odd combination of disks).
    - bmicraft 3y ago
      RAID5 should actually get you n-1 disks of space (if they're equally large), while raid1 only gets you n/2
      - __david__ 3y ago
        Raid1 actually gets you less than that. If you mirror 5 drives that only gives you 1 drive worth of data (with 4 redundancies). But… you have to weigh it against losing all your data. With raid5 if any 2 of your disks die at the same time then you've lost your volume and most likely 100% of your data (no matter how many disks you have). With raid1 you have to lose _all_ your disks before that happens. Typically that's also just 2 but you can mirror 3 or more drives if you need some data to be _really_ resistant to disk failures and you don't care about "wasting" n-1 times the space. So you end up trading off efficiency of storage space and resiliency to data loss. Myself, disks got cheap enough that I always just buy 2 disks and mirror them. I find it easier to reason about overall, especially in the face of a degraded array.

keep_reading 3y ago

> it will not mount a RAID in degraded mode automatically, I remember when btrfs was very young and they announced the ability to create mirrors. I tested this out and was pleased, and then I tested the failure scenario: pull a drive, try to boot. It wouldn't boot! I jump into IRC and ask if it's expected that you can't boot from a degraded mirror and the answer was "not supported" which means the mirror is pointless. Obviously it has improved since then as there's a way to force it to work, but I returned to ZFS on FreeBSD and never looked back

nwmcsween 3y ago
Yeah this is the probably the worst default for a fs that supports raid.
- wtallis 3y ago
  It's the safest default. If you want a filesystem that automatically downgrades the safety of your data when things go wrong, you should have to opt-in to such behavior (and btrfs does make that possible).
  - keep_reading 3y ago
    There's less risk of damage from running off a single disk in a faulted mirror than to rebuild the mirror. The stress the drive is under to rebuild can and will kill drives at the end of their life.
    - wtallis 3y ago
      If one half of a mirrored pair dies, you're left with one copy of the data on a drive that's at risk of also dying, plus whatever portion of the data you have backed up. If you then write new data to that drive, it isn't mirrored anywhere and is also guaranteed to not be backed up yet. So the newly written data is at equal or higher risk than the data that was already on the surviving drive. It's not hard to see how silently accepting new writes that cannot be mirrored could be an unacceptable risk to some end users. With btrfs, you can easily add a new drive so that new writes can be accepted and mirrored immediately, then start rebuilding the old data with lost redundancy (possibly after running another incremental backup, which will be less stressful to the drive than the full rebuild). But the "add a new drive" step happens outside the kernel, in userspace and possibly in meatspace, so it can't be a default action for the kernel to take.