6 ms·

Maybe I'm just cynical but I think the ship has sailed for BTRFS RAID 5/6. It's now part of the global mindshare that BTRFS RAID 5/6 == data loss, no one wants

Maybe I'm just cynical but I think the ship has sailed for BTRFS RAID 5/6. It's now part of the global mindshare that BTRFS RAID 5/6 == data loss, no one wants to be the guinea pig that proves it works. Better to direct resources towards bcachefs or ZFS IMO.

viraptor 4y ago
Ideally we'd be working based on actual information rather than global mindshare. There will be guinea pigs to test it. There are already quite a few people running it despite the warnings. As much as I want to see a wide use of bcachefs, it's still years away. As someone who actually wants to store data - why would you direct resources to bcachefs which is known experimental, rather than btrfs which plainly documented raid5 as not ready and now may decide to change it to ready... if it is? (I'm donating to bcachefs patreon, but commenting from the PoV of someone choosing a solution today)
- gjs278 4y ago
- doublepg23 4y ago
  Is bcachefs years away? My understanding is the critical chain of code is pretty well tested - bcache.
  - viraptor 4y ago
    From widespread use, yes. Bcache was already stable at least 7 years ago. Bcachefs it's not even included in upstream kernel yet, much less any distro as an official option. There's only some plan to start submitting it soon™ and it will take months to complete.
- ilyt 4y ago
  Right but "actual information" here is "developers over and over fail to nail what ZFS and MDRAID did well decades ago". There is no reason to trust their next try will be "the one"
  - viraptor 4y ago
    Did well in what context? RAID5? There were enough people with motivation to spend time on it rather than other things for mdadm/zfs. It's not like btrfs devs wouldn't know how to implement it if you paid them to work full time on it.
  - _kdave 4y ago
    MDRAID had write hole until 4.4 (2015, https://lwn.net/Articles/665299/ https://lwn.net/Articles/665299/) and it had been long awaited back then. ZFS to my knowledge deals with that by variable stripe length that has its own problems but yeah it works. Btrfs changes implementation of the stripe update while preserving the on-disk format (i.e. can't do the same as ZFS without introducing incompatible change), intent log/bitmap have been proposed (which would be the MDRAID approach) but that's another incompatible change. So the 'next one' is at the cost of performance but with same compatibility.
denkmoon 4y ago
Do Meta not develop btrfs for their internal use? I don't think community sentiment is a big factor for them.
- Scaevolus 4y ago
  Meta and related companies have much better systems available for durability than RAID5. I would be moderately surprised if they commonly use any standard RAID level.
  - wtallis 4y ago
    Using traditional RAID (or moderate improvements on it, such as provided by btrfs and zfs) to provide drive-level redundancy seems like a waste of time when you also have to worry about redundancy between servers, racks, datacenters and regions. All of the arguments for why it's better to have something like zfs handle RAID rather than layering something on top of a traditional hardware RAID controller also work for explaining why you should prefer managing storage more globally rather than layering on top of something like zfs—if you have the resources to develop and maintain a true "full stack" storage solution. Which Facebook/Meta obviously does, when they can do things like publish their own spec documents that SSD vendors design around.
  - AdamJacobMuller 4y ago
    I imagine the "does not boot degraded" was mildly annoying to them, until they figured out the rootfsopts flag and deployed it. That whole thread above debating not booting degraded really made me lose any faith in btrfs.
- _kdave 4y ago
  AFAIK they use it internally, there are articles on lwn.net how, the use cases is for root filesystems and containers. I'm not sure I understand what you mean by the community sentiment, there are examples of code they'd developed internally first and sent it upstream, but in all cases I remember there were no problems. What can happen in the community is e.g. how the patches are organized or if the changelogs are complete. It's of course easier to develop something internally, if it touches other subsystems or if there's enough coverage just for the new code the test/fix/deploy cycle is much flexible. Once it's supposed to go through mailinlists or convincing other maintainers to accept changes it takes longer and must stick to the development cycle. This benefits both sides in the long run.
deleted 4y ago
[deleted]
ansible 4y ago
We had tried RAID-1 on btrfs, and the experience was not that great either. Our current systems, when they use btrfs, are on top of md (mdadm) RAID-1 arrays.
WastingMyTime89 4y ago
ZFS will never be mainlined so it’s beyond useless.
- waynesonfire 4y ago
  ZFS is widely used by many users and organizations. Just because it is not currently included in the Linux kernel does not mean that it is useless. There is ongoing work to integrate ZFS into the Linux kernel, and in the meantime, many users continue to find value in using ZFS on their systems. As an alternative, you could try using FreeBSD, which includes ZFS as part of the kernel. This can make it easier to take advantage of the features and benefits of ZFS without having to install and configure it separately. Additionally, because ZFS is integrated into the kernel, it can take advantage of FreeBSD's advanced security and performance features.
  - WastingMyTime89 4y ago
    Enlighten me on how a filesystem with an incompatible license is ever going to be mainlined. Having to use an out-of-tree patchset is an immediate no as far as I’m concerned. I also have no intention of switching to an hobbyist operating system, thank you.
    - skyyler 4y ago
      You're already using a hobbyist operating system... Your horse isn't as high as you think.
      - WastingMyTime89 4y ago
        Linux hasn’t really been a hobbyist operating system for years as anyone taking even a casual glance at a list of contributors would know but sure let’s all pretend FreeBSD is somehow relevant or properly maintained. That’s probably in the same universe where ZFS has a chance of being mainlined and isn’t heavily encumbered by Oracle owned patents. Why people would contribute to this mess is beyond me but everyone is free to do what they want.
coptun 4y ago
I've been rocking raid 5/6 btrfs for years for my personal data, no problems apart from the speed. A scrub literally takes months.
- sedatk 4y ago
  Yeah same here. Directory deletions are excruciatingly slow too.
- zeotroph 4y ago
  Months? I thought it would just go through all the data and verify that the stored checksums match. Even if it fetches filedata more than once to ensure the parity bits are ok, how can it take longer than reading the entire disk a couple times?
  - coptun 4y ago
    Well this is just a desktop not a server, it isn't on 24/7.
eikenberry 4y ago
Ideally on a ZFS replacement that has a compatible license.
- mekster 4y ago
  Using Ubuntu is hell of a lot faster than waiting for such a file system.
  - timbit42 4y ago
    Ubuntu is embracing an extending Linux through it's Snap repo. Dump it now.
    - michaelmrose 4y ago
      Alternatively just continue ignoring snap until it goes away
      - timbit42 4y ago
        I do, by not using Ubuntu.
  - eikenberry 4y ago
    So is using another file system.

viraptor 4y ago

Ideally we'd be working based on actual information rather than global mindshare. There will be guinea pigs to test it. There are already quite a few people running it despite the warnings. As much as I want to see a wide use of bcachefs, it's still years away. As someone who actually wants to store data - why would you direct resources to bcachefs which is known experimental, rather than btrfs which plainly documented raid5 as not ready and now may decide to change it to ready... if it is? (I'm donating to bcachefs patreon, but commenting from the PoV of someone choosing a solution today)

gjs278 4y ago
doublepg23 4y ago
Is bcachefs years away? My understanding is the critical chain of code is pretty well tested - bcache.
- viraptor 4y ago
  From widespread use, yes. Bcache was already stable at least 7 years ago. Bcachefs it's not even included in upstream kernel yet, much less any distro as an official option. There's only some plan to start submitting it soon™ and it will take months to complete.
ilyt 4y ago
Right but "actual information" here is "developers over and over fail to nail what ZFS and MDRAID did well decades ago". There is no reason to trust their next try will be "the one"
- viraptor 4y ago
  Did well in what context? RAID5? There were enough people with motivation to spend time on it rather than other things for mdadm/zfs. It's not like btrfs devs wouldn't know how to implement it if you paid them to work full time on it.
- _kdave 4y ago
  MDRAID had write hole until 4.4 (2015, https://lwn.net/Articles/665299/ https://lwn.net/Articles/665299/) and it had been long awaited back then. ZFS to my knowledge deals with that by variable stripe length that has its own problems but yeah it works. Btrfs changes implementation of the stripe update while preserving the on-disk format (i.e. can't do the same as ZFS without introducing incompatible change), intent log/bitmap have been proposed (which would be the MDRAID approach) but that's another incompatible change. So the 'next one' is at the cost of performance but with same compatibility.

denkmoon 4y ago

Do Meta not develop btrfs for their internal use? I don't think community sentiment is a big factor for them.

Scaevolus 4y ago
Meta and related companies have much better systems available for durability than RAID5. I would be moderately surprised if they commonly use any standard RAID level.
- wtallis 4y ago
  Using traditional RAID (or moderate improvements on it, such as provided by btrfs and zfs) to provide drive-level redundancy seems like a waste of time when you also have to worry about redundancy between servers, racks, datacenters and regions. All of the arguments for why it's better to have something like zfs handle RAID rather than layering something on top of a traditional hardware RAID controller also work for explaining why you should prefer managing storage more globally rather than layering on top of something like zfs—if you have the resources to develop and maintain a true "full stack" storage solution. Which Facebook/Meta obviously does, when they can do things like publish their own spec documents that SSD vendors design around.
- AdamJacobMuller 4y ago
  I imagine the "does not boot degraded" was mildly annoying to them, until they figured out the rootfsopts flag and deployed it. That whole thread above debating not booting degraded really made me lose any faith in btrfs.
_kdave 4y ago
AFAIK they use it internally, there are articles on lwn.net how, the use cases is for root filesystems and containers. I'm not sure I understand what you mean by the community sentiment, there are examples of code they'd developed internally first and sent it upstream, but in all cases I remember there were no problems. What can happen in the community is e.g. how the patches are organized or if the changelogs are complete. It's of course easier to develop something internally, if it touches other subsystems or if there's enough coverage just for the new code the test/fix/deploy cycle is much flexible. Once it's supposed to go through mailinlists or convincing other maintainers to accept changes it takes longer and must stick to the development cycle. This benefits both sides in the long run.

deleted 4y ago

[deleted]

ansible 4y ago

We had tried RAID-1 on btrfs, and the experience was not that great either. Our current systems, when they use btrfs, are on top of md (mdadm) RAID-1 arrays.

WastingMyTime89 4y ago

ZFS will never be mainlined so it’s beyond useless.

waynesonfire 4y ago
ZFS is widely used by many users and organizations. Just because it is not currently included in the Linux kernel does not mean that it is useless. There is ongoing work to integrate ZFS into the Linux kernel, and in the meantime, many users continue to find value in using ZFS on their systems. As an alternative, you could try using FreeBSD, which includes ZFS as part of the kernel. This can make it easier to take advantage of the features and benefits of ZFS without having to install and configure it separately. Additionally, because ZFS is integrated into the kernel, it can take advantage of FreeBSD's advanced security and performance features.
- WastingMyTime89 4y ago
  Enlighten me on how a filesystem with an incompatible license is ever going to be mainlined. Having to use an out-of-tree patchset is an immediate no as far as I’m concerned. I also have no intention of switching to an hobbyist operating system, thank you.
  - skyyler 4y ago
    You're already using a hobbyist operating system... Your horse isn't as high as you think.
    - WastingMyTime89 4y ago
      Linux hasn’t really been a hobbyist operating system for years as anyone taking even a casual glance at a list of contributors would know but sure let’s all pretend FreeBSD is somehow relevant or properly maintained. That’s probably in the same universe where ZFS has a chance of being mainlined and isn’t heavily encumbered by Oracle owned patents. Why people would contribute to this mess is beyond me but everyone is free to do what they want.

coptun 4y ago

I've been rocking raid 5/6 btrfs for years for my personal data, no problems apart from the speed. A scrub literally takes months.

sedatk 4y ago
Yeah same here. Directory deletions are excruciatingly slow too.
zeotroph 4y ago
Months? I thought it would just go through all the data and verify that the stored checksums match. Even if it fetches filedata more than once to ensure the parity bits are ok, how can it take longer than reading the entire disk a couple times?
- coptun 4y ago
  Well this is just a desktop not a server, it isn't on 24/7.

eikenberry 4y ago

Ideally on a ZFS replacement that has a compatible license.

mekster 4y ago
Using Ubuntu is hell of a lot faster than waiting for such a file system.
- timbit42 4y ago
  Ubuntu is embracing an extending Linux through it's Snap repo. Dump it now.
  - michaelmrose 4y ago
    Alternatively just continue ignoring snap until it goes away
    - timbit42 4y ago
      I do, by not using Ubuntu.
- eikenberry 4y ago
  So is using another file system.