3 ms·

Conversely, I haven't had a single btrfs issue on my home server, which has gradually expanded from a 2x2TB RAID1. New disks get added to the pool via 'device a

Conversely, I haven't had a single btrfs issue on my home server, which has gradually expanded from a 2x2TB RAID1. New disks get added to the pool via 'device add' and off we go. That's one of the biggest reasons why I use btrfs, it lets me turn storage into a large generic pool instead of individual partitions and drives, and it's very flexible when combining drives of different sizes. The only issue I did have was a bad ASM1061-based SATA controller, which would corrupt data in weird ways, throw up alignment errors, generic I/O errors, all kinds of issue. That can hardly be blamed on btrfs. Because it was a RAID1 pool and a checksumming filesystem, I could switch the affected drive to a known good SATA controller and correct all of the corrupted data, to bring everything back in line. I found it very robust and undramatic. I strongly assume it would have worked more or less the same on ZFS, but ZFS is less flexible when adding disks (have to add new separate vdevs instead of integrating individual drives), so I stick with btrfs. Don't buy ASM1061-based SATA controllers. At the very least, get something Marvell-based, or an LSI-based SAS controller in IT mode, if you want to get fancy.

minimaul 5y ago
Yeah, I notice some people have very different experiences. It's odd. For me it's a mix of NVMe drives, SATA SSDs on Intel SATA, or SATA SSDs on LSI SAS HBAs (in IT mode). It doesn't seem to be pinned to specific hardware, sadly - and the exact same hardware works fine with zfs/ext4/xfs.
- KozmoNau7 5y ago
  It would be tempting to chalk it up to marginal storage controllers and/or memory or even flaky power supplies. I've seen all three result in very weird and hard to diagnose issues, which presumably a more critical and checksumming filesystem is simply better at sussing out than a more old-fashioned FS like ext4 or XFS. But from the impression of the type of hardware you're running, I highly doubt that is the case. And of course ZFS would be affected, too.
  - minimaul 5y ago
    I agree - but in cases of marginal hardware I'd expect to see more strange issues than just specifically FS corruption with btrfs. At one point it was limited to if I was using compression, but using compression was the major reason I wanted btrfs for that use rather than ext4/xfs.
  - hnlmorg 5y ago
    > And of course ZFS would be affected, too. In theory, yes. But one of the design goals of ZFS was to be resilient against faulty hardware. On paper that seems an impossible task but I've seen first hand just how impressive it is where I had a faulty raid controller which would randomly disconnect HDDs but only under heavy load. It took weeks to debug this -- it was so bad it actually caused Solaris to crash (which made fault diagnosis all the harder). I then switched to FreeBSD, which was a little more stable so enabled me to get a little more meaningful diagnostics and when I realised what the fault was, I was amazed I hadn't actually lost any data. At one point the fault was so bad it completely trashed the superblock but FreeBSD's ZFS tooling meant I could recover that superblock and the entire outage took all of 10 minutes. On the flip-side, the few times I've used btrfs I've either been left scratching my head at the confusing tooling, or pulling my hair out because of data loss. On one system, the storage pool -- which literally just consisted of one HDD and no clever raiding -- reverted itself by 6 months. I was an early adopter of ZFS and never ever experienced so much as a blip despite running it on hardware that really should have caused ZFS to go tits up. Yet Btrfs would get it's knickers in a twist on stable hardware. I get my stories here are purely anecdotal but I her the same kinds of stories crop up over and over again from other people too. So I find it very hard to take Btrfs seriously as a ZFS replacement. But then given ZFS is an older and more mature project and ZFS already works pretty damn well, I also question why anyone would want a replacement to ZFS when they could just use ZFS.
    - arp242 5y ago
      > But then given ZFS is an older and more mature project and ZFS already works pretty damn well, I also question why anyone would want a replacement to ZFS when they could just use ZFS. Basically just licencing. The great irony is that btrfs was started by Oracle (not sure if they're still involved) before they acquired Sun. A relicensing effort to GPL would probably be better, but, you know, Oracle. I argued in the past that OpenZFS should have contributors sign a small CLA which permits relicensing to GPL to prepare for this nonetheless, it's a small effort and could save a lot of effort later.

minimaul 5y ago

Yeah, I notice some people have very different experiences. It's odd. For me it's a mix of NVMe drives, SATA SSDs on Intel SATA, or SATA SSDs on LSI SAS HBAs (in IT mode). It doesn't seem to be pinned to specific hardware, sadly - and the exact same hardware works fine with zfs/ext4/xfs.

KozmoNau7 5y ago
It would be tempting to chalk it up to marginal storage controllers and/or memory or even flaky power supplies. I've seen all three result in very weird and hard to diagnose issues, which presumably a more critical and checksumming filesystem is simply better at sussing out than a more old-fashioned FS like ext4 or XFS. But from the impression of the type of hardware you're running, I highly doubt that is the case. And of course ZFS would be affected, too.
- minimaul 5y ago
  I agree - but in cases of marginal hardware I'd expect to see more strange issues than just specifically FS corruption with btrfs. At one point it was limited to if I was using compression, but using compression was the major reason I wanted btrfs for that use rather than ext4/xfs.
- hnlmorg 5y ago
  > And of course ZFS would be affected, too. In theory, yes. But one of the design goals of ZFS was to be resilient against faulty hardware. On paper that seems an impossible task but I've seen first hand just how impressive it is where I had a faulty raid controller which would randomly disconnect HDDs but only under heavy load. It took weeks to debug this -- it was so bad it actually caused Solaris to crash (which made fault diagnosis all the harder). I then switched to FreeBSD, which was a little more stable so enabled me to get a little more meaningful diagnostics and when I realised what the fault was, I was amazed I hadn't actually lost any data. At one point the fault was so bad it completely trashed the superblock but FreeBSD's ZFS tooling meant I could recover that superblock and the entire outage took all of 10 minutes. On the flip-side, the few times I've used btrfs I've either been left scratching my head at the confusing tooling, or pulling my hair out because of data loss. On one system, the storage pool -- which literally just consisted of one HDD and no clever raiding -- reverted itself by 6 months. I was an early adopter of ZFS and never ever experienced so much as a blip despite running it on hardware that really should have caused ZFS to go tits up. Yet Btrfs would get it's knickers in a twist on stable hardware. I get my stories here are purely anecdotal but I her the same kinds of stories crop up over and over again from other people too. So I find it very hard to take Btrfs seriously as a ZFS replacement. But then given ZFS is an older and more mature project and ZFS already works pretty damn well, I also question why anyone would want a replacement to ZFS when they could just use ZFS.
  - arp242 5y ago
    > But then given ZFS is an older and more mature project and ZFS already works pretty damn well, I also question why anyone would want a replacement to ZFS when they could just use ZFS. Basically just licencing. The great irony is that btrfs was started by Oracle (not sure if they're still involved) before they acquired Sun. A relicensing effort to GPL would probably be better, but, you know, Oracle. I argued in the past that OpenZFS should have contributors sign a small CLA which permits relicensing to GPL to prepare for this nonetheless, it's a small effort and could save a lot of effort later.