5 ms·

This is just silly. As other have asked, how is this different from any other storage solution? (It's not.) The only thing to glean from this video is that LT

This is just silly. As other have asked, how is this different from any other storage solution? (It's not.) The only thing to glean from this video is that LTT didn't know what they were doing.

namibj 5y ago
Btrfs lets you keep metadata at a stronger bitrot-protection level fairly easily, and as long as that is still ok, you'll only use the data blocks that rotted, so at LTT's scale just a few seconds of footage in total.
- NavinF 5y ago
  Even if that’s true, btrfs would be entirely impractical for them because btrfs-raid6 is broken. Their petabyte server would become a 1/3 petabyte server without raidz2.
  - namibj 5y ago
    But that's more because no one dared to try and actually fix Btrfs parity raid. Misaligned incentives causing lack of funding for something as flexible and generally resilient as Btrfs.
    - guipsp 5y ago
      My data doesn't really care if the reason for brtfs losing data is lack of funding
    - matheusmoreira 5y ago
      Can you elaborate on this? I see NAS companies like Synology using btrfs in their products. I don't understand why btrfs RAID 5/6 is still not stable.
      - deleted 5y ago
        [deleted]
      - genghizkhan 5y ago
        Synology deploys btrfs on top of md-raid. They do not trust btrfs's raid layer.
      - namibj 5y ago
        Because there are some write holes around iirc power failures and, more importantly, restore basically doesn't work (the code just doesn't exist in a working variant). This isn't easy code to write, and it's Linux-Kernel-C, far from "comfy" code to write. I assume it's that kind of effect that continues to keep all competent people from fixing Btrfs's RAID 5/6 code. (If I could gather the motivation, I could probably do it, but that's far from fun to do and I'm already dire for motivation-to-code... it needs to be someone who likes that work, ideally sponsored so they don't loose money from working on that project.)
- mustache_kimono 5y ago
  > Btrfs lets you keep metadata at a stronger bitrot-protection level I really have no idea what BTRFS feature this refers to, but ZFS stores redundant copies of metadata. See: redundant_metadata, https://zfsonlinux.org/manpages/0.8.0/man8/zfs.8.html https://zfsonlinux.org/manpages/0.8.0/man8/zfs.8.html
  - namibj 5y ago
    Block group specific profiles. The normal these days is SINGLE for data and DUP for metadata. See how one converts here: https://btrfs.readthedocs.io/en/latest/btrfs-balance.html https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
    - mustache_kimono 5y ago
      Okay, I still don't understand your point. BTRFS stores duplicate copies of the metadata. Good for BTRFS. I don't understand how this would have prevented what LTT experienced. First, as someone else said -- no one would use a BTRFS raid5/6 array in production. No one. Second, I don't think what LTT experienced was bit rot. There were far, far too many data errors for that be simple bit rot. My guess is bad cable, bad HBA which maybe doesn't support something like ASPM, etc. Third, are we even sure the problem was corrupted metadata? Fourth, ZFS keeps redundant copies of metadata too (as do many filesystems.) So the question is what's your point?
      - namibj 5y ago
        That you could have dialed up the redundancy for metadata to 3x or even 4x copies being stored, and just because a block got bitrot doesn't mean other blocks fail to read. I know no one would use Btrfs raid5/6 in production, but that's because no one wanted to enough to get the required development done. If it was just file blocks that got corrupted, there would be just very minor damage in total and a scrub that logs all damaged blocks typically proceeds at sequential read speeds from what I've observed in the past.
        5y ago

This is just silly. As other have asked, how is this different from any other storage solution? (It's not.) The only thing to glean from this video is that LTT didn't know what they were doing.

namibj 5y ago

Btrfs lets you keep metadata at a stronger bitrot-protection level fairly easily, and as long as that is still ok, you'll only use the data blocks that rotted, so at LTT's scale just a few seconds of footage in total.

NavinF 5y ago
Even if that’s true, btrfs would be entirely impractical for them because btrfs-raid6 is broken. Their petabyte server would become a 1/3 petabyte server without raidz2.
- namibj 5y ago
  But that's more because no one dared to try and actually fix Btrfs parity raid. Misaligned incentives causing lack of funding for something as flexible and generally resilient as Btrfs.
  - guipsp 5y ago
    My data doesn't really care if the reason for brtfs losing data is lack of funding
  - matheusmoreira 5y ago
    Can you elaborate on this? I see NAS companies like Synology using btrfs in their products. I don't understand why btrfs RAID 5/6 is still not stable.
    - deleted 5y ago
      [deleted]
    - genghizkhan 5y ago
      Synology deploys btrfs on top of md-raid. They do not trust btrfs's raid layer.
    - namibj 5y ago
      Because there are some write holes around iirc power failures and, more importantly, restore basically doesn't work (the code just doesn't exist in a working variant). This isn't easy code to write, and it's Linux-Kernel-C, far from "comfy" code to write. I assume it's that kind of effect that continues to keep all competent people from fixing Btrfs's RAID 5/6 code. (If I could gather the motivation, I could probably do it, but that's far from fun to do and I'm already dire for motivation-to-code... it needs to be someone who likes that work, ideally sponsored so they don't loose money from working on that project.)
mustache_kimono 5y ago
> Btrfs lets you keep metadata at a stronger bitrot-protection level I really have no idea what BTRFS feature this refers to, but ZFS stores redundant copies of metadata. See: redundant_metadata, https://zfsonlinux.org/manpages/0.8.0/man8/zfs.8.html https://zfsonlinux.org/manpages/0.8.0/man8/zfs.8.html
- namibj 5y ago
  Block group specific profiles. The normal these days is SINGLE for data and DUP for metadata. See how one converts here: https://btrfs.readthedocs.io/en/latest/btrfs-balance.html https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
  - mustache_kimono 5y ago
    Okay, I still don't understand your point. BTRFS stores duplicate copies of the metadata. Good for BTRFS. I don't understand how this would have prevented what LTT experienced. First, as someone else said -- no one would use a BTRFS raid5/6 array in production. No one. Second, I don't think what LTT experienced was bit rot. There were far, far too many data errors for that be simple bit rot. My guess is bad cable, bad HBA which maybe doesn't support something like ASPM, etc. Third, are we even sure the problem was corrupted metadata? Fourth, ZFS keeps redundant copies of metadata too (as do many filesystems.) So the question is what's your point?
    - namibj 5y ago
      That you could have dialed up the redundancy for metadata to 3x or even 4x copies being stored, and just because a block got bitrot doesn't mean other blocks fail to read. I know no one would use Btrfs raid5/6 in production, but that's because no one wanted to enough to get the required development done. If it was just file blocks that got corrupted, there would be just very minor damage in total and a scrub that logs all damaged blocks typically proceeds at sequential read speeds from what I've observed in the past.
      - 5y ago