4 ms·

> which rules out all RAID levels except 0, which is nuts. RAID 1, you mean? Because that way you still have a complete copy of your data if one drive fails.

> which rules out all RAID levels except 0, which is nuts. RAID 1, you mean? Because that way you still have a complete copy of your data if one drive fails. BTRFS is excellent for the use case of a wide variety of mismatched drives, because it supports adding and removing drives and rebalancing the array. But for the moment only the RAID 1 modes are really trustworthy. I have a NAS consisting of drives whose advertised capacities in GB are: 1000, 1000, 1920, 2000, 2048, 2050, 3840, 3840, 4000. But it started as a pile of infamously unreliable 3TB hard drives.

hinkley 3y ago
Raid 1 does not work with heterogenous disk arrays.
- ilyt 3y ago
  Eh, it can kinda work. If you have 1, 2,3,4TB drives you can make RAID10 out of first GB, then stitch 2TB RAID0 from second and third and RAID1 it with 2TB out of 4th drive. then raid1 out of remaining GB on 3rd and 4th. Then glue it together with LVM and hope for best.
  - wtallis 3y ago
    Or you can just put all those devices into one BTRFS RAID 1 filesystem and not have to worry about it.
    - ilyt 3y ago
      BTRFS doesn't exactly had great history with fs bugs
    - tatersolid 3y ago
      Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…
      - wtallis 3y ago
        I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility of BTRFS gives you a lot more options than eg. ZFS for getting out of a tight spot without the full wipe and restore hassle. But both can still leave you stuck, especially if you ignore the need for monitoring and preventive maintenance.
        tatersolid 3y ago
        Disk full is not a “corner case”. It should be one of the first test cases a filesystem test suite should have, and that test should always pass in every configuration. The fact that total data loss happened regularly on full volumes many years after a “stable” release means BTRFS is either fundamentally flawed in design or incompetently implemented. I have no idea how it was accepted into mainline Linux. The fact that a certain large NAS vendor used BTRFS by default (without documentation at the time of purchase) has cost my org and others lots of downtime and money.
        wtallis 3y ago
        > Disk full is not a “corner case”. I didn't say it was. There are plenty of disk full scenarios that can be handled by BTRFS without trouble, especially when adding more disks (even temporarily) is an option. There are ways to get stuck with a full fs, but it's not a guarantee you'll end up in a "wipe and restore from backup" situation. (And to be fair, ZFS also gets very problematic when completely full; this is a general problem for CoW filesystems.)
        ilyt 3y ago
        Sooo basically if you were using XFS you wouldn't need to do anything at all till it got 99% full, and if you got it there you'd just get "disk is full" message instead of some catastrophe? I don't need that kind of stress in my life... And with plain RAID6 I can still add hard drives if needed. Yeah the case of buying bigger drives is still a bit iffy but btrfs RAID1 gonna lose me more space than RAID6+LVM+xfs anyway...
  - hinkley 3y ago
    There is what a man can do, and what he should do. Nobody normal is going to do any of this.
- wtallis 3y ago
  BTRFS RAID 1 works just fine with heterogeneous disks with no extra configuration required; I would not have been able to provide examples from experience otherwise. When writing new data, it allocates two blocks on two separate devices, preferring to allocate space on whatever devices have the most free space. This may not be optimal from a performance perspective, but it does mean there's minimal wasted capacity in most configurations. (The RAID1c3 and RAID1c4 modes work the same, but with three or four copies of each block, all on separate devices.) Were you assuming that BTRFS RAID 1 meant every block of data gets mirrored across every device? I've seen that assumption before, from people who don't realize there are two ways to generalize RAID1 to more than two devices.
keeperofdakeys 3y ago
The BTRFS raid1 feature is very badly named, its best called two-copy. Skipping some details, two copies of data are written to the two drives with the most free space. So you can have multiple mismatched drives. However you get no striping, and data is only read from one drive, so performance is limited to that of one drive for reads and writes. Plus with mismatched drives, smaller drives go unused unless you write enough data.
- wtallis 3y ago
  > The BTRFS raid1 feature is very badly named, You're going to have to support that claim a bit better. The core idea of RAID 1 is mirroring data, which BTRFS RAID 1 mode definitely does. Striping is not an essential part of RAID 1 (hence RAID 10), and reading data from two disks in parallel is an optional performance optimization that is not performed by all RAID 1 implementations (but could be implemented for BTRFS RAID 1: https://stackoverflow.com/questions/55408256/btrfs-raid-1-which-device-gets-the-reads https://stackoverflow.com/questions/55408256/btrfs-raid-1-wh... ). > Plus with mismatched drives, smaller drives go unused unless you write enough data. Yes, the allocation is suboptimal from a performance perspective, as I've already said. But it is simple and straightforward and reasonably good at avoiding putting you into a situation where manually issuing a rebalance command is necessary. If you do need better performance, there's a RAID 10 mode. But since my NAS is currently a motley pile of SSDs, I don't need to to anything extra to have decent performance.
  - keeperofdakeys 3y ago
    RAID1 is about mirroring disks, BTRFS RAID1 mirrors block groups. Plus traditionally a RAID1 of 3 disks will mirror the same data on all disks, which is different to how the RAID1 mode on BTRFS acts. So the name leads to misunderstandings since it doesn't act like RAID1 at all. It'd be way easier to talk about if it had a unique name, and you could say "It's like RAID1". Despite all that I do like the mode, and use it in a few places.
    - PlutoIsAPlanet 3y ago

hinkley 3y ago

Raid 1 does not work with heterogenous disk arrays.

ilyt 3y ago
Eh, it can kinda work. If you have 1, 2,3,4TB drives you can make RAID10 out of first GB, then stitch 2TB RAID0 from second and third and RAID1 it with 2TB out of 4th drive. then raid1 out of remaining GB on 3rd and 4th. Then glue it together with LVM and hope for best.
- wtallis 3y ago
  Or you can just put all those devices into one BTRFS RAID 1 filesystem and not have to worry about it.
  - ilyt 3y ago
    BTRFS doesn't exactly had great history with fs bugs
  - tatersolid 3y ago
    Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…
    - wtallis 3y ago
      I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility of BTRFS gives you a lot more options than eg. ZFS for getting out of a tight spot without the full wipe and restore hassle. But both can still leave you stuck, especially if you ignore the need for monitoring and preventive maintenance.
      - tatersolid 3y ago
        Disk full is not a “corner case”. It should be one of the first test cases a filesystem test suite should have, and that test should always pass in every configuration. The fact that total data loss happened regularly on full volumes many years after a “stable” release means BTRFS is either fundamentally flawed in design or incompetently implemented. I have no idea how it was accepted into mainline Linux. The fact that a certain large NAS vendor used BTRFS by default (without documentation at the time of purchase) has cost my org and others lots of downtime and money.
        wtallis 3y ago
        > Disk full is not a “corner case”. I didn't say it was. There are plenty of disk full scenarios that can be handled by BTRFS without trouble, especially when adding more disks (even temporarily) is an option. There are ways to get stuck with a full fs, but it's not a guarantee you'll end up in a "wipe and restore from backup" situation. (And to be fair, ZFS also gets very problematic when completely full; this is a general problem for CoW filesystems.)
      - ilyt 3y ago
        Sooo basically if you were using XFS you wouldn't need to do anything at all till it got 99% full, and if you got it there you'd just get "disk is full" message instead of some catastrophe? I don't need that kind of stress in my life... And with plain RAID6 I can still add hard drives if needed. Yeah the case of buying bigger drives is still a bit iffy but btrfs RAID1 gonna lose me more space than RAID6+LVM+xfs anyway...
- hinkley 3y ago
  There is what a man can do, and what he should do. Nobody normal is going to do any of this.
wtallis 3y ago
BTRFS RAID 1 works just fine with heterogeneous disks with no extra configuration required; I would not have been able to provide examples from experience otherwise. When writing new data, it allocates two blocks on two separate devices, preferring to allocate space on whatever devices have the most free space. This may not be optimal from a performance perspective, but it does mean there's minimal wasted capacity in most configurations. (The RAID1c3 and RAID1c4 modes work the same, but with three or four copies of each block, all on separate devices.) Were you assuming that BTRFS RAID 1 meant every block of data gets mirrored across every device? I've seen that assumption before, from people who don't realize there are two ways to generalize RAID1 to more than two devices.

keeperofdakeys 3y ago

The BTRFS raid1 feature is very badly named, its best called two-copy. Skipping some details, two copies of data are written to the two drives with the most free space. So you can have multiple mismatched drives. However you get no striping, and data is only read from one drive, so performance is limited to that of one drive for reads and writes. Plus with mismatched drives, smaller drives go unused unless you write enough data.

wtallis 3y ago
> The BTRFS raid1 feature is very badly named, You're going to have to support that claim a bit better. The core idea of RAID 1 is mirroring data, which BTRFS RAID 1 mode definitely does. Striping is not an essential part of RAID 1 (hence RAID 10), and reading data from two disks in parallel is an optional performance optimization that is not performed by all RAID 1 implementations (but could be implemented for BTRFS RAID 1: https://stackoverflow.com/questions/55408256/btrfs-raid-1-which-device-gets-the-reads https://stackoverflow.com/questions/55408256/btrfs-raid-1-wh... ). > Plus with mismatched drives, smaller drives go unused unless you write enough data. Yes, the allocation is suboptimal from a performance perspective, as I've already said. But it is simple and straightforward and reasonably good at avoiding putting you into a situation where manually issuing a rebalance command is necessary. If you do need better performance, there's a RAID 10 mode. But since my NAS is currently a motley pile of SSDs, I don't need to to anything extra to have decent performance.
- keeperofdakeys 3y ago
  RAID1 is about mirroring disks, BTRFS RAID1 mirrors block groups. Plus traditionally a RAID1 of 3 disks will mirror the same data on all disks, which is different to how the RAID1 mode on BTRFS acts. So the name leads to misunderstandings since it doesn't act like RAID1 at all. It'd be way easier to talk about if it had a unique name, and you could say "It's like RAID1". Despite all that I do like the mode, and use it in a few places.
  - PlutoIsAPlanet 3y ago