4 ms·

I am not sure why this person feels compelled to use btrfs raid1 when mdadm raid1 exists and works just fine at the block device level.

trillic 5y ago
Exactly, especially if he is using snapshots, BTRFS can cause serious write amplification for frequently snapshotted and changed data.
- etbe 5y ago
  I'm not sure it makes a lot of difference unless you have hundreds of snapshots. The way changes work to BTRFS is that every change without snapshots does most of the same stuff that a change with snapshots does. If however you have many snapshots (more than 500 according to rumours, I've seen it with over 1000) performance can get really bad. If your script to make snapshot backups works and the script to remove old ones doesn't work properly then the system runs fine for months and suddenly goes ridiculously slow. A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.
- MertsA 5y ago
  What's nice is that as of Linux 5.16 BTRFS now supports ZNS (https://zonedstorage.io/ https://zonedstorage.io/) to drastically reduce write amplification compared to traditional SSDs by bypassing a lot of the FTL and moving to an interface that's similar to host managed SMR drives.
j16sdiz 5y ago
If a bit is flipped on one of the SSD, btrfs can recover using checksum. Not sure how useful it is, but this is what btrfs/zfs people say
- tuetuopay 5y ago
  Happened on my NAS: had a corruption due to a bitflip, BTRFS recovered fine.
- amarshall 5y ago
  As a ZFS user: quite useful. I’ve had numerous of cases of corruption repaired by routine scrubs. Almost every other filesystem (btrfs being an exception afaik) would have been none the wiser.
  - ksec 5y ago
    Useful for HDD, but on SSD? When both the NAND and Controller has Error Correction?
    - rhn_mk1 5y ago
      It's just as useful on either. Error correction predates SSDs.
    - magicalhippo 5y ago
      Errors can happen during transmission as well, especially if cables or backplanes are involved.
    - etbe 5y ago
      In addition to cables and backplanes that have been mentioned, the entire path from RAM to PCIe to disk controller has non zero risks of errors. That said I'm pretty sure that the BTRFS errors I've seen from SSDs have all been the fault of the SSDs in question. Also note that hard drives have had internal error detection and correction codes for decades, so SSDs doesn't change things much. Probably most checksum errors are misplaced writes and missing writes.
    - MertsA 5y ago
      HDDs also use ECC hence why read errors occur on failing drives instead of just silently returning corrupt data. That's not an ironclad guarantee and plenty of places in the data path can let an errant cosmic ray flip bits. BTRFS doesn't even eliminate this, it just reduces the surface area and most importantly gives you a way to durably detect or correct this from when the checksum is generated in the kernel onwards.
  - hardwaresofton 5y ago
tuetuopay 5y ago
BTRFS RAID1 has numerous advantages compared to regular RAID1. It does not mirrors whole disks, but chunks of the disk, usually 1GB chunks. This has numerous implications: - drive size can be mixed without usable space loss (kind of, with some combinations you end up losing a bit) - you don't need an even number of drives - you don't need a raid0 atop several raid1 to use e.g. 6 drives, it's all one big raid - a drive can be swapped for a bigger or smaller drive without issue, meaning spare drives don't need to be identical etc.
- justsomehnguy 5y ago
  Just use LVM thin-provisioned RAID1 volume.
- matheusmoreira 5y ago
  This is such a great feature, it makes it really easy and convenient to expand btrfs arrays. It's such a pity that the parity support is still unstable. Now that ZFS has RAID-Z expansion I don't need to wait anymore.
- brnt 5y ago
  How would a Btrfs RAID1 array of (equally sized) spinning rust and SSDs work? Will it slow down to the speed of the HDD, or will I get SSD speeds and will the HDD eventually catch up? If not by default, then by option?
  - rhn_mk1 5y ago
    For the reading, btrfs currently chooses one of the drives at random, I think based on the PID of the process that's doing the reading. For writing, you have to wait for the slow one.
  - etbe 5y ago
    If you want to mix SSD and hard drives then you probably want bcache.
etbe 5y ago
In the post I wrote 'between the 2 SSDs SMART reports 18 uncorrectable errors and “btrfs device stats” reports 55 errors on one of them'. That's 37 errors that BTRFS found that mdadm wouldn't have found. 37 errors out of 500G of data isn't a lot, but it depends where it occurs, if it's in the root directory then it will mess up your day (annoyingly BTRFS doesn't track where the errors were so you can't easily determine what the implications of not having BTRFS might be). That is not unusual, I have had similar things on almost all my systems that run BTRFS or ZFS. Also I use snapshots too.

trillic 5y ago

Exactly, especially if he is using snapshots, BTRFS can cause serious write amplification for frequently snapshotted and changed data.

etbe 5y ago
I'm not sure it makes a lot of difference unless you have hundreds of snapshots. The way changes work to BTRFS is that every change without snapshots does most of the same stuff that a change with snapshots does. If however you have many snapshots (more than 500 according to rumours, I've seen it with over 1000) performance can get really bad. If your script to make snapshot backups works and the script to remove old ones doesn't work properly then the system runs fine for months and suddenly goes ridiculously slow. A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.
MertsA 5y ago
What's nice is that as of Linux 5.16 BTRFS now supports ZNS (https://zonedstorage.io/ https://zonedstorage.io/) to drastically reduce write amplification compared to traditional SSDs by bypassing a lot of the FTL and moving to an interface that's similar to host managed SMR drives.

j16sdiz 5y ago

If a bit is flipped on one of the SSD, btrfs can recover using checksum. Not sure how useful it is, but this is what btrfs/zfs people say

tuetuopay 5y ago
Happened on my NAS: had a corruption due to a bitflip, BTRFS recovered fine.
amarshall 5y ago
As a ZFS user: quite useful. I’ve had numerous of cases of corruption repaired by routine scrubs. Almost every other filesystem (btrfs being an exception afaik) would have been none the wiser.
- ksec 5y ago
  Useful for HDD, but on SSD? When both the NAND and Controller has Error Correction?
  - rhn_mk1 5y ago
    It's just as useful on either. Error correction predates SSDs.
  - magicalhippo 5y ago
    Errors can happen during transmission as well, especially if cables or backplanes are involved.
  - etbe 5y ago
    In addition to cables and backplanes that have been mentioned, the entire path from RAM to PCIe to disk controller has non zero risks of errors. That said I'm pretty sure that the BTRFS errors I've seen from SSDs have all been the fault of the SSDs in question. Also note that hard drives have had internal error detection and correction codes for decades, so SSDs doesn't change things much. Probably most checksum errors are misplaced writes and missing writes.
  - MertsA 5y ago
    HDDs also use ECC hence why read errors occur on failing drives instead of just silently returning corrupt data. That's not an ironclad guarantee and plenty of places in the data path can let an errant cosmic ray flip bits. BTRFS doesn't even eliminate this, it just reduces the surface area and most importantly gives you a way to durably detect or correct this from when the checksum is generated in the kernel onwards.
- hardwaresofton 5y ago

tuetuopay 5y ago

BTRFS RAID1 has numerous advantages compared to regular RAID1. It does not mirrors whole disks, but chunks of the disk, usually 1GB chunks. This has numerous implications: - drive size can be mixed without usable space loss (kind of, with some combinations you end up losing a bit) - you don't need an even number of drives - you don't need a raid0 atop several raid1 to use e.g. 6 drives, it's all one big raid - a drive can be swapped for a bigger or smaller drive without issue, meaning spare drives don't need to be identical etc.

justsomehnguy 5y ago
Just use LVM thin-provisioned RAID1 volume.
matheusmoreira 5y ago
This is such a great feature, it makes it really easy and convenient to expand btrfs arrays. It's such a pity that the parity support is still unstable. Now that ZFS has RAID-Z expansion I don't need to wait anymore.
brnt 5y ago
How would a Btrfs RAID1 array of (equally sized) spinning rust and SSDs work? Will it slow down to the speed of the HDD, or will I get SSD speeds and will the HDD eventually catch up? If not by default, then by option?
- rhn_mk1 5y ago
  For the reading, btrfs currently chooses one of the drives at random, I think based on the PID of the process that's doing the reading. For writing, you have to wait for the slow one.
- etbe 5y ago
  If you want to mix SSD and hard drives then you probably want bcache.

etbe 5y ago

In the post I wrote 'between the 2 SSDs SMART reports 18 uncorrectable errors and “btrfs device stats” reports 55 errors on one of them'. That's 37 errors that BTRFS found that mdadm wouldn't have found. 37 errors out of 500G of data isn't a lot, but it depends where it occurs, if it's in the root directory then it will mess up your day (annoyingly BTRFS doesn't track where the errors were so you can't easily determine what the implications of not having BTRFS might be). That is not unusual, I have had similar things on almost all my systems that run BTRFS or ZFS. Also I use snapshots too.