4 ms·

But btrfs RAID-5 still suffers from the 'write hole' problem where data can be permanently lost. We looked at RAID-5 performance for both ZFS and btrfs, and btr

But btrfs RAID-5 still suffers from the 'write hole' problem where data can be permanently lost. We looked at RAID-5 performance for both ZFS and btrfs, and btrfs is about twice the throughput. One of the reasons for the performance difference is the following: when ZFS writes a stripe, it waits for all blocks and the parity block to confirm they have been written before atomically updating the metadata for that stripe; when btrfs writes a stripe, it just updates the metadata and all blocks in parallel. Hence if btrfs crashes, metadata may say a stripe is written but not all blocks are written. It will be interesting to see the performance hit on btrfs when they finally fix the write hole problem. My prediction is that it will perform similarly to ZFS for RAID5. Currently, for writes, we can get about 400MB/s for btrfs RAID5, but only 220MB/s for ZFS with five 5400 rpm 2.5" disks.

poizan42 10y ago
That should be exposed as an option so the user can choose whether they need performance or reliability.
krondor 10y ago
Interesting findings. I'm curious also to see how fixed width stripes change the performance profile in btrfs (presently not supported). To be clear though, the raid5/6 write hole only applies to power failures with data in flight. It is still a concern, of course, but I think depending on your environment is acceptable to some (redundant PSUs, well engineered PDUs and UPS systems). Personally, I'm of increasing opinion that parity rebuilds aren't worth it anyway. I'd rather raid10, raid1, or raid0 depending on use. If I have to take a system out of production during parity rebuild (because IO activity is too intense for performant use), might as well not parity rebuild and simply reload the system on failure and rely on other cluster nodes.
- jamesblonde 10y ago
  Raid5 is not dead yet (https://www.cafaro.net/2014/05/26/why-raid-5-is-not-dead-yet/ https://www.cafaro.net/2014/05/26/why-raid-5-is-not-dead-yet...). The problem with failures during rebuilds is overblown, IMO. Manufacturer quoted URE failure rate (probability of failure to read) is overstated - instead of 1×10^14, they are mostly like 1×10^15 or higher. Full disclosure: we're actually doing erasure coding in HDFS over Raid5 on servers (double insurance - if the raid array goes down, we can recover from other servers in HDFS). But our expectation for 6x4TB arrays is not for a 70%+ chance of a URE during a rebuild, rather a couple of percent. With ZFS or btrfs, it won't actually matter for us, as we'll only lose a block on a URE- that we can recover from the rest of the cluster.
  - technion 10y ago
    The problem with failures during rebuilds is overblown I thought I was the only one who believed that. I've said this on reddit before and ended on like -20 votes with people blatantly arguing I'm falsifying an "impossibility". I've got roughly 30 arrays in production, between 4 and 12 disks in each. All are RAID5 + hotspare. If you believe the maths people keep quoting, the odds of seeing a total failure in a given year is close to 100%. I started using this configuration, across varying hardware, over 15 years ago and I've been growing in number since. I'm not pretending one example proves the rule, or that it's totally safe and I would run a highly critical environment this way (before anyone comments: these environments do not meet that definition), but people have tried to show maths that there's a six nine likelihood of failure, and I just don't for a second believe I'm that lucky.
    - jamesblonde 10y ago
      Follow the money who is selling the raid5 is dead story. The main worry is correlated failures if you have the Sam types.of drives in arrays and they reach their end of life.
    - cmurf 10y ago
      Well at least on Linux, by default almost everyone (using consumer drives) has their array in a very common misconfiguration. And this leads to raid5 collapse much sooner than it should. The misconfiguration is the drive's SCT ERC timeout is greater than the kernel's SCSI command timer. So what happens on a URE is, the drive does "deep recovery" if it's a consumer drive, and keeps trying to recover that bad sector well beyond the default command timer of the kernel, which is 30 seconds. At 30 seconds the kernel assumes something's wrong and does a link reset. On SATA drives this obliterates the command queue and any other state in the drive. The drive doesn't report a read error, doesn't report what sector had the problem, and so RAID can't do its job and fix the problem by reconstructing the missing data from parity and writing the data back to that bad sector. So it's inevitable these bad sectors pop up here and there, and then if there's a single drive failure, in effect you get one or more full stripes with two or more missing strips, and now those whole stripes are lost just as if it were a 2-disk failure. It is possible to recover from this but it's really tedious and as far as I know there are no user space tools to make such recovery easy. I wouldn't be surprised if lots of NAS's using Linux were configured this way, and the user didn't use recommended drives because, FU vendor those drives are expensive, etc.
- cmurf 10y ago
  For scaleable storage, get rid of conventional RAID for data. I'd like to see n-way (definable) copies of metadata, and single copies of data. On top is a cluster file system like GlusterFS. When a device dies, the file system merely rebuilds metadata, and then informs GlusterFS of the missing data due to the failed drive(s). And then the file system deletes the reference to all missing/damaged files from that missing drive. No degraded state ever happens. This way Gluster knows not to even make requests from that brick. If the brick were raid56 and the cluster fs isn't aware, requests happen with degraded read/writes which suck performance wise. Plus Gluster (or even Ceph for that matter) might use some logic and say, well that data doesn't even need to be replicated on that brick, it's better for the network/use loading if the new copy is on this other brick over here.
  - jamesblonde 10y ago
    Sounds like hdfs....
xioxox 10y ago
Have you tried btrfs on an md RAID device (for interest)? We've run it for a while, but we don't care too much about performance.
- krondor 10y ago
  That seems ill advised, one of the benefits of btrfs is it obviates the need for lvm and mdadm. I guess in your case you have a more stable raid5/6 opportunity, but you're losing many of the raid benefits present in btrfs natively. I'd also imaging it could be slower or introduce IO issues others haven't tested. Though I really have no idea, never seen anyone do that before.
  - xioxox 10y ago
    It's been great for our use case - we've been using it for about 5 years like this. We use it for rsyncing data onto as a backup (not the only one!), making daily snapshots. md is more flexible than zfs's raid, but less flexible as native btrfs raid.
- jamesblonde 10y ago
  No, but a lot of people on the btrfs mailing list recommend this due to the lack of stability for raid5/6 on btrfs.

poizan42 10y ago

That should be exposed as an option so the user can choose whether they need performance or reliability.

krondor 10y ago

Interesting findings. I'm curious also to see how fixed width stripes change the performance profile in btrfs (presently not supported). To be clear though, the raid5/6 write hole only applies to power failures with data in flight. It is still a concern, of course, but I think depending on your environment is acceptable to some (redundant PSUs, well engineered PDUs and UPS systems). Personally, I'm of increasing opinion that parity rebuilds aren't worth it anyway. I'd rather raid10, raid1, or raid0 depending on use. If I have to take a system out of production during parity rebuild (because IO activity is too intense for performant use), might as well not parity rebuild and simply reload the system on failure and rely on other cluster nodes.

jamesblonde 10y ago
Raid5 is not dead yet (https://www.cafaro.net/2014/05/26/why-raid-5-is-not-dead-yet/ https://www.cafaro.net/2014/05/26/why-raid-5-is-not-dead-yet...). The problem with failures during rebuilds is overblown, IMO. Manufacturer quoted URE failure rate (probability of failure to read) is overstated - instead of 1×10^14, they are mostly like 1×10^15 or higher. Full disclosure: we're actually doing erasure coding in HDFS over Raid5 on servers (double insurance - if the raid array goes down, we can recover from other servers in HDFS). But our expectation for 6x4TB arrays is not for a 70%+ chance of a URE during a rebuild, rather a couple of percent. With ZFS or btrfs, it won't actually matter for us, as we'll only lose a block on a URE- that we can recover from the rest of the cluster.
- technion 10y ago
  The problem with failures during rebuilds is overblown I thought I was the only one who believed that. I've said this on reddit before and ended on like -20 votes with people blatantly arguing I'm falsifying an "impossibility". I've got roughly 30 arrays in production, between 4 and 12 disks in each. All are RAID5 + hotspare. If you believe the maths people keep quoting, the odds of seeing a total failure in a given year is close to 100%. I started using this configuration, across varying hardware, over 15 years ago and I've been growing in number since. I'm not pretending one example proves the rule, or that it's totally safe and I would run a highly critical environment this way (before anyone comments: these environments do not meet that definition), but people have tried to show maths that there's a six nine likelihood of failure, and I just don't for a second believe I'm that lucky.
  - jamesblonde 10y ago
    Follow the money who is selling the raid5 is dead story. The main worry is correlated failures if you have the Sam types.of drives in arrays and they reach their end of life.
  - cmurf 10y ago
    Well at least on Linux, by default almost everyone (using consumer drives) has their array in a very common misconfiguration. And this leads to raid5 collapse much sooner than it should. The misconfiguration is the drive's SCT ERC timeout is greater than the kernel's SCSI command timer. So what happens on a URE is, the drive does "deep recovery" if it's a consumer drive, and keeps trying to recover that bad sector well beyond the default command timer of the kernel, which is 30 seconds. At 30 seconds the kernel assumes something's wrong and does a link reset. On SATA drives this obliterates the command queue and any other state in the drive. The drive doesn't report a read error, doesn't report what sector had the problem, and so RAID can't do its job and fix the problem by reconstructing the missing data from parity and writing the data back to that bad sector. So it's inevitable these bad sectors pop up here and there, and then if there's a single drive failure, in effect you get one or more full stripes with two or more missing strips, and now those whole stripes are lost just as if it were a 2-disk failure. It is possible to recover from this but it's really tedious and as far as I know there are no user space tools to make such recovery easy. I wouldn't be surprised if lots of NAS's using Linux were configured this way, and the user didn't use recommended drives because, FU vendor those drives are expensive, etc.
cmurf 10y ago
For scaleable storage, get rid of conventional RAID for data. I'd like to see n-way (definable) copies of metadata, and single copies of data. On top is a cluster file system like GlusterFS. When a device dies, the file system merely rebuilds metadata, and then informs GlusterFS of the missing data due to the failed drive(s). And then the file system deletes the reference to all missing/damaged files from that missing drive. No degraded state ever happens. This way Gluster knows not to even make requests from that brick. If the brick were raid56 and the cluster fs isn't aware, requests happen with degraded read/writes which suck performance wise. Plus Gluster (or even Ceph for that matter) might use some logic and say, well that data doesn't even need to be replicated on that brick, it's better for the network/use loading if the new copy is on this other brick over here.
- jamesblonde 10y ago
  Sounds like hdfs....

xioxox 10y ago

Have you tried btrfs on an md RAID device (for interest)? We've run it for a while, but we don't care too much about performance.

krondor 10y ago
That seems ill advised, one of the benefits of btrfs is it obviates the need for lvm and mdadm. I guess in your case you have a more stable raid5/6 opportunity, but you're losing many of the raid benefits present in btrfs natively. I'd also imaging it could be slower or introduce IO issues others haven't tested. Though I really have no idea, never seen anyone do that before.
- xioxox 10y ago
  It's been great for our use case - we've been using it for about 5 years like this. We use it for rsyncing data onto as a backup (not the only one!), making daily snapshots. md is more flexible than zfs's raid, but less flexible as native btrfs raid.
jamesblonde 10y ago
No, but a lot of people on the btrfs mailing list recommend this due to the lack of stability for raid5/6 on btrfs.