4 ms·
> "Here's a fact: the upstream's official statement is that RAID 1 isn't production-ready." That doesn't seem to match anything I've seen. The btrfs status pag
by wtallis 9y ago
> "Here's a fact: the upstream's official statement is that RAID 1 isn't production-ready."
That doesn't seem to match anything I've seen. The btrfs status page classifies RAID1 and RAID10 features as "mostly OK". The one and only documented caveat is that when a disk in a RAID1 fails, you should mount the filesystem as read-only until you are ready to fix the problem by either replacing the failed disk or converting to a non-RAID profile. There's a real bug underlying this limitation, but the only way to encounter this bug is to be quite cavalier about how you handle a degraded array, by doing something that you shouldn't expect to be safe.
- rodgerd 9y ago> but the only way to encounter this bug is to be quite cavalier about how you handle a degraded array, by doing something that you shouldn't expect to be safe. The way to encounter it is to mount the array a second time. That's hardly "cavalier handling". A decade in, and RAID 1 doesn't work right. Saying this clearly offends the feels here at HN, but it's a fact.
- wtallis 9y ago> The way to encounter it is to mount the array a second time. That's hardly "cavalier handling". `mount` is not the same as `mount -o degraded,rw`. The latter should raise the eyebrows of anyone paying attention to what they're typing. Odds are that you'll have to consult the docs to even find these mount options, because it doesn't happen automatically. This is where any careful, sane user who's concerned about their data would spend a few more minutes thinking through their entire recovery procedure. There's one corner case to RAID1 recovery where the current tooling does not fully prevent a careless user from putting the FS in a bad state. This is not the same as "RAID1 doesn't work right".
- mioelnir 9y agoI don't think a careful, sane user who's concerned about their data would be on btrfs in the first place.
- wtallis 9y agoNo, probably not. But this particular bug isn't the reason why users who need very high level assurances should avoid btrfs for now. The nature of this bug does not does lend itself to using it as an argument that btrfs is unsafe in general.
- mulle_nat 9y agoConsider that most users will be careless, because running a filesystem to them is as exiting as any of the other dozen OS services. Having a corner case like this in RAID1 to me is like having a corner case in encryption... That I immediately run into this case on my first btrfs RAID1 problem, makes me think that there is a steep slope downwards to this corner.
- wtallis 9y agoIf when a drive in your RAID fails, your first response is anything but checking the docs for the recovery procedure, you're going to end up disappointed sooner or later. When your fault-tolerant system tells you something broke leaving it in a fragile state and your response is to tell the system to ignore that and pretend everything is normal, expect trouble. You're working in a domain where there is no one right answer, so the system cannot magically anticipate how to handle the exceptional situation. None of the above is the fault of btrfs. None of this is avoidable. This problem has to be faced by ZFS, too. What is avoidable is that btrfs could make it harder to do the equivalent of `mount -o degraded,rw`. But ultimately, there will be some mechanism for modifying a degraded array, and it'll get documented and then excerpted in blog posts and StackOverflow answers without all the context, and users will find a way to work themselves into a corner. There are all kinds of ways to do this with ZFS, too. ZFS tends to default to the approach of requiring you to copy all your data elsewhere and rebuild your array from scratch. What btrfs is doing here is no worse, except that it's a bit less up-front about the limitation because it's actually completely avoidable and this is a fixable UI bug, not a deep-seated architectural limitation.
- michaelmrose 9y agoMostly OK != production ready. A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily. I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement. Another reason is in a non enterprise use case a desktop users might not instantly have a disk available and might mistakenly believe that this is no more risky than having only one disk in the first place. This is ridiculously flaky for something that is supposed to be years in development with backing from major companies. This confirms my feeling that btrfs belongs in the waste basket.
- wtallis 9y ago> Mostly OK != production ready. Don't nitpick about what a two-word summary implies when there's only one underlying bug at issue. Just address the bug itself. "Production ready" never means 100% bug-free. > A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily. If you are comfortable dropping from the RAID1 profile to a non-RAID profile, btrfs requires you to explicitly make this conversion; there's no safe way to make it automatic. Forcing the filesystem to accept writes while it's still in RAID1 mode but is incapable of providing RAID1 data integrity is something that you should expect to cause problems. > I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement. You can mount read-only as many times as you want if you're going to copy the remaining data to a different filesystem. If you want to do an in-place recovery, the current limitation is only that you shouldn't mount the degraded array as writable until you're ready to make it not degraded anymore.
- mulle_nat 9y agoRan into this with my RAID1: https://www.spinics.net/lists/linux-btrfs/msg62346.html https://www.spinics.net/lists/linux-btrfs/msg62346.html. I then ditched btrfs, and I'll never try it out again.
- wtallis 9y ago> I wish that if a drive fail, the btrfs filesystem still mounts rw and leave the OS running, but warns the user of a failing disk and easily allow the addition of a new drive to reintroduce redundancy. This might make sense for you, but is insane as a default policy. Manual intervention should be required before the FS will accept writes with less than the configured degree of redundancy. Silently mounting and hoping the user notices something in their logs is too dangerous. > I created a raid1 btrfs filesystem by converting an existing single btrfs instance into a degraded raid1, then added the other drive This seems backwards. Why not add the second drive, then convert to RAID1? - Note that the patch to work around the refusal to mount is extremely simple, and that the patch is quite safe if used properly. But it's not really an acceptable solution for upstreaming, because it will lead to bigger problems if used in more complicated situations. There are several potential solutions that would be safe and widely deployable, but all involve changing far more than two lines of code.
- pas 9y agoAgreed, that default policy should be manual intervention, but currently I don't see how I can set a different policy. For example in a large cluster where that degradation is expected and monitored for. How do you add a second drive? Is there a brtfs raid faq/howto?
- wtallis 9y agoIn a large cluster, you'd probably plan on replacing every drive that failed rather than reconfigure to use less redundancy. So in that case, you'd probably want the hot spare feature to be stabilized and upstreamed. Then the FS could automatically copy over (or reconstruct from parity) data for a missing disk, without modifying data on the surviving disks. In environments with a smaller budget that want to get the system back up and running before a replacement drive is available, it could be valuable to be able to pre-specify that the system should rebalance with less redundancy when a drive goes missing. I'm not aware of any work to implement this kind of feature. No enterprise customer would want or use this feature, and even a home user on a shoestring budget wouldn't necessarily want this rebalancing to happen automatically. (What if the drive was only temporarily missing, such as from a failed or loose SATA cable? You wouldn't want to do a ton of re-writing of data only to have to reverse it on the next boot.)
- linuxready 9y agoDo you have a link to this real bug ? It would be interesting to follow it until it is resolved.
- wtallis 9y agoThere are plenty of instances of this bug being brought up on the mailing list. One of them is already linked elsewhere in this discussion, and the btrfs status page (also linked from this discussion) has further mailing list links. Basically, btrfs doesn't want to allow a writeable mount when it might be missing some data. If there's some data on the FS that isn't stored with the RAID1 profile, then the kernel can't safely assume that the missing drive didn't have more chunks like that, holding data that wasn't mirrored on one of the surviving drives. But it's currently not possible to convert from RAID1 to non-RAID or to rebuild the array with a replacement without mounting the degraded array as writeable, which leads to non-RAID data being written. That puts the FS in a state that cannot be automatically judged safe at mount time, and the FS remains in that state until the recovery is complete (either converting from RAID1 to non-RAID, or replacing the failed drive). There's no easy way to require the user to specify at the time of the `mount -o degraded,rw` whether they intend to resolve the situation by ceasing to use RAID1 or by replacing the failed drive. That leaves users with the opportunity to do neither and instead make the situation worse.
- linuxready 9y agoThanks for the explanation. I was hoping for a Github issue number (or Bugzilla or whatever) to easily track this bug, but perhaps the Btrfs dev team doesn't work with issue number ? At least for RAID1, it seems that implementing RAID1 N-way mirroring would ease the process to recover from a failed drive. In case of drive failure, we could use the remaining drive in read-only mode to copy the data to a new drive, hence creating a RAID1 array with two working drives and one failed drive. The OS should then allow to boot in rw mode, and from there it is easy to remove the failed drive from the RAID1 array. However it seems that RAID1 N-way mirroring (with N > 2) is not even on the roadmap at this moment. Have I misunderstood something or does this approach make sense ?