6 ms·

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://ars

• klysm 4y ago
  > * It won't boot on a degraded array by default, requiring manual action to mount it That by itself is a complete deal breaker.
  • chomp 4y ago
    Why? Are you in a position where you can’t modify grub or fstab?
    • dale_glass 4y ago
      Because RAID is a High Availability technology. It's not a backup. It doesn't protect against mistakes, bugs, or the server failing in some other way. What it's good for is for ensuring that work keeps happening if a disk fails. And disk failures happen to be fairly frequent, since spinning rust is a rather delicate technology. If somebody has to connect to the machine and fiddle with it by hand it means that the system has been down, possibly for hours, when that was the exact thing you were trying to prevent by setting up RAID on it.
      • viraptor 4y ago
        You don't have to modify it after a failure. You can set it right now to explicitly say "when I end up in a degraded state, I want to boot anyway". > What it's good for is for ensuring that work keeps happening if a disk fails. Different services have different priorities. I want my work servers to keep running, but my home server under the desk to stop until I replace the failed drive. Btrfs gives you a choice.
    • alerighi 4y ago
      No, because I want to be elsewhere on a Sunday evening than in front of a computer modifying fstab to bring a production server back online.
      • RealStickman_ 4y ago
        Unless you want to reboot the server this shouldn't affect you in any way.
        • ndsipa_pomu 4y ago
          Unless you get a power outage or some other event that causes a reboot (e.g. STONITH)
  • josephcsible 4y ago
    IMO, the inconvenience this causes is outweighed by the prevention of data loss for people who otherwise would never have noticed the degraded array.
    • ClumsyPilot 4y ago
      I think this is the difference in target audience - for home use, maybe I won't have monitoring, so it should fail to boot. For enterprise, if you dont have monitoring setup it's on you
    • ilyt 4y ago
      Right but if it's remote server you're fucked and you can't login on it to even assess what happened
      • kiririn 4y ago
        That’s what the likes of dropbear-initramfs is for
      • josephcsible 4y ago
        Don't servers tend to include some kind of out-of-band management like HP iLO, Dell iDRAC, or OpenBMC?
      • ndsipa_pomu 4y ago
        This is why I want servers to always boot to an OS whenever possible. It's fine if production data is not available until someone fixes an array, but without the OS, it's a pain to even figure out what's happened, let alone fix it.
  • cycomanic 4y ago
    Hold on, people make this out to be like if a disk fails it requires immediate intervention. However if the system already booted it stays up. Who reboots a production system without monitoring it? Also if you boot with a degraded disk are you not asking for massive trouble, because if another disk fails you might end up with data loss which is IMO much worse on a production system than not being able to boot until you add another disk.
    • justsomehnguy 4y ago
      > Who reboots a production system without monitoring it? Anyone. > Also if you boot with a degraded disk are you not asking for massive trouble > not being able to boot until you add another disk Great. I'm just a [sys]admin who were given a task to do something on $server. I jump around the red tape, claw out a 15 minute downtime because the task requires reboot, proceed with all that corporate dance with notification emails. I do my thing, reboot the server and it doesn't return back online. Suddenly I broke the server, missed maintenance window, amount of mails with CC and RE: in my mailbox grows in geometric progression and the most important - now I need to find out who were responsible for the server, contact him and [kick his ass] ask him to diagnose what is going on. Bonus points if: server doesn't have a meaningful BMC/iLO/iDRAC with KVM console it does have it but it's broken for some reason eg requires Java 6 on Vista server was configured 10 years ago by a greybeard who is not only retired but already died from the old age server is 6000km away from any place with a replacement disks and the earliest time when you can send the replacement would be the next spring when the ice would break and taw enough for the ships to move. Of course you can hire a helo to deliver it, which get your ass chewed for an unplanned $50k expenses Also: https://news.ycombinator.com/item?id=33963000 https://news.ycombinator.com/item?id=33963000
      • wtallis 4y ago
        It sounds like you're hypothesizing a long chain of bad decisions, and then ridiculing btrfs for taking the choice that means your next bad decision only culminates in (predictable, preventable) downtime rather than data loss.
  • happymellon 4y ago
    Unless you give it permission. It's an option to enable. If you needing to configure something is a dealbreaker then having RAID is probably not for you anyway.
• diegocg 4y ago
  No, they have plans for a new different RAID implementation as part of the "extent tree v2" thing AFAIK
• dekhn 4y ago
  Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.
  • dale_glass 4y ago
    That seems like a very hacky way to accomplish something that should just work. I mean, it's better than nothing, but it seems far from ideal. If it's your own personal box, and you're okay with that fiddling, fine. But if people are supposed to get work done while the admin gets a new disk, it's not great.
    • mlyle 4y ago
      There's pretty solid reasons why you might choose to have a very simple RAID1 set for a boot volume-- because you want to be sure to boot even if the huge array is degraded, and because you want faster I/O for the OS install, etc...
  • simcop2387 4y ago
    The way I like to do it is with mirroring of the nvme storage and then lvm snapshots on top of it. Done correctly you get boot off either of them but they're kept in sync automagically by the OS. then you snapshot the boot volume regularly and copy those to some kind of archival storage for backups. That lets you then even hotswap them if you need to while keeping the root filesystem workable.
  • antod 4y ago
    The way I used to do it (back then getting grub/initrd raid aware wasn't really doable - not sure about now) with mdadm was to use RAID1 on the boot volume such that a disk from a broken mirror was still usable by itself. Other volumes were RAID 5 or 10 etc.
    • koolba 4y ago
• _kdave 4y ago
  > It won't boot on a degraded array by default, requiring manual action to mount it If you want it to behave like that then add 'degraded' to fstab. That a device is missing can have unknown reasons, the user should know better and resolve it or allow such boot. It's not automatic as there's no way to inform the user that it's degraded state.
  • dale_glass 4y ago
    I don't quite understand the use case here. If I'm setting up RAID it's because I want the system to stay up. That's the only purpose for it. If a device goes missing for "unknown reasons", then the machine should still work, and I'll figure out what happened when monitoring pokes me and says RAID is degraded.
    • viraptor 4y ago
      The use case is: Enough drives failed that your raid is degraded. Any more data you write is not replicated and it may be due to software/hardware issue that will kill more drives soon. It's up to you to choose at that point - is availability more important for you (add degraded to fstab), or data consistency (deal with the array first).
    • _kdave 4y ago
      Yeah but monitoring is not something that comes with the filesystem. If you have to set up the system to be a HA and configure monitoring, email notifications whatever, making sure the filesystem is created with redundant profiles, then I'm expecting that also adding the 'degraded' to the fstab is part of the configuration.
    • p1necone 4y ago
      > That's the only purpose for it. That's not the only purpose for it. There's three reasons I can think of that you might set up a RAID array: * You want better uptime. (your use case) * You want to protect from data loss. (my assumption was that this is the most common use case, but I could be wrong. This also helps with uptime because there's nothing worse for uptime than having to restore lost data from a cold backup) * You want better performance, data integrity be damned. (RAID 0) Booting a RAID array with a failed disk is a bad idea if you care a lot about not losing data, because now you're one less disk failure away.
• chasil 4y ago
  Regarding resilvering, I am amazed by ZFS' capability of quickly bringing a stale mirror back into synchronization. I have used the FUSE port of ZFS to write only one member of a mirroset, then upon mounting both members elsewhere, the stale mirror was very quickly resilvered, so ZFS was able to determine only the blocks needing to be refreshed. In btrfs, I understand that this requires a rebalance, which will read/write every used portion of the filesystem.
  • patmorgan23 4y ago
    Yeah ZFS does some sort of block level hash table magic. Very IO efficient for what it's doing.
  • cmurf 4y ago
    Scrub, not balance. It's primarily a read operation. The on-disk format has all the information needed to do an automatic abbreviated scrub, but the feature isn't implemented yet.
    • chasil 4y ago
      No, I've heard that a scrub will not reallocate missing blocks on mirrors. A rebalance was required in the article below, and this situation really requires attention. ZFS is far friendlier in a crisis. "We'll even manually trigger a scrub—a procedure that storage admins generally understand to look for and automatically repair any data issues... even though we manually initiated a scrub and let it finish, our array is still inconsistent and even outright non-mountable, because it ran for a little while without a disk and then that disk was re-added. The command that we were supposed to run was btrfs balance—with both drives connected and a btrfs balance run, it does correct the missing blocks, and we can now mount degraded on only the other disk, /dev/vdc. But this was a very tortuous path, with a lot of potential for missteps and zero discoverability." https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
      • cmurf 4y ago
• jiggawatts 4y ago
  The list of bugs and counter-intuitive design decisions in BTRFS RAID should make anyone pause before ever considering it as a viable filesystem for anything. It's like people justifying MySQL saying that it's okay for it to lose or corrupt data, and that transactional integrity doesn't matter as much as people say it does. Yeah, maybe if you're website is a blog. But anyone storing real data should run away screaming from systems like this. Maybe MySQL is sort-of-okay now? I dunno. It's possible BTRFS RAID 5 won't eat your data or crash your server regularly now. I'm going to stay away from both anyway.
  • happymellon 4y ago
    I've had more issues with my ZFS arrays than the BTRFS ones. The Linux implementation is pretty awful, with it demanding that it uses the absolute `/dev/sdx` reference even if you try to build it using serials or other unchaging reference. Replace a disk and then after the next restart it fails to bring up the RAID because `sdf` points to a different disk. At least BTRFS uses internal UUIDs so can bring up the arrays when the disk ids change. Fairly fundamental stuff, rather than complaining about an option not being enabled by default.
    • wokkel 4y ago
      Uh, using /dev/disk/by-id to assemble a zfs array is pretty standard for zol. I never use device names and shuffling disks has never required any manual intervention on my part. Was this a very old ZoL?
      • vetinari 4y ago
        At a time, it was the default, and you had to go out of your way to use /dev/disk/by-id (you had to create your pool, export it, and then import it by id), only to be hit by another issue, where when preparing grub, it was looking for the IDs straight in /dev (that's how I learned that ZPOOL_VDEV_NAME_PATH=YES exists). Yeah, putting / on zfs was not smart. Nowadays, when the distro doesn't support it out of the box and doesn't come with kernel+bootloader+zfs tested together (i.e. proxmox, ubuntu), I wouldn't do it.
• cmurf 4y ago
  Automatic degraded mount at boot time isn't implemented by md either, it's implemented by a dracut module (or whatever creates the initramfs). The gist is that when mdadm assembly fails, the dracut module's script performs a 5 minute wait to see if all devices will appear, and then if not, tries to assemble degraded. These days, with UEFI prevalent, a system won't boot without EFI system partitions replicated (and possibly sync'd) on every drive.
  • ndsipa_pomu 4y ago
    You can boot from a MD RAID1 system that's degraded easily enough. Here's some info on how to implement it https://outflux.net/blog/archives/2018/04/19/uefi-booting-and-raid1/ https://outflux.net/blog/archives/2018/04/19/uefi-booting-an...
• xorcist 4y ago
  Remember that DM is still there for you even if you use btrfs. Every discussion about btrfs immediately escalates to how the btrfs RAID support develops. But btrfs is useful on its own without that mode. Should you wish to put a btrfs file system on software RAID (DM), you can still do that. The btrfs RAID modes are something else, where you cut the DM driver out of the picture. That's not going to be as well tested as plain old DM, even if it works, and it doesn't offer any functionality beyond plain DM RAID. If the use case is storing real data, you want the data storage layer to be as well tested as possible. There's also more to software RAID than the block device. You want a mature resilvering daemon on a well tested schedule, and you need working monitoring to alert you of issues. RAID without monitoring will only delay the inevitable.
  • exceptione 4y ago
    DM = device mapper wikipedia: The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the logical volume manager (LVM), software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots.[1] Device mapper works by passing data from a virtual block device, which is provided by the device mapper itself, to another block device. Data can be also modified in transition, which is performed, for example, in the case of device mapper providing disk encryption or simulation of unreliable hardware behavior.
  • matheusmoreira 4y ago
    Yeah but DM doesn't have the btrfs flexible block allocation scheme that allows me to expand capacity one drive at a time and with drives of different capacities. This is very important for a home user like me. To expand a typical RAID setup, you have to essentially fail every single drive in the array and replace them one by one with new identical drives. Expanding an array of N drives requires N resilvers which not only takes a long time but increases the chance of actually failing those drives due to massive I/O load. It's easier to build a second array and copy the data over. This combined with the fact you have to buy all the hardware up front makes it cost prohibitive for home users. You can't buy drives one at a time and expand capacity as needed.