9 ms·

There are some modes that are horribly broken in btrfs (e.g. most of RAID except RAID1), but the common options are safe and in continuous use.

by ibotty 6y ago

nix23 6y ago
>but the common options are safe and in continuous use Which would be Raid5/6 and 1
ansible 6y ago
We've recently suffered a catastrophic failure with btrfs RAID-1 on two fileservers recently. They were both RAID-1 with two 4TB drives, stock Ubuntu 16.04. With one of the two, the filesystem was allowed to fill to 99% capacity (no automatic monitoring), so obviously that is operator error, and I'm not aware of any filesystem that can handle such situations gracefully. The system became unresponsive, with btrfs-transaction taking up an increasing percentage of CPU time. Removing files and snapshots did not see an increase in free space. So what is super-curious is that the other server, which only ever got up to about 30% full, also started exhibiting the same symptoms: unresponsive, high load from btrfs-transaction. I was able to mount the filesystems in read-only mode, and recover the files, and checked it against the offsite backup. So no data loss, only a service loss. Both systems had 10 or so subvolumes, and read-only snapshots were taken three times per day for each subvolume. After 4 years, that's close to 15000 snapshots, maybe more. I searched around, but didn't find anything particularly relevant to this issue.
- eptcyka 6y ago
  I suffered a similar breakdown with no raid. But this was about 5 years ago. The filesystem filled up, had plenty of snapshots, but removing snapshots did not actually clean up any space. Had this happen twice. So I don't think this is related to RAID. Just regular old butter filesystem. On a slightly unrelated note, I once suffered a complete failure of BTRFS where after a shutdown it just wouldn't mount anything again. Interestingly, on IRC, I was told that his is because the firmware on my Samsung NVME SSD was buggy. It might be, but ext4 has not failed me once in that regard.
  - cmurf 6y ago
    With SSD it's common that they start returning zeros or garbage, sometimes transiently, before they fail. In the case of single device Btrfs, it basically acts as an early warning detection system, and it will be a lot more sensitive to this than other file systems because both metadata (the fs itself) and your data, are checksummed. And the data is a much bigger portion of the payload on your drive. It's way more likely to be hit by hardware related issues, including even memory bit flips. An inability to mount Btrfs means metadata has been hit by some kind of corruption, and there are more structures in Btrfs that are critical. If they're hit, you see mount failure. And repair can be harder on Btrfs as well. But it also has a lot more opportunities for recovery that aren't very well understood or discussed. In part because very serious problems like this aren't that common. And also, quite a lot of people give up. Maybe they have backups and just start over with a new file system and restore. Or through no fault of their own they don't persevere with 'btrfs restore' - which is a very capable tool but requires specialized knowledge right now to use it effectively. One of the things that'll take a mindset shift is the idea of emphasizing recoveries over repairs. One improvement coming soonish (hopefully end of the year) is more tolerant read-only rescue mount option, making it possible for users to recover with normal tools rather than 'btrfs restore'.
    - eptcyka 6y ago
      That SSD seemed healthy in every other respect and it's still chugging along today just fine. On ZFS no less. The kicker in the catastrophic failure was that none of the recovery tools worked and I'd need to write my own recovery tools to fill in the gaps where the metadata got corrupted. I also did not have much time to faff about - I imaged the disk and reinstalled the machine. As for the metadata, I was under the impression that there were no checksums for metadata in BTRFS?
      - cmurf 6y ago
        Superblocks, and every node and leaf in every tree is checksummed. While it's possible to disable checksumming and copy-on-write for data, it's not possible for metadata (the fs itself).
- codys 6y ago
  My experience with btrfs is from a few years ago, so it's possible they've changed the semantics here since then, but this might be applicable: Btrfs requires a regularly run administrative task via "btrfs balance". Balancing consolidates data between partially filled block groups, which will restore consumed space even if a block group is not entirely empty. Typically, one runs balance with a set of gradually increasing parameters to control the records affected (ie: bash script that starts with block groups that are more empty and proceeds towards ones with a higher percentage utilization). In my experience with btrfs a few years ago, running a balance in some situations would result is incredibly high system wide latency. The man page [1] does not appear to reference the free space issue directly, so I'm not sure if they've removed this need. 1: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balanc...
  - ansible 6y ago
    Interesting. I would assume that this is not a factor with RAID-1, where all system, metadata, and data is duplicated. I could see this as being very important for the higher RAID levels. We have since stopped deploying any sort of btrfs RAID (even RAID-1), and have gone back to using Linux MD.
    - LanternLight83 6y ago
      Free space and the space taken up by metadata and snapshots can take a little attention to figure out https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/ https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-sp...
      - ansible 6y ago
        Now that is interesting. If I could mount the failed btrfs filesystem read-write for an appreciable length of time, I could try the balance operation. I'll need to set up a cron job on the other systems to run balance on a regular basis alongside scrub (obviously not at the exact same time).
    - 6y ago

There are some modes that are horribly broken in btrfs (e.g. most of RAID except RAID1), but the common options are safe and in continuous use.

by ibotty 6y ago

There are some modes that are horribly broken in btrfs (e.g. most of RAID except RAID1), but the common options are safe and in continuous use.

nix23 6y ago

>but the common options are safe and in continuous use Which would be Raid5/6 and 1

ansible 6y ago

We've recently suffered a catastrophic failure with btrfs RAID-1 on two fileservers recently. They were both RAID-1 with two 4TB drives, stock Ubuntu 16.04. With one of the two, the filesystem was allowed to fill to 99% capacity (no automatic monitoring), so obviously that is operator error, and I'm not aware of any filesystem that can handle such situations gracefully. The system became unresponsive, with btrfs-transaction taking up an increasing percentage of CPU time. Removing files and snapshots did not see an increase in free space. So what is super-curious is that the other server, which only ever got up to about 30% full, also started exhibiting the same symptoms: unresponsive, high load from btrfs-transaction. I was able to mount the filesystems in read-only mode, and recover the files, and checked it against the offsite backup. So no data loss, only a service loss. Both systems had 10 or so subvolumes, and read-only snapshots were taken three times per day for each subvolume. After 4 years, that's close to 15000 snapshots, maybe more. I searched around, but didn't find anything particularly relevant to this issue.

eptcyka 6y ago
I suffered a similar breakdown with no raid. But this was about 5 years ago. The filesystem filled up, had plenty of snapshots, but removing snapshots did not actually clean up any space. Had this happen twice. So I don't think this is related to RAID. Just regular old butter filesystem. On a slightly unrelated note, I once suffered a complete failure of BTRFS where after a shutdown it just wouldn't mount anything again. Interestingly, on IRC, I was told that his is because the firmware on my Samsung NVME SSD was buggy. It might be, but ext4 has not failed me once in that regard.
- cmurf 6y ago
  With SSD it's common that they start returning zeros or garbage, sometimes transiently, before they fail. In the case of single device Btrfs, it basically acts as an early warning detection system, and it will be a lot more sensitive to this than other file systems because both metadata (the fs itself) and your data, are checksummed. And the data is a much bigger portion of the payload on your drive. It's way more likely to be hit by hardware related issues, including even memory bit flips. An inability to mount Btrfs means metadata has been hit by some kind of corruption, and there are more structures in Btrfs that are critical. If they're hit, you see mount failure. And repair can be harder on Btrfs as well. But it also has a lot more opportunities for recovery that aren't very well understood or discussed. In part because very serious problems like this aren't that common. And also, quite a lot of people give up. Maybe they have backups and just start over with a new file system and restore. Or through no fault of their own they don't persevere with 'btrfs restore' - which is a very capable tool but requires specialized knowledge right now to use it effectively. One of the things that'll take a mindset shift is the idea of emphasizing recoveries over repairs. One improvement coming soonish (hopefully end of the year) is more tolerant read-only rescue mount option, making it possible for users to recover with normal tools rather than 'btrfs restore'.
  - eptcyka 6y ago
    That SSD seemed healthy in every other respect and it's still chugging along today just fine. On ZFS no less. The kicker in the catastrophic failure was that none of the recovery tools worked and I'd need to write my own recovery tools to fill in the gaps where the metadata got corrupted. I also did not have much time to faff about - I imaged the disk and reinstalled the machine. As for the metadata, I was under the impression that there were no checksums for metadata in BTRFS?
    - cmurf 6y ago
      Superblocks, and every node and leaf in every tree is checksummed. While it's possible to disable checksumming and copy-on-write for data, it's not possible for metadata (the fs itself).
codys 6y ago
My experience with btrfs is from a few years ago, so it's possible they've changed the semantics here since then, but this might be applicable: Btrfs requires a regularly run administrative task via "btrfs balance". Balancing consolidates data between partially filled block groups, which will restore consumed space even if a block group is not entirely empty. Typically, one runs balance with a set of gradually increasing parameters to control the records affected (ie: bash script that starts with block groups that are more empty and proceeds towards ones with a higher percentage utilization). In my experience with btrfs a few years ago, running a balance in some situations would result is incredibly high system wide latency. The man page [1] does not appear to reference the free space issue directly, so I'm not sure if they've removed this need. 1: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balance https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-balanc...
- ansible 6y ago
  Interesting. I would assume that this is not a factor with RAID-1, where all system, metadata, and data is duplicated. I could see this as being very important for the higher RAID levels. We have since stopped deploying any sort of btrfs RAID (even RAID-1), and have gone back to using Linux MD.
  - LanternLight83 6y ago
    Free space and the space taken up by metadata and snapshots can take a little attention to figure out https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/ https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-sp...
    - ansible 6y ago
      Now that is interesting. If I could mount the failed btrfs filesystem read-write for an appreciable length of time, I could try the balance operation. I'll need to set up a cron job on the other systems to run balance on a regular basis alongside scrub (obviously not at the exact same time).
  - 6y ago