7 ms·

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built

by RX14 7y ago

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible cause is distro choice tends to be per person and how up to date said distro keeps it's kernel. But, I'm not sure.

thfuran 7y ago
I think the probable cause is that it's not common bugs that cause the corruption but uncommon ones. Most of the time, they work fine. But you really want a stronger guarantee than that out of your filesystem.
- SEJeff 7y ago
  Historically, the biggest bugs in btrfs were when you came close to filling up the filesystem. For the longest time, you'd get -ENOSPC (no space left) even when you had many Gb of space left due to really bad metadata and block level space usage.
  - blattimwind 7y ago
    Let's not forget about various performance issues which were exacerbated by "low free space" conditions (i.e. after you filled the volume beyond 80 % these started to pop up). A file system that will sometimes go down well into the fractional IOPS range is not very useful. Some of these are fixed by now, though.
    - SEJeff 7y ago
      "the biggest bugs in btrfs were when you came close to filling up the filesystem" :) I used to read every email on btrfs-devel for a year or so.
  - machawinka 7y ago
    This is my experinence too. Works great with lots of free space, as soon as space gets tight, performance deteriorates really fast. Nevertheless, for me it has been worthy.
  - danudey 7y ago
    I'm a huge Mac fanboy, but APFS really kicks me in the teeth sometimes. Aside from things like snapshots, clones, etc. not being accessible to users (well, not really), or being able to create subvolumes at specific mount points which forget those mount points next reboot, it had an extremely strange behavior (possibly relating to snapshots/CoW?) where once it was full, it stayed full forever until you rebooted. Basically, any time a runaway process filled my disk, I just had to hard-reboot and hope I didn't have any unsaved work or state that I needed to preserve. Really makes me hope that Apple is going to further extend APFS to not just be baby's first CoW volume-management filesystem.
    - scottlamb 7y ago
      > it had an extremely strange behavior (possibly relating to snapshots/CoW?) where once it was full, it stayed full forever until you rebooted. Do you have Time Machine enabled? I think it uses snapshots, which explains why the filesystem stays full. I've hit this myself and was initially surprised to see rm not improving matters (possibly even making it worse) but it makes sense with snapshots. The working on reboot was a surprise. I'd put off fixing the machine for at least a week, and when I went to actually fix it, it was quite anticlimatic to just reboot and have it work. Maybe it checks for this condition on reboot and dumps Time Machine snapshots if so. That was the less scary part of my macOS filesystem integrity worries. My full disk started when it was staging a full Time Machine backup after I got a dialog saying: > Time Machine completed a verification of your backups on "my.nas.address". To improve reliability, Time Machine must create a new backup for you. ...for the Nth time. I don't know for certain if the problem is with Apple's software or with my NAS's (Synology) but these backups are clearly not as reliable as one would hope...
- zepearl 7y ago
  Personally I think that in the case of a CoW filesystem, bugs which cause corruption should be very uncommon because of the very nature of the CoW mechanism, especially if coupled together with data checksums as publicized in the case of BTRFS. If things still get trashed then I tend to think that the very foundation of the FS is bad. But maybe I'm just naive :)
- kbenson 7y ago
  There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE
dmos62 7y ago
> btrfs loves you or hates you This is how superstitious traditions start, and ritualistic sacrifice in particular, I'd think.
- rnd0 7y ago
  >and ritualistic sacrifice in particular, I'd think. Does data loss count as a sacrifice in this instance?
  - wtallis 7y ago
    If it does, I think the ZFS "rebuild the pool from scratch" should as well, since that seems far more ritualistic.
BubRoss 7y ago
One anecdote of a filesystem working fine and one anecdote of it becoming a disaster don't cancel each other out. I wouldn't buy a $5 USB thumb drive if half the people said it lost their data and half said it worked fine.
- rnd0 7y ago
  I'd buy it -but only for short-term use to sneakernet shit I already had backed up reliably somewhere else. of course, where we run into problems is that btrfs is meant to be the reliable backup. Oops.
  - BubRoss 7y ago
    You realize that there are $5 thumb drives that work, just like there are filesystems that actually work right? There isn't any benefit to using something broken, these problems have been solved.
unixhero 7y ago
Same and same. Never saw any problems with btrfs. Really like the memory consumption of btrfs!
josteink 7y ago
> I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. I tried, I really tried to like btrfs. On the servers/workstations I’ve had few serious issues, but a few “gotchas” you need to know to keep things running smoothly. On every laptop I’ve had, I’ve had btrfs fail on me. Repeatedly. So I gave up on it. ZFS for me these days.
packetlost 7y ago
In my experience, btrfs is very fragile in power loss or kernel crash/panic scenarios. It very consistently causes soft lockups on file read/writes after power loss until you run a `brtfs check --repair` on it. My experience is mostly on Arch, so it's not a case where it's out of date and missing patches.
- danudey 7y ago
  This was my experience. We had a brief power outage at work and my btrfs (root) partition was toast. Spent a whole day rebuilding my system afterwards. Will definitely not go that route again. The only difference is that none of the repair tools were able to recover the filesystem, but I was able to dump the files themselves to a new disk to recover them. Really not sure why, it was very strange.
- cmurf 7y ago
  Sounds like hardware problems in the storage stack. Btrfs developers contributed the dm-log-writes target to the kernel, expressly for conducting power loss tests on file systems. All the file systems benefit from this work. https://www.kernel.org/doc/Documentation/device-mapper/log-writes.txt https://www.kernel.org/doc/Documentation/device-mapper/log-w... And Btrfs is doing the right thing these days. I recently conducted system resource starvation tests where a compile process spun off enough threads to soak the system to the point it becomes unresponsive. I did over 100 forced power off tests while the Btrfs file system was being written to as part of that compile. Zero complaints: not on mount, not on scrubs, not with btrfs check, and not any while in normal operation following those power offs. If you want to complain about Btrfs, complain about the man page warning to not use --repair without advice from a developer. You did know about that warning, right?
  - packetlost 7y ago
    100% was not a hardware problem. Works fine on other filesystems ️
    - cmurf 7y ago
paulddraper 7y ago
> an effect where btrfs loves you or hates you Same thing happens with operating systems.
scottlamb 7y ago
> It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). Why wouldn't you expect it to survive that? Is there a particular reason to believe those drives are broken? I.e., are they older consumer drives known to lie about cache flushes? do they have bad sectors? How have you abused it? What kind of load? Did you fill the filesystem (which another commenter mentioned seems to be a common element of most sad btrfs stories)? did your system frequently lose power while under write load? Lacking more details, I'd just say one user experiencing 0 bugs in 5 years should be completely unremarkable. I expect filesystems to be very reliable, so a lot of people having stories of corruption means stay away from btrfs. Having some people with stories of no corruption doesn't really move the needle. Together, these stories still mean stay away from btrfs!
- cmurf 7y ago
  That's hyperbole, it can't be taken seriously. OpenSUSE uses Btrfs by default, if there were more problems outside what's expected by md+LVM+ext4 (or XFS), which is the feature comprised by Btrfs and then some, they wouldn't have made the on-going investments they have. Facebook has been using it in production with thousands of installations for years. You want details from people experiencing zero problems, but you don't ask for details from people who are? That's a weird way to go about conducting the necessary autopsies, to discover and fix bugs. Anyway, I monitor the upstream filesystems lists, and they all have bugs. They're all fixing bugs. They're all adding new features. And that introduces bugs that need fixing. It's not that remarkable, until of course someone suggests only one file system is to be avoided, while also providing no details, but depends on conjecture.
  - scottlamb 7y ago
    I asked RX14 why they called out their lack of problems as remarkable ("survive the most fucked up situations"). It sounds strange, as I mentioned. I don't need to ask people who've had problems because I've had them myself, in unremarkable circumstances, a while back. I'm sure I could find reports on the mailing list as well, in which others have already asked for details.
awill 7y ago
>but there seems to be an effect where btrfs loves you or hates you Surely it depends on the btrfs implementation. e.g. Arch Linux getting daily kernel updates vs an enterprise distro
- packetlost 7y ago
  Just as unstable on Arch as of a month or two ago.

by RX14 7y ago

thfuran 7y ago

I think the probable cause is that it's not common bugs that cause the corruption but uncommon ones. Most of the time, they work fine. But you really want a stronger guarantee than that out of your filesystem.

SEJeff 7y ago
Historically, the biggest bugs in btrfs were when you came close to filling up the filesystem. For the longest time, you'd get -ENOSPC (no space left) even when you had many Gb of space left due to really bad metadata and block level space usage.
- blattimwind 7y ago
  Let's not forget about various performance issues which were exacerbated by "low free space" conditions (i.e. after you filled the volume beyond 80 % these started to pop up). A file system that will sometimes go down well into the fractional IOPS range is not very useful. Some of these are fixed by now, though.
  - SEJeff 7y ago
    "the biggest bugs in btrfs were when you came close to filling up the filesystem" :) I used to read every email on btrfs-devel for a year or so.
- machawinka 7y ago
  This is my experinence too. Works great with lots of free space, as soon as space gets tight, performance deteriorates really fast. Nevertheless, for me it has been worthy.
- danudey 7y ago
  I'm a huge Mac fanboy, but APFS really kicks me in the teeth sometimes. Aside from things like snapshots, clones, etc. not being accessible to users (well, not really), or being able to create subvolumes at specific mount points which forget those mount points next reboot, it had an extremely strange behavior (possibly relating to snapshots/CoW?) where once it was full, it stayed full forever until you rebooted. Basically, any time a runaway process filled my disk, I just had to hard-reboot and hope I didn't have any unsaved work or state that I needed to preserve. Really makes me hope that Apple is going to further extend APFS to not just be baby's first CoW volume-management filesystem.
  - scottlamb 7y ago
    > it had an extremely strange behavior (possibly relating to snapshots/CoW?) where once it was full, it stayed full forever until you rebooted. Do you have Time Machine enabled? I think it uses snapshots, which explains why the filesystem stays full. I've hit this myself and was initially surprised to see rm not improving matters (possibly even making it worse) but it makes sense with snapshots. The working on reboot was a surprise. I'd put off fixing the machine for at least a week, and when I went to actually fix it, it was quite anticlimatic to just reboot and have it work. Maybe it checks for this condition on reboot and dumps Time Machine snapshots if so. That was the less scary part of my macOS filesystem integrity worries. My full disk started when it was staging a full Time Machine backup after I got a dialog saying: > Time Machine completed a verification of your backups on "my.nas.address". To improve reliability, Time Machine must create a new backup for you. ...for the Nth time. I don't know for certain if the problem is with Apple's software or with my NAS's (Synology) but these backups are clearly not as reliable as one would hope...
zepearl 7y ago
Personally I think that in the case of a CoW filesystem, bugs which cause corruption should be very uncommon because of the very nature of the CoW mechanism, especially if coupled together with data checksums as publicized in the case of BTRFS. If things still get trashed then I tend to think that the very foundation of the FS is bad. But maybe I'm just naive :)
kbenson 7y ago
There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: https://www.youtube.com/watch?v=fE2KDzZaxvE https://www.youtube.com/watch?v=fE2KDzZaxvE

dmos62 7y ago

> btrfs loves you or hates you This is how superstitious traditions start, and ritualistic sacrifice in particular, I'd think.

rnd0 7y ago
>and ritualistic sacrifice in particular, I'd think. Does data loss count as a sacrifice in this instance?
- wtallis 7y ago
  If it does, I think the ZFS "rebuild the pool from scratch" should as well, since that seems far more ritualistic.

BubRoss 7y ago

One anecdote of a filesystem working fine and one anecdote of it becoming a disaster don't cancel each other out. I wouldn't buy a $5 USB thumb drive if half the people said it lost their data and half said it worked fine.

rnd0 7y ago
I'd buy it -but only for short-term use to sneakernet shit I already had backed up reliably somewhere else. of course, where we run into problems is that btrfs is meant to be the reliable backup. Oops.
- BubRoss 7y ago
  You realize that there are $5 thumb drives that work, just like there are filesystems that actually work right? There isn't any benefit to using something broken, these problems have been solved.

unixhero 7y ago

Same and same. Never saw any problems with btrfs. Really like the memory consumption of btrfs!

josteink 7y ago

> I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. I tried, I really tried to like btrfs. On the servers/workstations I’ve had few serious issues, but a few “gotchas” you need to know to keep things running smoothly. On every laptop I’ve had, I’ve had btrfs fail on me. Repeatedly. So I gave up on it. ZFS for me these days.

packetlost 7y ago

In my experience, btrfs is very fragile in power loss or kernel crash/panic scenarios. It very consistently causes soft lockups on file read/writes after power loss until you run a `brtfs check --repair` on it. My experience is mostly on Arch, so it's not a case where it's out of date and missing patches.

danudey 7y ago
This was my experience. We had a brief power outage at work and my btrfs (root) partition was toast. Spent a whole day rebuilding my system afterwards. Will definitely not go that route again. The only difference is that none of the repair tools were able to recover the filesystem, but I was able to dump the files themselves to a new disk to recover them. Really not sure why, it was very strange.
cmurf 7y ago
Sounds like hardware problems in the storage stack. Btrfs developers contributed the dm-log-writes target to the kernel, expressly for conducting power loss tests on file systems. All the file systems benefit from this work. https://www.kernel.org/doc/Documentation/device-mapper/log-writes.txt https://www.kernel.org/doc/Documentation/device-mapper/log-w... And Btrfs is doing the right thing these days. I recently conducted system resource starvation tests where a compile process spun off enough threads to soak the system to the point it becomes unresponsive. I did over 100 forced power off tests while the Btrfs file system was being written to as part of that compile. Zero complaints: not on mount, not on scrubs, not with btrfs check, and not any while in normal operation following those power offs. If you want to complain about Btrfs, complain about the man page warning to not use --repair without advice from a developer. You did know about that warning, right?
- packetlost 7y ago
  100% was not a hardware problem. Works fine on other filesystems ️
  - cmurf 7y ago

paulddraper 7y ago

> an effect where btrfs loves you or hates you Same thing happens with operating systems.

scottlamb 7y ago

> It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). Why wouldn't you expect it to survive that? Is there a particular reason to believe those drives are broken? I.e., are they older consumer drives known to lie about cache flushes? do they have bad sectors? How have you abused it? What kind of load? Did you fill the filesystem (which another commenter mentioned seems to be a common element of most sad btrfs stories)? did your system frequently lose power while under write load? Lacking more details, I'd just say one user experiencing 0 bugs in 5 years should be completely unremarkable. I expect filesystems to be very reliable, so a lot of people having stories of corruption means stay away from btrfs. Having some people with stories of no corruption doesn't really move the needle. Together, these stories still mean stay away from btrfs!

cmurf 7y ago
That's hyperbole, it can't be taken seriously. OpenSUSE uses Btrfs by default, if there were more problems outside what's expected by md+LVM+ext4 (or XFS), which is the feature comprised by Btrfs and then some, they wouldn't have made the on-going investments they have. Facebook has been using it in production with thousands of installations for years. You want details from people experiencing zero problems, but you don't ask for details from people who are? That's a weird way to go about conducting the necessary autopsies, to discover and fix bugs. Anyway, I monitor the upstream filesystems lists, and they all have bugs. They're all fixing bugs. They're all adding new features. And that introduces bugs that need fixing. It's not that remarkable, until of course someone suggests only one file system is to be avoided, while also providing no details, but depends on conjecture.
- scottlamb 7y ago
  I asked RX14 why they called out their lack of problems as remarkable ("survive the most fucked up situations"). It sounds strange, as I mentioned. I don't need to ask people who've had problems because I've had them myself, in unremarkable circumstances, a while back. I'm sure I could find reports on the mailing list as well, in which others have already asked for details.

awill 7y ago

>but there seems to be an effect where btrfs loves you or hates you Surely it depends on the btrfs implementation. e.g. Arch Linux getting daily kernel updates vs an enterprise distro

packetlost 7y ago
Just as unstable on Arch as of a month or two ago.