4 ms·

is btrfs as a project even healthy? a startup I worked at looked hard at btrfs since the COW model and better integrity verification were both very, very usefu

is btrfs as a project even healthy? a startup I worked at looked hard at btrfs since the COW model and better integrity verification were both very, very useful from our POV. But btrfs in practice was just simply unsuitable; not only were there basic reliability issues (corrupted filesystems) and really bad corner case behavior (full filesystem in particular), we noted that it was not actually fully endian neutral, at least at the time, which caused filesystems between x86 and a BE ISA to appear to work, and then to be horribly corrupted if taken from a BE embedded system and mounted on x86. Given the volume of btrfs negative experiences that are going to need to be overcome (for example, many of the posts in [1]), maybe people have just given up? If ZoL licensing wasn't a problem, would anyone even be interested? [1] https://news.ycombinator.com/item?id=15087754 https://news.ycombinator.com/item?id=15087754

LeoPanthera 7y ago
I routinely hear about how btrfs is unreliable, unstable, and corrupts your data. But it's also been the default filesystem for SUSE Linux and Synology NAS products for a long time now, and they don't seem to be having any problems. I don't know what to believe.
- Leace 7y ago
  I'm using btrfs for a year on my laptop and there are no problems. I did some research before as I considered ZFS too but it seems most btrfs criticism is from old versions that did have bugs or using configurations that have known flaws (raid5/6, see [0]). ZFS, as much as I like it, will unfortunately never be part of the kernel and users will constantly be put into this kind of situations: https://www.phoronix.com/scan.php?page=news_item&px=ZFS-On-Linux-5.0-Problem https://www.phoronix.com/scan.php?page=news_item&px=ZFS-On-L... [0]: https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
- berkut 7y ago
  ReadyNAS products have also been using it for even longer. They don't use the RAID support from BTRFS (they do it themselves in software a level up using mdadm), only the snapshot, COW and general storage. There very occasionally (based on forum posts) seem to be issues with metadata fragmentation for some people when there's a lot of metadata, but otherwise seems very reliable (I've got two myself).
- dnr 7y ago
  I think it's good to get more data out there, positive and negative, so I'll add: I'm using btrfs on a home server (and sometimes desktop) for half a year now and haven't had any issues. It's fast (on nvme ssd), I can play with snapshots. It is admittedly a very lightly-loaded system. I filled up the root filesystem by accident once, which is supposed to be a very bad situation for btrfs. I was able to ssh in, diagnose stuff, delete the offending files, and then I did a minor rebalance to clean up some of the unnecessary metadata space that got allocated (not essential but I wanted to try it). No big deal and still works fine. I haven't used any btrfs raid features. I mirror it (via daily rsync) to an ext4 filesystem on another device to guard against filesystem-related bugs and device failure.
- diegocg 7y ago
  Facebook runs many thousands of btrfs filesystems every day, and it seems to works fine for them. They have helped to iron out a lot of bugs (and developed several features for it) I personally have been using it for maybe 7 years and it works just fine, and snapshots are awesome. To me, btrfs is being more stable than Ext3/4 back when I used them.
- Mister_Snuggles 7y ago
  Anecdotally, I've found btrfs unreliable. Here is a comment I made a week ago: ---------------------------- Personally, BTRFS feels like it has a ways to go before it's ready for prime-time. I've had two major and one minor BTRFS-related issues that have scared me away from it. 1) One of my computers got its BTRFS filesystem into a state where it would hang when trying to mount read/write. What I suspect is that there was some filesystem thing happening in the background when I rebooted the machine. I rebooted via the GUI and there was no sign that something was happening in the background, so this was really a normal thing that a user would do. No amount of fixing was able to get it back, but I was able to boot from the installation media, mount it read-only, and copy the data elsewhere. 2) Virtually all of the Linux servers at work will randomly hang for between minutes and hours. This was eventually traced to a BTRFS-scrub process that the OS vendor schedules to run weekly. The length and impact of the hang seemed to be based on how much write activity happens - servers where all the heavy activity happens on NFS mounts saw no impact, but servers that write a lot of logs to the local filesystem would get severely crippled on a weekly basis. We've moved a bunch of our write-heavy filesystems to non-BTRFS options as a result of this. 3) This is a more minor issue, but still speaks to my experience. I had a VM that was basically a remote desktop for me to use. Generally speaking it would hang hard after a few days of uptime with no actual usage. When I reinstalled it on a non-BTRFS (sorry, can't remember which filesystem I used) filesystem it was rock solid. I have no proof that this had anything to do with BTRFS. All of these were things that happened around a year ago, they may not be a true representation of the current state of BTRFS. But they've burned me badly, so now any use of BTRFS will be evaluated very carefully. In contrast, I've been running ZFS on a couple of FreeBSD servers, with fairly write-heavy loads, and have had no issues that were filesystem-related. Even under severe space and memory constraints ZFS has been rock solid. ---------------------------- The first problem is directly attributable to BTRFS. There is no way a filesystem should get corrupted by a simple user-initiated reboot, regardless of what the system is doing in the background. The second problem is a combination of BTRFS and the distribution. The distribution added a weekly job which did a BTRFS scrub (IIRC), under certain workloads that would completely hang machines for minutes to hours. The time this ran seems to be based on when the OS was installed, so as luck would have it these brought production systems down during business hours. The third problem is something I have no idea about. It could be BTRFS, it could be something completely different, I honestly have no idea.
- bscphil 7y ago
  Btrfs is reliable and stable - when you're running it on a single disk. That's what tends to the default mode for operating system installs that use Btrfs. I use it on all my single-disk installs to get checksumming, COW, and transparent compression. It also seems to be pretty solid in enterprise-friendly RAID10 software mode. Anything outside of that should be treated with extreme caution and avoided for anything you can't afford to lose. For RAID5/6 in particular, as far as I know, little has changed since the discovery of a major corruption-creating bug (which supposedly was going to require a complete rethink of major portions of the Btrfs architecture). It's an embarrassing situation, but no one seems to want to work on it because it would require a lot of work to fix, and anyone who's paying people to work on Btrfs isn't going to run RAID6 in production anyway. It's unfortunate, but the real best advice to anyone who wants software RAID and doesn't know exactly what they're getting into is "just use ZFS".
josefbacik 7y ago
Not sure what you are talking about, it’s definitely endian neutral. You can’t take different page sized fs and put it on another page size arch, but that’s a separate issue.
- foobiekr 7y ago
  2013 MIPS on Cavium as big endian -vs- Intel x86-64; appeared to work, did not. Page size not an issue, appeared to work, occasionally crazy with dumps containing values that are recognizably scrambled. It's been awhile since then, but that was our experience. I don't have access to the records or email from the time. Maybe it's been fixed.
guywhocodes 7y ago
Wow did we work on the same startup? We ended up with a ZoL cluster with better but not great stability instead with less than half the performance for latency critical loads. But at least it was correct.

LeoPanthera 7y ago

I routinely hear about how btrfs is unreliable, unstable, and corrupts your data. But it's also been the default filesystem for SUSE Linux and Synology NAS products for a long time now, and they don't seem to be having any problems. I don't know what to believe.

Leace 7y ago
I'm using btrfs for a year on my laptop and there are no problems. I did some research before as I considered ZFS too but it seems most btrfs criticism is from old versions that did have bugs or using configurations that have known flaws (raid5/6, see [0]). ZFS, as much as I like it, will unfortunately never be part of the kernel and users will constantly be put into this kind of situations: https://www.phoronix.com/scan.php?page=news_item&px=ZFS-On-Linux-5.0-Problem https://www.phoronix.com/scan.php?page=news_item&px=ZFS-On-L... [0]: https://btrfs.wiki.kernel.org/index.php/Gotchas https://btrfs.wiki.kernel.org/index.php/Gotchas
berkut 7y ago
ReadyNAS products have also been using it for even longer. They don't use the RAID support from BTRFS (they do it themselves in software a level up using mdadm), only the snapshot, COW and general storage. There very occasionally (based on forum posts) seem to be issues with metadata fragmentation for some people when there's a lot of metadata, but otherwise seems very reliable (I've got two myself).
dnr 7y ago
I think it's good to get more data out there, positive and negative, so I'll add: I'm using btrfs on a home server (and sometimes desktop) for half a year now and haven't had any issues. It's fast (on nvme ssd), I can play with snapshots. It is admittedly a very lightly-loaded system. I filled up the root filesystem by accident once, which is supposed to be a very bad situation for btrfs. I was able to ssh in, diagnose stuff, delete the offending files, and then I did a minor rebalance to clean up some of the unnecessary metadata space that got allocated (not essential but I wanted to try it). No big deal and still works fine. I haven't used any btrfs raid features. I mirror it (via daily rsync) to an ext4 filesystem on another device to guard against filesystem-related bugs and device failure.
diegocg 7y ago
Facebook runs many thousands of btrfs filesystems every day, and it seems to works fine for them. They have helped to iron out a lot of bugs (and developed several features for it) I personally have been using it for maybe 7 years and it works just fine, and snapshots are awesome. To me, btrfs is being more stable than Ext3/4 back when I used them.
Mister_Snuggles 7y ago
Anecdotally, I've found btrfs unreliable. Here is a comment I made a week ago: ---------------------------- Personally, BTRFS feels like it has a ways to go before it's ready for prime-time. I've had two major and one minor BTRFS-related issues that have scared me away from it. 1) One of my computers got its BTRFS filesystem into a state where it would hang when trying to mount read/write. What I suspect is that there was some filesystem thing happening in the background when I rebooted the machine. I rebooted via the GUI and there was no sign that something was happening in the background, so this was really a normal thing that a user would do. No amount of fixing was able to get it back, but I was able to boot from the installation media, mount it read-only, and copy the data elsewhere. 2) Virtually all of the Linux servers at work will randomly hang for between minutes and hours. This was eventually traced to a BTRFS-scrub process that the OS vendor schedules to run weekly. The length and impact of the hang seemed to be based on how much write activity happens - servers where all the heavy activity happens on NFS mounts saw no impact, but servers that write a lot of logs to the local filesystem would get severely crippled on a weekly basis. We've moved a bunch of our write-heavy filesystems to non-BTRFS options as a result of this. 3) This is a more minor issue, but still speaks to my experience. I had a VM that was basically a remote desktop for me to use. Generally speaking it would hang hard after a few days of uptime with no actual usage. When I reinstalled it on a non-BTRFS (sorry, can't remember which filesystem I used) filesystem it was rock solid. I have no proof that this had anything to do with BTRFS. All of these were things that happened around a year ago, they may not be a true representation of the current state of BTRFS. But they've burned me badly, so now any use of BTRFS will be evaluated very carefully. In contrast, I've been running ZFS on a couple of FreeBSD servers, with fairly write-heavy loads, and have had no issues that were filesystem-related. Even under severe space and memory constraints ZFS has been rock solid. ---------------------------- The first problem is directly attributable to BTRFS. There is no way a filesystem should get corrupted by a simple user-initiated reboot, regardless of what the system is doing in the background. The second problem is a combination of BTRFS and the distribution. The distribution added a weekly job which did a BTRFS scrub (IIRC), under certain workloads that would completely hang machines for minutes to hours. The time this ran seems to be based on when the OS was installed, so as luck would have it these brought production systems down during business hours. The third problem is something I have no idea about. It could be BTRFS, it could be something completely different, I honestly have no idea.
bscphil 7y ago
Btrfs is reliable and stable - when you're running it on a single disk. That's what tends to the default mode for operating system installs that use Btrfs. I use it on all my single-disk installs to get checksumming, COW, and transparent compression. It also seems to be pretty solid in enterprise-friendly RAID10 software mode. Anything outside of that should be treated with extreme caution and avoided for anything you can't afford to lose. For RAID5/6 in particular, as far as I know, little has changed since the discovery of a major corruption-creating bug (which supposedly was going to require a complete rethink of major portions of the Btrfs architecture). It's an embarrassing situation, but no one seems to want to work on it because it would require a lot of work to fix, and anyone who's paying people to work on Btrfs isn't going to run RAID6 in production anyway. It's unfortunate, but the real best advice to anyone who wants software RAID and doesn't know exactly what they're getting into is "just use ZFS".

josefbacik 7y ago

Not sure what you are talking about, it’s definitely endian neutral. You can’t take different page sized fs and put it on another page size arch, but that’s a separate issue.

foobiekr 7y ago
2013 MIPS on Cavium as big endian -vs- Intel x86-64; appeared to work, did not. Page size not an issue, appeared to work, occasionally crazy with dumps containing values that are recognizably scrambled. It's been awhile since then, but that was our experience. I don't have access to the records or email from the time. Maybe it's been fixed.

guywhocodes 7y ago

Wow did we work on the same startup? We ended up with a ZoL cluster with better but not great stability instead with less than half the performance for latency critical loads. But at least it was correct.