4 ms·

Probably rhymes with "btrfs balance". I'v had the misfortune of using btrfs in production with a few hundred machines on Ubuntu 14.04. It's one of the most fin

Probably rhymes with "btrfs balance". I'v had the misfortune of using btrfs in production with a few hundred machines on Ubuntu 14.04. It's one of the most finicky filesystems I've ever used. It's probably better in newer kernels, but if you have a lot of churn it requires constant care and feeding and tends to cause kernel softlockups fairly commonly. Also this: http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp http://blog.pgaddict.com/posts/friends-dont-let-friends-use-... and this: https://www.phoronix.com/scan.php?page=news_item&px=CoreOS-Btrfs-To-EXT4-OverlayFS https://www.phoronix.com/scan.php?page=news_item&px=CoreOS-B... and this (old but a few of the points still apply): http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/70 http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-... That being said I'm sure there are plenty of use cases for which btrfs in its current capacity is more than sufficient.

robszumski 11y ago
At CoreOS and we tried really hard to make btrfs happen, but it really came down to how different it operated than other file systems. It was mainly a UX issue and thus fell into my lap. The major issue is that regular debugging tools that folks have been using forever like `df -h` aren't just non-functional, they actively misrepresent the state of the file system. The most common example is indicating that you have plenty of free space when in fact you're out. We had to write a lot of documentation to teach people how btrfs works and how to debug it: https://coreos.com/os/docs/latest/btrfs-troubleshooting.html https://coreos.com/os/docs/latest/btrfs-troubleshooting.html The second major issue is that rebalancing requires free space, which is the problem that most folks are trying to fix with a rebalance operation. Catch-22 in the worst way. Containers vary in size and can restart frequently, churning through the btrfs chunks without filling them up, leaving around a lot of empty space that needs to be rebalanced.
- ryao 11y ago
  I hit that rebalancing needs space (and therefore can ENOSPC) issue at work when trying to compile ZFS on CoreOS on Digital Ocean before CoreOS switched from btrfs to ext4 and overlayfs. Getting ENOSPC on btrfs rebalance when you are seeing regular writes return ENOSPC is a really annoying problem.
nisa 11y ago
> I'v had the misfortune of using btrfs in production with a few hundred machines on Ubuntu 14.04 You are not alone. btrfs seems to be kind of stable - as in does not corrupt itself anymore - with 4.2 but it's been a nasty ride. It's an experimental filesystem that is neither complete nor stable yet. I wish this would be better communicated. It's needless frustrating: If you search for btrfs you come across a few slide decks that tell you: It's fine you can use it... after the first strange problems you'll subscribe the mailing list and every other day there is some post that shines some light into strange behavior and stuff that is not implemented. If you want checksumming on your single hdd backup disk btrfs is fine. For everything else you are up to some surprises... basically everything volume management and RAID is pretty much experimental and has strange behavior. Performance is not even a topic. I remember the ML discussion on this OLTP blogpost and the majority of responses was: Don't run databases on btrfs you stupid! I'd rather would read a technical discussion about the problems but from reading the ML it seems like it's too complex and few understand the complexity. @bcantrill called it a shit-show in some podcast and while it maybe technically not true it sure does looks like that. If you want peace of mind use mdraid+ext4 (or xfs) - ZFS on Linux has a lot of problems for heavy usage but the community is IMHO more invested in making it a good Linux citizen. On the other hand: This stuff is complicated and everyone expects miracles. I'm just looking at it from sysadmin perspective and on Linux both suck at the moment. But ZFS won't eat your data and has far better tooling. If you need something that works for high load on Linux I'd use neither.

robszumski 11y ago

At CoreOS and we tried really hard to make btrfs happen, but it really came down to how different it operated than other file systems. It was mainly a UX issue and thus fell into my lap. The major issue is that regular debugging tools that folks have been using forever like `df -h` aren't just non-functional, they actively misrepresent the state of the file system. The most common example is indicating that you have plenty of free space when in fact you're out. We had to write a lot of documentation to teach people how btrfs works and how to debug it: https://coreos.com/os/docs/latest/btrfs-troubleshooting.html https://coreos.com/os/docs/latest/btrfs-troubleshooting.html The second major issue is that rebalancing requires free space, which is the problem that most folks are trying to fix with a rebalance operation. Catch-22 in the worst way. Containers vary in size and can restart frequently, churning through the btrfs chunks without filling them up, leaving around a lot of empty space that needs to be rebalanced.

ryao 11y ago
I hit that rebalancing needs space (and therefore can ENOSPC) issue at work when trying to compile ZFS on CoreOS on Digital Ocean before CoreOS switched from btrfs to ext4 and overlayfs. Getting ENOSPC on btrfs rebalance when you are seeing regular writes return ENOSPC is a really annoying problem.

nisa 11y ago

> I'v had the misfortune of using btrfs in production with a few hundred machines on Ubuntu 14.04 You are not alone. btrfs seems to be kind of stable - as in does not corrupt itself anymore - with 4.2 but it's been a nasty ride. It's an experimental filesystem that is neither complete nor stable yet. I wish this would be better communicated. It's needless frustrating: If you search for btrfs you come across a few slide decks that tell you: It's fine you can use it... after the first strange problems you'll subscribe the mailing list and every other day there is some post that shines some light into strange behavior and stuff that is not implemented. If you want checksumming on your single hdd backup disk btrfs is fine. For everything else you are up to some surprises... basically everything volume management and RAID is pretty much experimental and has strange behavior. Performance is not even a topic. I remember the ML discussion on this OLTP blogpost and the majority of responses was: Don't run databases on btrfs you stupid! I'd rather would read a technical discussion about the problems but from reading the ML it seems like it's too complex and few understand the complexity. @bcantrill called it a shit-show in some podcast and while it maybe technically not true it sure does looks like that. If you want peace of mind use mdraid+ext4 (or xfs) - ZFS on Linux has a lot of problems for heavy usage but the community is IMHO more invested in making it a good Linux citizen. On the other hand: This stuff is complicated and everyone expects miracles. I'm just looking at it from sysadmin perspective and on Linux both suck at the moment. But ZFS won't eat your data and has far better tooling. If you need something that works for high load on Linux I'd use neither.