7 ms·

Proposal to move CoreOS off of btrfs

63 points by jsnell 12y ago

HorizonXP 12y ago
Thank goodness. btrfs has bitten me way too often on CoreOS. I eventually succumbed to wiping the /var/lib/docker mounted directory on every reboot to minimize the occurrence of the low disk space issues, but nevertheless, issues still exist. Even today, I still have some cases where docker will complain about not being able to create the container/file, due to some inane error. I've had to reboot machines many times just to try to get it to cooperate. Furthermore, that solution of wiping the partition means that my machines need to redownload their docker images on every reboot, which means they come up more slowly than absolutely necessary.
- Gigablah 12y ago
  Yep. It's fun when your entire etcd cluster dies at the same time (because you started a global unit that pulls a huge docker container). You'd think they have safeguards to automatically rebalance on certain conditions (metadata filling up, etc) but no, you have to babysit btrfs.
  - saryant 12y ago
    I'm glad I'm not the only one with these problems. I was starting to think I was going crazy with my etcd+btrfs issues.
  - 23david 12y ago
    hrm... what's the failure scenario here? Are there any good btrfs management utils that can help with running btrfs in prod?
    - throwaway_nsa2 12y ago
      Stop it with the anti-American attitude. We are keeping tabs on you.
    - throwaway_nsa3 12y ago
      Stop it with the anti-American attitude. We are keeping tabs on you.
- sureshv 12y ago
  I've seen lots devicemapper failures running docker build on ext4 (AWS SSD EBS). Really painful to have to rebuild on failures since these issue seem to happen quite often.
- Shish2k 12y ago
  > Even today, I still have some cases where docker will complain about not being able to create the container/file, due to some inane error. I've had to reboot machines many times just to try to get it to cooperate. I get those, and am not using btrfs...
  - peteridah 12y ago
    yes this, I get those too and no btrfs
    - Gigablah 12y ago
      Is it this one? https://github.com/docker/docker/issues/5684 https://github.com/docker/docker/issues/5684 Or this? https://github.com/docker/docker/issues/4036 https://github.com/docker/docker/issues/4036
- 23david 12y ago
  This is ridiculous. CoreOS is managing their own distro, so why not just package in ZFS and get it over with. It's been how many years now waiting for the next great ZFS competitor? If nobody is able to improve on ZFS, how about we just all jump on the bandwagon and move on with our lives? If nothing else, having more people using ZFS may inspire someone to actually improve on it. BTRFS has such a limited feature set in comparison to ZFS that with the advent of ZFS on Linux, it's really hard to understand the community's continued backing of BTRFS as the Linux community's CoW filesystem competitor to ZFS. Spend a few weeks using ZFS on Linux on a machine with a few SSDs and a few HDDs, or just enable it on a linux laptop with a SSD drive, and it's hard to go back to anything else.
  - tacticus 12y ago
    Because sun decided to sabotage it when they released it and oracle haven't stepped back yet.
    - throwaway90446 12y ago
      Sabotage? How, exactly? I have heard of no major corruption or performance bugs in ZFS, what am I missing?
      - kbar13 12y ago
        licensing
      - ecnahc515 12y ago
        From http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue "ZFS is licensed under the Common Development and Distribution License (CDDL), and the Linux kernel is licensed under the GNU General Public License Version 2 (GPLv2). While both are free open source licenses they are restrictive licenses. The combination of them causes problems because it prevents using pieces of code exclusively available under one license with pieces of code exclusively available under the other in the same binary. "
      - 12y ago
- jaytaylor 12y ago
  I've been bitten badly by btrfs before as well. I wish it were feasible to get CoreOS (and Docker, for that matter) to add support for ZFS. I use LXC + ZFS every day and it is stable, reasonably fast, and all around wonderful!!! sigh.
visural 12y ago
Am I right in saying that the complaints with btrfs in CoreOS are specifically around its use in conjunction with Docker? (Interested as I'm thinking about building a homebrew NAS/general purpose server w/ btrfs, there's a lot of outdated info on btrfs but I was getting the impresssion that it's now a pretty stable and useable filesystem)
- lstamour 12y ago
  I can say that ZFS has worked great for me on BSD-based home servers. Haven't used ZFS with Linux yet, though it's possible to do so, it's just unpopular partly for licensing reasons. I suspect what type of RAID you do may have greater consequences than what file system you pick, particularly if your distro is already designed for serving files on the file system you choose. Oh and working out all the AFP/Samba bits are fun, because there's always something that surprises you.
  - bashinator 12y ago
    I've got severely burned by ZFS on linux running in AWS. Heavy NFS load (ZFS NFS, not linux kernel NFS) caused a kernel panic, pretty reproducibly. This was on Ubuntu 12.04 with the offical ZoL PPA sources, so YMMV.
    - 23david 12y ago
      For managing volumes, ZFS on Linux works great. But for managing NFS, I'd definitely go with a separate NFS implementation if I wanted to heavily use NFS. The primarily developers/users of the ZFS on Linux port are mainly using it for highly-available single-machine volume management, and exporting volumes to gluster or other clustering filesystems for use in massive HPC clusters like those over at Lawrence Livermore National Labs. http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-1024.pdf http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-... ZFS is meant for managing local drives, and to make it performant you need to configure SSD partitions to act as an L2 Arc cache. The online documentation is pretty good, so after going through the docs it should be pretty clear how to set ZFS up properly for your use cases. But it sounds like you're using EBS drives? If so, not sure why you'd want to use ZFS. Last I checked, ext2 or xfs was the way to go with EBS drives on AWS. AWS has so much stuff going on in the background to ensure reliability/availability of EBS volumes that adding another layer isn't worth it IMO and I've seen similar kernel panics running other complicated volume managers on top of EBS.
preillyme 12y ago
The out-of-space / metadata balancing problem has bitten me more times than I care to count. It's essentially a fact of life that I have to blow away /var/lib/docker and all its subvolumes every few weeks on any given machine, to clear an out-of-space problem.
- Jonanin 12y ago
  Copying a sentence from the first reply on the list?
- deleted 12y ago
  [deleted]
- mrmondo 12y ago
  That bug has been fixed, please update your kernel
discardorama 12y ago
So... I just installed Ubuntu 14.10 on my (new) home PC, and chose BTRFS because I figured it was stable by now. Should I be worried? I do plan on playing with docker.
- __david__ 12y ago
  I've not used it in about a year, but when I did it was slow as molasses. I didn't find any corruption issues, but the backup data I was storing there was at least an order of magnitude slower to access on btrfs than on xfs, which I eventually switched to. Yes, really, an order of magnitude. It's too bad, I really love the idea of btrfs, but there's no way I would ever run a filesystem on it yet.
- Gigablah 12y ago
  Just make sure you have plenty of disk space. Otherwise: http://ram.kossboss.com/btrfs-cant-mount-readwrite-full-space-issues-2-fixes/ http://ram.kossboss.com/btrfs-cant-mount-readwrite-full-spac...
- jckt 12y ago
  I've been using it on 2 home machines, for the home partitions, one using a HDD and one SSD. Been at it for over 2 years now, and I don't think I've ever had any filesystem related hiccups. Granted, they don't do anything special, just home stuff, but one is at around 75% capacity, so I guess you don't need that much extra disc space.
- rogerbinns 12y ago
  I've been using btrfs on several machines for about 3 years. It is not stable: https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_status https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_... When it goes wrong (about every 4 months for me), you will end up in a nightmare. Generally attempts to fix issues will make things worse, various tools and pages contradict each other, and the devs are only interested in the latest kernel version. Much of this is deliberate - the code is written to not sweep things under the rug, which means you can hit problems and not recover. Use backups and make sure you can restore. The reason why I keep using it is because there is no silent corruption as you get with ext4. A scrub can verify every byte of data is unaltered and recover if using anything other than single profile. Compression, volume management, cheap snapshots etc also make working with it nice. Until things go wrong. The single biggest "going wrong" is running out of space. Copy on write filesystems by their nature leave existing content alone and write new information in the spare space, eventually doing a garbage collect of obsolete data. When you are out of space that gets rather difficult with bizarre symptoms and tricky recovery.
finid 12y ago
If Btrfs is as bad as this proposal makes it look, why did the folks at openSUSE make it the default file system for the root partition on openSUSE 13.2?
- mrmondo 12y ago
  Yeah I haven't experienced any BTRFS problems, it could just be people stuck on RHEL running kernel 2.6 / 3.2.
- krakensden 12y ago
  Outside of docker, most people aren't using snapshots super heavily, which sounds like the root cause here.
doublerebel 12y ago
ZFS is coming for Docker. This is a natural move. Now that enterprise relies on the cloud they need enterprise-reliable storage -- and ZFS is proven. Looking forward to the ZFS-hosted cloud future.
- colechristensen 12y ago
  You mean Joyent?[1] Which has existed for quite a while. 1. https://www.joyent.com/ https://www.joyent.com/
- 23david 12y ago
  Funny thing is that if ZFS happens in Docker it'll be only because the community demanded it over the opposition of the core Docker maintainers. AFAICT, Aufs is being dropped due to massive opposition from Redhat. (Apparently AUFS code is too horrible to merge/support???) Despite its spotty track record, BTRFS is the only current recommended/supported CoW file system, and OverlayFS looks to be the docker-blessed next-gen Aufs alternative. Based on past experiences, I don't see Docker-blessed ZFS support coming anytime soon, but I hope I'm wrong. Maybe someone over at Joyent or Oracle can grease the wheels here? :-)
gojomo 12y ago
If so many people are so regularly bitten by these BTRFS issues – they're easy-to-reproduce and painful – shouldn't they be relatively easy to prioritize and fix?
- mrmondo 12y ago
  I'd like to see a real-world, reproducible BTRFS bug on 3.18 that's a serious problem.
mrmondo 12y ago
First they came for your SELinux... and you did nothing. Then they came for your Docker... and you did nothing. Now they've come for your filesystem... Madness! Just as BTRFS is finally stable and fast CoreOS decide yet again to ship. We've been using BTRFS in production for over a year now (and heavily with Docker) and haven't suffered any problems at all(1), in fact we actively simulate failures to practice / test repair processes which have all been successful. To me, while diversity is great CoreOS is going the path of Ubuntu with it's NIH syndrome. (1) With the exclusion of a slow docker push/pull bug that's about to be patched: https://github.com/docker/docker/pull/9720 https://github.com/docker/docker/pull/9720
- akerl_ 12y ago
  Can you clarify how this is an example of NIH? They're switching the default from btrfs to ext4, not writing their own filesystem. Also, it's necessary if they want to switch to overlayfs as the Docker backend, because btrfs lacks whiteout support which causes problems with overlayfs. Switching to overlayfs for Docker makes a lot of sense, given that you get the dedupe benefits of AUFS with lower overhead than devicemapper/btrfs and it's now part of the mainline kernel tree.
  - mrmondo 12y ago
    overlayfs is still in its infancy and you're then still relying on an underlying filesystem, also unless I'm mistaken OverlayFS doesn't provide compression and checksumming?
rickycook 12y ago
For everyone complaining about the metadata rebalancing, btrfs 3.18 (CoreOS is currently using 3.17) added auto rebalancing: https://btrfs.wiki.kernel.org/index.php/Balance_Filters https://btrfs.wiki.kernel.org/index.php/Balance_Filters https://btrfs.wiki.kernel.org/index.php?title=Main_Page#News https://btrfs.wiki.kernel.org/index.php?title=Main_Page#News

63 points by jsnell 12y ago

Thank goodness. btrfs has bitten me way too often on CoreOS. I eventually succumbed to wiping the /var/lib/docker mounted directory on every reboot to minimize the occurrence of the low disk space issues, but nevertheless, issues still exist. Even today, I still have some cases where docker will complain about not being able to create the container/file, due to some inane error. I've had to reboot machines many times just to try to get it to cooperate. Furthermore, that solution of wiping the partition means that my machines need to redownload their docker images on every reboot, which means they come up more slowly than absolutely necessary.

Gigablah 12y ago
Yep. It's fun when your entire etcd cluster dies at the same time (because you started a global unit that pulls a huge docker container). You'd think they have safeguards to automatically rebalance on certain conditions (metadata filling up, etc) but no, you have to babysit btrfs.
- saryant 12y ago
  I'm glad I'm not the only one with these problems. I was starting to think I was going crazy with my etcd+btrfs issues.
- 23david 12y ago
  hrm... what's the failure scenario here? Are there any good btrfs management utils that can help with running btrfs in prod?
  - throwaway_nsa2 12y ago
    Stop it with the anti-American attitude. We are keeping tabs on you.
  - throwaway_nsa3 12y ago
    Stop it with the anti-American attitude. We are keeping tabs on you.
sureshv 12y ago
I've seen lots devicemapper failures running docker build on ext4 (AWS SSD EBS). Really painful to have to rebuild on failures since these issue seem to happen quite often.
Shish2k 12y ago
> Even today, I still have some cases where docker will complain about not being able to create the container/file, due to some inane error. I've had to reboot machines many times just to try to get it to cooperate. I get those, and am not using btrfs...
- peteridah 12y ago
  yes this, I get those too and no btrfs
  - Gigablah 12y ago
    Is it this one? https://github.com/docker/docker/issues/5684 https://github.com/docker/docker/issues/5684 Or this? https://github.com/docker/docker/issues/4036 https://github.com/docker/docker/issues/4036
23david 12y ago
This is ridiculous. CoreOS is managing their own distro, so why not just package in ZFS and get it over with. It's been how many years now waiting for the next great ZFS competitor? If nobody is able to improve on ZFS, how about we just all jump on the bandwagon and move on with our lives? If nothing else, having more people using ZFS may inspire someone to actually improve on it. BTRFS has such a limited feature set in comparison to ZFS that with the advent of ZFS on Linux, it's really hard to understand the community's continued backing of BTRFS as the Linux community's CoW filesystem competitor to ZFS. Spend a few weeks using ZFS on Linux on a machine with a few SSDs and a few HDDs, or just enable it on a linux laptop with a SSD drive, and it's hard to go back to anything else.
- tacticus 12y ago
  Because sun decided to sabotage it when they released it and oracle haven't stepped back yet.
  - throwaway90446 12y ago
    Sabotage? How, exactly? I have heard of no major corruption or performance bugs in ZFS, what am I missing?
    - kbar13 12y ago
      licensing
    - ecnahc515 12y ago
      From http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue "ZFS is licensed under the Common Development and Distribution License (CDDL), and the Linux kernel is licensed under the GNU General Public License Version 2 (GPLv2). While both are free open source licenses they are restrictive licenses. The combination of them causes problems because it prevents using pieces of code exclusively available under one license with pieces of code exclusively available under the other in the same binary. "
    - 12y ago
jaytaylor 12y ago
I've been bitten badly by btrfs before as well. I wish it were feasible to get CoreOS (and Docker, for that matter) to add support for ZFS. I use LXC + ZFS every day and it is stable, reasonably fast, and all around wonderful!!! sigh.

visural 12y ago

Am I right in saying that the complaints with btrfs in CoreOS are specifically around its use in conjunction with Docker? (Interested as I'm thinking about building a homebrew NAS/general purpose server w/ btrfs, there's a lot of outdated info on btrfs but I was getting the impresssion that it's now a pretty stable and useable filesystem)

lstamour 12y ago
I can say that ZFS has worked great for me on BSD-based home servers. Haven't used ZFS with Linux yet, though it's possible to do so, it's just unpopular partly for licensing reasons. I suspect what type of RAID you do may have greater consequences than what file system you pick, particularly if your distro is already designed for serving files on the file system you choose. Oh and working out all the AFP/Samba bits are fun, because there's always something that surprises you.
- bashinator 12y ago
  I've got severely burned by ZFS on linux running in AWS. Heavy NFS load (ZFS NFS, not linux kernel NFS) caused a kernel panic, pretty reproducibly. This was on Ubuntu 12.04 with the offical ZoL PPA sources, so YMMV.
  - 23david 12y ago
    For managing volumes, ZFS on Linux works great. But for managing NFS, I'd definitely go with a separate NFS implementation if I wanted to heavily use NFS. The primarily developers/users of the ZFS on Linux port are mainly using it for highly-available single-machine volume management, and exporting volumes to gluster or other clustering filesystems for use in massive HPC clusters like those over at Lawrence Livermore National Labs. http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-1024.pdf http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-... ZFS is meant for managing local drives, and to make it performant you need to configure SSD partitions to act as an L2 Arc cache. The online documentation is pretty good, so after going through the docs it should be pretty clear how to set ZFS up properly for your use cases. But it sounds like you're using EBS drives? If so, not sure why you'd want to use ZFS. Last I checked, ext2 or xfs was the way to go with EBS drives on AWS. AWS has so much stuff going on in the background to ensure reliability/availability of EBS volumes that adding another layer isn't worth it IMO and I've seen similar kernel panics running other complicated volume managers on top of EBS.

preillyme 12y ago

The out-of-space / metadata balancing problem has bitten me more times than I care to count. It's essentially a fact of life that I have to blow away /var/lib/docker and all its subvolumes every few weeks on any given machine, to clear an out-of-space problem.

Jonanin 12y ago
Copying a sentence from the first reply on the list?
deleted 12y ago
[deleted]
mrmondo 12y ago
That bug has been fixed, please update your kernel

discardorama 12y ago

So... I just installed Ubuntu 14.10 on my (new) home PC, and chose BTRFS because I figured it was stable by now. Should I be worried? I do plan on playing with docker.

__david__ 12y ago
I've not used it in about a year, but when I did it was slow as molasses. I didn't find any corruption issues, but the backup data I was storing there was at least an order of magnitude slower to access on btrfs than on xfs, which I eventually switched to. Yes, really, an order of magnitude. It's too bad, I really love the idea of btrfs, but there's no way I would ever run a filesystem on it yet.
Gigablah 12y ago
Just make sure you have plenty of disk space. Otherwise: http://ram.kossboss.com/btrfs-cant-mount-readwrite-full-space-issues-2-fixes/ http://ram.kossboss.com/btrfs-cant-mount-readwrite-full-spac...
jckt 12y ago
I've been using it on 2 home machines, for the home partitions, one using a HDD and one SSD. Been at it for over 2 years now, and I don't think I've ever had any filesystem related hiccups. Granted, they don't do anything special, just home stuff, but one is at around 75% capacity, so I guess you don't need that much extra disc space.
rogerbinns 12y ago
I've been using btrfs on several machines for about 3 years. It is not stable: https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_status https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_... When it goes wrong (about every 4 months for me), you will end up in a nightmare. Generally attempts to fix issues will make things worse, various tools and pages contradict each other, and the devs are only interested in the latest kernel version. Much of this is deliberate - the code is written to not sweep things under the rug, which means you can hit problems and not recover. Use backups and make sure you can restore. The reason why I keep using it is because there is no silent corruption as you get with ext4. A scrub can verify every byte of data is unaltered and recover if using anything other than single profile. Compression, volume management, cheap snapshots etc also make working with it nice. Until things go wrong. The single biggest "going wrong" is running out of space. Copy on write filesystems by their nature leave existing content alone and write new information in the spare space, eventually doing a garbage collect of obsolete data. When you are out of space that gets rather difficult with bizarre symptoms and tricky recovery.

finid 12y ago

If Btrfs is as bad as this proposal makes it look, why did the folks at openSUSE make it the default file system for the root partition on openSUSE 13.2?

mrmondo 12y ago
Yeah I haven't experienced any BTRFS problems, it could just be people stuck on RHEL running kernel 2.6 / 3.2.
krakensden 12y ago
Outside of docker, most people aren't using snapshots super heavily, which sounds like the root cause here.

doublerebel 12y ago

ZFS is coming for Docker. This is a natural move. Now that enterprise relies on the cloud they need enterprise-reliable storage -- and ZFS is proven. Looking forward to the ZFS-hosted cloud future.

colechristensen 12y ago
You mean Joyent?[1] Which has existed for quite a while. 1. https://www.joyent.com/ https://www.joyent.com/
23david 12y ago
Funny thing is that if ZFS happens in Docker it'll be only because the community demanded it over the opposition of the core Docker maintainers. AFAICT, Aufs is being dropped due to massive opposition from Redhat. (Apparently AUFS code is too horrible to merge/support???) Despite its spotty track record, BTRFS is the only current recommended/supported CoW file system, and OverlayFS looks to be the docker-blessed next-gen Aufs alternative. Based on past experiences, I don't see Docker-blessed ZFS support coming anytime soon, but I hope I'm wrong. Maybe someone over at Joyent or Oracle can grease the wheels here? :-)

gojomo 12y ago

If so many people are so regularly bitten by these BTRFS issues – they're easy-to-reproduce and painful – shouldn't they be relatively easy to prioritize and fix?

mrmondo 12y ago
I'd like to see a real-world, reproducible BTRFS bug on 3.18 that's a serious problem.

mrmondo 12y ago

First they came for your SELinux... and you did nothing. Then they came for your Docker... and you did nothing. Now they've come for your filesystem... Madness! Just as BTRFS is finally stable and fast CoreOS decide yet again to ship. We've been using BTRFS in production for over a year now (and heavily with Docker) and haven't suffered any problems at all(1), in fact we actively simulate failures to practice / test repair processes which have all been successful. To me, while diversity is great CoreOS is going the path of Ubuntu with it's NIH syndrome. (1) With the exclusion of a slow docker push/pull bug that's about to be patched: https://github.com/docker/docker/pull/9720 https://github.com/docker/docker/pull/9720

akerl_ 12y ago
Can you clarify how this is an example of NIH? They're switching the default from btrfs to ext4, not writing their own filesystem. Also, it's necessary if they want to switch to overlayfs as the Docker backend, because btrfs lacks whiteout support which causes problems with overlayfs. Switching to overlayfs for Docker makes a lot of sense, given that you get the dedupe benefits of AUFS with lower overhead than devicemapper/btrfs and it's now part of the mainline kernel tree.
- mrmondo 12y ago
  overlayfs is still in its infancy and you're then still relying on an underlying filesystem, also unless I'm mistaken OverlayFS doesn't provide compression and checksumming?

rickycook 12y ago

For everyone complaining about the metadata rebalancing, btrfs 3.18 (CoreOS is currently using 3.17) added auto rebalancing: https://btrfs.wiki.kernel.org/index.php/Balance_Filters https://btrfs.wiki.kernel.org/index.php/Balance_Filters https://btrfs.wiki.kernel.org/index.php?title=Main_Page#News https://btrfs.wiki.kernel.org/index.php?title=Main_Page#News