8 ms·

CoreOS Update: btrfs, docker 0.9, add users, writable /etc, and more

79 points by chrisfarms 13y ago

druiid 13y ago
Last I heard btrfs was still not considered a stable main production filesystem. Has this changed recently? Even just late last year I had not heard of production 'success' stories on it. Unless something has majorly changed, the idea of using btrfs for the main FS is a bit scary.
- gregkh 13y ago
  SuSE has been using btrfs as their "default stable production filesystem" for a few years now, with no problems. So it isn't a "recent" change at all.
- SEJeff 13y ago
  RHEL7 will support btrfs as a technology preview, which means they'll likely support it fully in a point release. That is a HUGE vote of confidence coming from their QA team.
- rodgerd 13y ago
  Oracle have supported it as a production filesystem in OEL for some time. SuSE have made it their default. It will be available in RHEL 7.
- craigyk 13y ago
  I've been a big proponent of ZFS, but recently btrfs has been making a lot of good progress with features. Most interesting to me is the ability to rebalance and switch redundancy levels. I have 20TB btrfs array in production as a backup target for a couple months now and it's been doing OK. I've even switched it from raid6 to raid10 and back. There's still a lot of rough edges though, which is why I'm only using it as a backup.
  - e12e 13y ago
    That's insane, unless you've also got tape/offsite/other-but-stable-fs-copy backup?
yeukhon 13y ago
Any well-known companies actually using CoreOS?
r4um 13y ago
Anyone using this in production yet?
- polvi 13y ago
  Lots of people playing with it... it is in a similar state as Docker, in that it is "pre-production" but people ignore that...
- lbotos 13y ago
  I want to but I'm waiting until they at least hit "beta". "During our alpha period, Chaos Monkey (i.e. random reboots) is built in and will give you plenty of opportunities to test out systemd." <- https://coreos.com/docs/quickstart/ https://coreos.com/docs/quickstart/
  - derefr 13y ago
    So your system isn't Highly Available enough to survive random reboots, then?
    - lbotos 13y ago
      My personal server isn't, no. It's not "omg it's making me money critical" but it's also not a temp dev box. :)
      - derefr 13y ago
        I guess I might be weird, but I love playing with HA distributed-failover tech for its own sake. fleetd means never having to say "oops, it crashed." I almost think it's worth it to switch the Chaos Monkey's toggle around: disabled in dev, but enabled in test and prod, to ensure you're using http://12factor.net/ http://12factor.net/ principles in your Docker containers. (I think Heroku does something similar by spontaneously killing containers every once in a while.)
  - robszumski 13y ago
    Totally understandable, but if you want to get started now, the chaos monkey can be turned off: http://coreos.com/docs/cluster-management/debugging/prevent-reboot-after-update/ http://coreos.com/docs/cluster-management/debugging/prevent-...
tomgruner 13y ago
Docker, CoreOS, and the entire ecosystem are moving really fast, it is a bit hard for me as a developer to keep up. For some projects we are creating containers with Docker and will need to get some things into production as soon as Docker 1.0 is declared stable. At the same time I want to switch to CoreOS, I also want to wait just a little while for blogs, tutorials, and tools to catch up to where CoreOS and Docker are at, and be usable in production by people who are not professional devops. The most exciting thing for me in this post is CoreOS CloudInit. It seems to be one of those tools that a small shop could use. It looks a bit like the yaml for fig, but is something that could be used in production as well. At the moment, I have been trying to solve everything a bit on my own with Makefiles, which include Makefiles for custom variables and Makefiles for commands that can be run against each container. It is working great for dev, but I could never really see how to run the containers in prod. The Makefiles didn't really seem like a prod solution, but CoreOS CloudInit looks like it could work. We also skipped boot2docker and use vagrant with ssh, so seeing that there is a Vagrant box which will run just like an EC2 box is pretty exciting. I would be excited if Digital Ocean would start supporting CoreOS too. [edit] Quite a few people would like CoreOS on Digital Ocean - http://digitalocean.uservoice.com/forums/136585-digital-ocean/suggestions/4250154-suport-coreos-as-a-deployment-platform http://digitalocean.uservoice.com/forums/136585-digital-ocea...
- derefr 13y ago
  > Quite a few people would like CoreOS on Digital Ocean I'm one of those, and currently trying to figure out a hacky workaround to make it possible. I'm already doing this[1] to get a current Ubuntu kernel running on DO. The same principle would seem to apply: boot into the CoreOS kernel, but with an initrd that mounts a different btrfs subvolume than the "bootstrap" OS. This would require DigitalOcean to have a btrfs-formatted image, though, because they don't offer any re-partitioning support... maybe having the initrd mount a loopback image containing a btrfs filesystem would work? I'm getting to the point, though, where it might be less effort to just start my own CoreOS-centered hosting service than to continue with DO... [1] https://gist.github.com/cpuguy83/6143347 https://gist.github.com/cpuguy83/6143347
  - 13y ago
jaytaylor 13y ago
Do any of you know why btrfs is being pushed rather than [the more mature and super flexible] zfs? There is lots of debate if you search google for comparisons, though they usually seem to end up favoring zfs, which is why I'm a little perplexed here. e.g. http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-is-better-than-btrfs http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-... Notably, I also ran into a show-stopping kernel bug [0] with btrfs within a day or two of the first production rollout of ShipBuilder [1] (it's an open-source self-hosted PaaS Heroku clone). Since switching to zfs there have been zero file-system related issues with any ShipBuilder production or staging environment so far as I am aware. [0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085 [1] http://shipbuilder.io http://shipbuilder.io
- rodgerd 13y ago
  Because it's illegal for anyone except Oracle to include ZFS with a Linux distribution, amongst other things. Take it up with Sunacle.
  - pekk 13y ago
    That's a pity, like Ksplice
  - patrickg_zill 13y ago
    Are you sure? I thought that ZFS for Linux can be distributed, but, only in source code form. So you would have to distributed the source code, then, build the ZFS module for it after / during installation.
    - vertex-four 13y ago
      That's pretty much it, except that nobody big has actually tried doing that (distributing an install CD which specifically builds ZFS as part of the standard install process), therefore it's not been in courts, therefore it's not proven that it's legally safe to do so, therefore anyone larger than a small business is unlikely to build on top of it.

79 points by chrisfarms 13y ago

Last I heard btrfs was still not considered a stable main production filesystem. Has this changed recently? Even just late last year I had not heard of production 'success' stories on it. Unless something has majorly changed, the idea of using btrfs for the main FS is a bit scary.

gregkh 13y ago
SuSE has been using btrfs as their "default stable production filesystem" for a few years now, with no problems. So it isn't a "recent" change at all.
SEJeff 13y ago
RHEL7 will support btrfs as a technology preview, which means they'll likely support it fully in a point release. That is a HUGE vote of confidence coming from their QA team.
rodgerd 13y ago
Oracle have supported it as a production filesystem in OEL for some time. SuSE have made it their default. It will be available in RHEL 7.
craigyk 13y ago
I've been a big proponent of ZFS, but recently btrfs has been making a lot of good progress with features. Most interesting to me is the ability to rebalance and switch redundancy levels. I have 20TB btrfs array in production as a backup target for a couple months now and it's been doing OK. I've even switched it from raid6 to raid10 and back. There's still a lot of rough edges though, which is why I'm only using it as a backup.
- e12e 13y ago
  That's insane, unless you've also got tape/offsite/other-but-stable-fs-copy backup?

yeukhon 13y ago

Any well-known companies actually using CoreOS?

r4um 13y ago

Anyone using this in production yet?

polvi 13y ago
Lots of people playing with it... it is in a similar state as Docker, in that it is "pre-production" but people ignore that...
lbotos 13y ago
I want to but I'm waiting until they at least hit "beta". "During our alpha period, Chaos Monkey (i.e. random reboots) is built in and will give you plenty of opportunities to test out systemd." <- https://coreos.com/docs/quickstart/ https://coreos.com/docs/quickstart/
- derefr 13y ago
  So your system isn't Highly Available enough to survive random reboots, then?
  - lbotos 13y ago
    My personal server isn't, no. It's not "omg it's making me money critical" but it's also not a temp dev box. :)
    - derefr 13y ago
      I guess I might be weird, but I love playing with HA distributed-failover tech for its own sake. fleetd means never having to say "oops, it crashed." I almost think it's worth it to switch the Chaos Monkey's toggle around: disabled in dev, but enabled in test and prod, to ensure you're using http://12factor.net/ http://12factor.net/ principles in your Docker containers. (I think Heroku does something similar by spontaneously killing containers every once in a while.)
- robszumski 13y ago
  Totally understandable, but if you want to get started now, the chaos monkey can be turned off: http://coreos.com/docs/cluster-management/debugging/prevent-reboot-after-update/ http://coreos.com/docs/cluster-management/debugging/prevent-...

tomgruner 13y ago

Docker, CoreOS, and the entire ecosystem are moving really fast, it is a bit hard for me as a developer to keep up. For some projects we are creating containers with Docker and will need to get some things into production as soon as Docker 1.0 is declared stable. At the same time I want to switch to CoreOS, I also want to wait just a little while for blogs, tutorials, and tools to catch up to where CoreOS and Docker are at, and be usable in production by people who are not professional devops. The most exciting thing for me in this post is CoreOS CloudInit. It seems to be one of those tools that a small shop could use. It looks a bit like the yaml for fig, but is something that could be used in production as well. At the moment, I have been trying to solve everything a bit on my own with Makefiles, which include Makefiles for custom variables and Makefiles for commands that can be run against each container. It is working great for dev, but I could never really see how to run the containers in prod. The Makefiles didn't really seem like a prod solution, but CoreOS CloudInit looks like it could work. We also skipped boot2docker and use vagrant with ssh, so seeing that there is a Vagrant box which will run just like an EC2 box is pretty exciting. I would be excited if Digital Ocean would start supporting CoreOS too. [edit] Quite a few people would like CoreOS on Digital Ocean - http://digitalocean.uservoice.com/forums/136585-digital-ocean/suggestions/4250154-suport-coreos-as-a-deployment-platform http://digitalocean.uservoice.com/forums/136585-digital-ocea...

derefr 13y ago
> Quite a few people would like CoreOS on Digital Ocean I'm one of those, and currently trying to figure out a hacky workaround to make it possible. I'm already doing this[1] to get a current Ubuntu kernel running on DO. The same principle would seem to apply: boot into the CoreOS kernel, but with an initrd that mounts a different btrfs subvolume than the "bootstrap" OS. This would require DigitalOcean to have a btrfs-formatted image, though, because they don't offer any re-partitioning support... maybe having the initrd mount a loopback image containing a btrfs filesystem would work? I'm getting to the point, though, where it might be less effort to just start my own CoreOS-centered hosting service than to continue with DO... [1] https://gist.github.com/cpuguy83/6143347 https://gist.github.com/cpuguy83/6143347
- 13y ago

jaytaylor 13y ago

Do any of you know why btrfs is being pushed rather than [the more mature and super flexible] zfs? There is lots of debate if you search google for comparisons, though they usually seem to end up favoring zfs, which is why I'm a little perplexed here. e.g. http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-is-better-than-btrfs http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-... Notably, I also ran into a show-stopping kernel bug [0] with btrfs within a day or two of the first production rollout of ShipBuilder [1] (it's an open-source self-hosted PaaS Heroku clone). Since switching to zfs there have been zero file-system related issues with any ShipBuilder production or staging environment so far as I am aware. [0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214085 [1] http://shipbuilder.io http://shipbuilder.io

rodgerd 13y ago
Because it's illegal for anyone except Oracle to include ZFS with a Linux distribution, amongst other things. Take it up with Sunacle.
- pekk 13y ago
  That's a pity, like Ksplice
- patrickg_zill 13y ago
  Are you sure? I thought that ZFS for Linux can be distributed, but, only in source code form. So you would have to distributed the source code, then, build the ZFS module for it after / during installation.
  - vertex-four 13y ago
    That's pretty much it, except that nobody big has actually tried doing that (distributing an install CD which specifically builds ZFS as part of the standard install process), therefore it's not been in courts, therefore it's not proven that it's legally safe to do so, therefore anyone larger than a small business is unlikely to build on top of it.