8 ms·

Immutable Infrastructure and Disposable Components

93 points by nahname 13y ago

Roboprog 13y ago
Throw a little outsourcing and offshoring into the mix, with people on offset disjoint schedules and this confusion gets even deeper. I love the idea of building servers on the fly from an image under some kind of revision control.
- lmm 13y ago
  Netflix does it right IMO: they have an automated process that builds a complete AMI from source control. http://www.infoq.com/news/2013/06/netflix http://www.infoq.com/news/2013/06/netflix
hashtree 13y ago
Been doing this for years at all levels, and it really works. It allows me as a single dev to spend less than five hours (amortized) a week on a decent sized infrastructure to support my business. This includes building multiple racks of custom servers coloed at multiple DCs, networking, private clouds, server admin, hardware maintenance, upgrades, deploys, etc... and saving me 100k+ per year over AWS and having to hire another.
- mwcampbell 13y ago
  How do you do this outside of an environment like AWS? Do you have to reboot a whole physical server with a new OS image every time you change something?
  - hashtree 13y ago
    I leverage SmartOS, Ansible, along with a service oriented architecture based upon Finagle with ZooKeeper. More information on service arch approach: http://www.youtube.com/watch?v=30v5WVFvlno http://www.youtube.com/watch?v=30v5WVFvlno
    - mwcampbell 13y ago
      I've been watching Joyent and SmartOS in particular with interest for a while. A few questions to satisfy my curiosity. First, I'm assuming that you're running SmartOS on your own dedicated servers, not a public cloud like Joyent. Is that correct? If not, the rest doesn't apply. Do you use Ansible in the global zone, in individual non-global zones, or both? Do you build a new zone for each deploy of an application and each new version of a piece of infrastructure? I imagine this is the way to achieve the idea of immutable infrastructure. And one other little thing: Do you boot SmartOS via a USB key, PXE, or something else?
      - hashtree 13y ago
        I use Ansible for both global and non-global zones. You are correct, both the deploys and the new pieces get new zones (the old are destroy after the migration). The idea of raising entire new zones for simple migrations sounds like a pain in the butt on multiple fronts. In actuality, it is much easier. Zero downtime migrations too. I personally use USB keys, however it is expected that any one zone and/or server could fail at any moment. Servers are dirt cheap if you do it right. I bet some pay more in RAM costs per month than an entire node costs me with 32GB RAM, E3-1230v3, SSD, etc. There are multiple server types. The E3s are the non-persistent processing nodes... 3RU chassis 8 possible nodes in said chassis... each about 1k with the specs above and they are about 40 ECUs per. Each microcloud is typically around 10RU (1/4 rack). Place at separate datacenters and you get some interesting stuff. I have a whole different approach to the homebase DC. Stuff that REALLY reduce costs.
- raphinou 13y ago
  Wondering how you manage you database server. Care to share you approach?
  - hashtree 13y ago
    I make use of polyglot persistence. Graph, columnar, document, relational, key-value... each used when they are the right tool for the job. Each scaling horizontally. It gets easy to blow them away. :)
simonsarris 13y ago
This is essentially what I do with my own server, but largely because I'm too dim to trust myself to upgrade things well. That being said, he's abstracting the problem a little bit: > If you absolutely know a system has been created via automation and never changed since the moment of creation, most of the problems I describe above disappear. Need to upgrade? No problem. Build a new, upgraded system and throw the old one away. New app revision? Same thing. Build a server (or image) with a new revision and throw away the old ones. The key here is created via automation. You get a choice, I think: * You can spend time/energy/worry/entropy on making sure the server is up to date * You can spend time/energy/worry/entropy on making sure the automated process is up to date The second one is a little easier, to be sure, though depending on the setup it could still involve a lot of research, and it's certainly more expensive. But hey, if you're in a position where trading some money to buy some time is an option and you have the cash to burn, I'd buy the time at every chance.
- ninetenel 13y ago
  Well, This is a nice thought ... for a retard
ckdarby 13y ago
So, basically this article doesn't provide any insights and or technical background on how any of this was achieved. Just promotes the author for conference speaking. Well played good sir you got on the front page of HN, well played.
- wiremine 13y ago
  I agree it is a bit self-promoting, but I think the concept itself, as-is, without technical details, is worthy of a HN discussion.
- brudgers 13y ago
  Thought experiments such as the author's are the insight. One of the distinguishing features of a thought experiment is a lack of implementation details in the domain to which the analogy is applied. Better not to become bogged down trying to build a faster than light locomotive.
krosaen 13y ago
Related piece from another Fowler: http://martinfowler.com/bliki/SnowflakeServer.html http://martinfowler.com/bliki/SnowflakeServer.html
- beat 13y ago
  I love that article. Now I use the term "snowflake server" all the time as a blunt criticism, and I find it's really sinking in on the management and ops people who hear it. Great analogy.
- mwcampbell 13y ago
  I'm conflicted on this. On the one hand, the phrase "snowflake server" encapsulates my observations about long-lived servers that are upgraded and extended in place. On the other hand: "Perfect replication is the enemy of any robust system." -- http://en.wikiquote.org/wiki/Daniel_Suarez http://en.wikiquote.org/wiki/Daniel_Suarez (read the other quotes from the same chapter for context)
  - tome 13y ago
    But this "immutable infrastructure" idea doesn't mean that the infrastructure is perfectly replicated across machines. It aims for perfect replication through time. That's something quite different.
    - mwcampbell 13y ago
      You have a point. Indeed, the practice of building specialized machine or container images, which seems to be the best way of achieving immutable infrastructure, may make servers more heterogeneous, since each image will contain only the pieces necessary to perform its specific function. Done right, immutable infrastructure may be more resilient in the face of parasites.
- KevinEldon 13y ago
  And also PhoenixServer: http://martinfowler.com/bliki/PhoenixServer.html http://martinfowler.com/bliki/PhoenixServer.html
beat 13y ago
Conceptually, it's good. But in practice, it's just an analogy. Immutability in programming languages is enforced at compile time and run time. Something like: val x = 1; x = 2; is an actual error and the compiler/runtime will give a "you can't do that". Immutable infrastructure, on the other hand, would require read-only enforcement on all aspects of configuration. It can be done, but not easily, and someone always has root. So what you're really talking about is the ability to reconstruct an environment programatically from scratch, with no manual intervention, which is laudable but not exactly immutable. This is the sort of thing that makes me appreciate the 12 Factor App idea (http://www.12factor.net http://www.12factor.net). Rather than trying to make configuration immutable, make it impossible. Don't rely on the existence/continuity of a filesystem at all. Don't use configuration files.
- peteretep 13y ago
  I guess deployment to Heroku or similar is truly immutable infrastructure...
  - beat 13y ago
    It's pretty close.
- bryanlarsen 13y ago
  Just because a property of the process isn't enforced in the code does not mean that the property doesn't exist in the process. In C, it's possible to change a const. But const is still a very valuable construct, even if it is possible to cheat.
  - beat 13y ago
    Yeah, I suppose this analogy is much more like const-correctness in C than it is like a truly immutable variable.
- samstokes 13y ago
  There is a kind of read-only enforcement when all configuration is produced by automation - you learn, often painfully, that any writes (i.e. manual hacks) are liable to be rewritten, without warning and at arbitrary times. There's no error message telling you you can't hack an extra vhost into httpd.conf, but once you've got paged at 3am because your hack disappeared, you pretty quickly learn to supply your own error message. Heroku's an extreme example of this - the dyno filesystem is perfectly writable, but it's definitely going away within 24 hours. (The large-company version of "all configuration is produced by automation", which produces a surprisingly similar result, is "all configuration is produced by another team".)
JulianMorrison 13y ago
So you deploy a new physical server to the rack? Bundle the OS image with the deploy? Ghost it with the OS image before deploying? The article is light on explaining the "how" part.
oliao 13y ago
I think the point made here is quite interesting: instead of relying on (puppet, chef,...) to do the "deltas", always generating the whole system from scratch. This has a couple of advantages: * You make sure everything is automated * You can always trust the resulting system works; if it were to be built from scratch
- brudgers 13y ago
  Immutable hardware requires [an analogous] garbage collector. Most of the next machine would have the same state as the old one: ; Machine -> Machine ; Makes the next machine (define (next-machine m) (make-machine (machine-name m) (machine-cpu m) (machine-storage m) (update-os CURRENT-OS) (machine-applications m)))
antientropic 13y ago
The idea of applying purely functional programming concepts to deployment is not new, see the Nix package manager and the NixOS Linux distribution built on top of it (http://nixos.org/ http://nixos.org/). This paper explicitly makes the link with functional programming: http://nixos.org/~eelco/pubs/nixos-jfp-final.pdf http://nixos.org/~eelco/pubs/nixos-jfp-final.pdf
DannoHung 13y ago
http://docker.io http://docker.io is how I'd do this as soon as they have a stable version for production. Pick a common (or at least mostly common) base OS for every server, configure your entire system on top of a LXC container, badda bing, badda boom. Quick, low profile, repeatable, redistributable, etc etc etc.
- incision 13y ago
  Much the same thought I have. I've been doing various things toward the same goal for a long time, but Docker helps on every count (quick, redistributable etc).
bumeye 13y ago
This idea of immutable infrastructure reminds me of NixOS [1]. A linux distribution where nearly the whole filesystem is immutable. Definitely worth a look. I'm not sure if it's fit for production usage, but i think it's the future of operating systems :). [1] http://nixos.org/nixos/ http://nixos.org/nixos/
- edwintorok 13y ago
  It has some features to deploy to a cluster as well: http://nixos.org/disnix/ http://nixos.org/disnix/ And I find its 'nixos-rebuild build-vm' command quite handy to test changes. On the other hand immutability of the OS image doesn't mean you can easily roll-back after an upgrade, especially if you've been running the new version for a while, because you might still have mutable state (see below). Sure if you store all your configuration in Nix, then it is possible to rebuild the old configuration as it was, but your mutable state is not managed by Nix. One such example is database files. You've upgraded to the new configuration, and perhaps even had a script to auto-upgrade from your old schema. Now you want to go back: what should happen to the database? Should it be entirely rolled back (from backups) to an old snapshot? But then you loose all the new data. Should you roll-back the schema change? Again what happens to the new data that doesn't fit the old schema? Another example of mutable state is remote nodes that you depend on for a particular service. You can roll back your local machine to an old version, but will it still be able to talk to your new remote nodes? It seems that the only way to handle this is to make sure you are always at least 1 major version backwards compatible, so you can freely upgrade/roll-back machine that are part of a cluster.
  - edwintorok 13y ago
    Apparently there is also NixOps for network/cluster deployment: http://hydra.nixos.org/build/5426864/download/1/manual/manual.html http://hydra.nixos.org/build/5426864/download/1/manual/manua...
mwcampbell 13y ago
I see that Wunderlist uses EC2. So I speculate that they create a fresh machine image for each new revision of the application, and also use immutable machine images for bits of infrastructure such as the database. But how do we put the idea of immutable infrastructure into practice on bare-metal servers? One option would be to set up one's own virtualization infrastructure, such as Xen or KVM. But that would undermine the I/O performance and resource consolidation advantages of running on bare metal. Docker looks promising; we just need to discover the best practices for running specific kinds of services (e.g. web applications) on top of it.
- Ixiaus 13y ago
  Jails have been around for a while in FreeBSD and they are often used for this exact purpose; the concept has only, really, entered the mainstream due to widespread usage of EC2/OpenStack and now Docker (which is basically FreeBSD Jails + ezjail but for Linux and a little different on some other fronts). Nothing beats a bare metal machine with FreeBSD + ZFS pool + ezjail; the really cool thing about ZFS is that you can build a "base jail" then just use ZFS to make snapshots of the base jail when creating new jails. So with that you can create versioned base jails that are running an upgraded ports tree, or more generally, an upgraded world build! It also makes transition between application versions easy because you can run a separate database jail(s) (maybe different versions of the db?) and have a jail for app v1 running while a jail for app v2 is running at the same time. What I would love to see is a tool for abstracting this deployment pattern on-top of ezjail. I've been considering doing this myself but my time is stretched too thin :(
  - mwcampbell 13y ago
    Interesting. The one core OS feature that FreeBSD seems to be missing is a server-class virtualization technology, like KVM on Linux. Please correct me if I'm wrong on that. Of course, Illumos has zones, ZFS, KVM, and DTrace. But it's much less widely used than even FreeBSD.
  - derefr 13y ago
    > the really cool thing about ZFS is that you can build a "base jail" then just use ZFS to make snapshots of the base jail when creating new jails btrfs also supports this, and that support is built into some LXC commands (lxc-clone, for instance.) Docker just relies on filesystem overlays and bind-mounts instead, though, so it doesn't really have need for full persisted-to-disk subtrees of the instantiated version (it's more like it holds the base version and the persisted delta on disk, and only brings them together in memory.)
awj 13y ago
We've been doing this for ages where I work. New EC2 AMIs are built inside a disk image. We can pretty easily use all of the standard tools through chroot, so installing packages is relatively simple and flexible. Instances are ready for use immediately after booting, and deploying new code is as simple as rebuilding the image and booting it. We can also easily scale up most forms of servers by simply booting more and adding them to the proxy as they come up. The only downside here is that firefighting still kind of remains a two step process. Even after investing a decent bit of time optimizing the process, it still takes 5-10 minutes to build and upload an image. From there it takes another 5-30 minutes to get EC2 to boot it, so there's a strong temptation to tweak things on the server. I've taken to patching, testing, and committing the fix on the tag for that release so I can simply apply a diff to get the server up to date. Not perfect, but it goes a long way to prevent regressions.
- mwcampbell 13y ago
  Do you mount the root file system read-only in these images? That seems like it would be a great way to lock things down further, if it's practical.
  - awj 13y ago
    No, mostly because it's inconvenient. Making the root filesystem read only means having to track down everything that tries to write to it and fix mounts for them. After a few years of working with this, I kind of wish we'd bitten that bullet at the start. The AMIs we produce have a root filesystem that's only like 2 gigs so it goes across the network faster. 2 gigs isn't a lot of space when something decides it wants to log or write out temp files in weird places.
mitchellh 13y ago
I just want to note that Packer (http://www.packer.io http://www.packer.io) fits perfectly into the model of immutable infrastructure. Packer is an open source tool for automatically creating machine images (perhaps for multiple platforms). The idea of quickly and easily creating these master images in a way that doesn't slow down agility to change infrastructure is crucial, and Packer enables that. Disclaimer: I wrote Packer.
- edwintorok 13y ago
  How does it compare to DisNix? http://nixos.org/disnix/ http://nixos.org/disnix/
  - mitchellh 13y ago
    As far as I know, disnix is just for Nix. Packer creates machine images for multiple platforms. It would make sense, if you were building a machine image for Nix, perhaps, that you would use this. But for an Ubuntu machine, you can't really. Unless I'm mistaken.
AsymetricCom 13y ago
The level of discussion here is about one level separated from buzzword bingo.
tieTYT 13y ago
I think I must be missing something obvious here, but how can you do this if you use something with state like a database? How do you create a production system from scratch without losing all the data?

93 points by nahname 13y ago

Throw a little outsourcing and offshoring into the mix, with people on offset disjoint schedules and this confusion gets even deeper. I love the idea of building servers on the fly from an image under some kind of revision control.

lmm 13y ago
Netflix does it right IMO: they have an automated process that builds a complete AMI from source control. http://www.infoq.com/news/2013/06/netflix http://www.infoq.com/news/2013/06/netflix

hashtree 13y ago

Been doing this for years at all levels, and it really works. It allows me as a single dev to spend less than five hours (amortized) a week on a decent sized infrastructure to support my business. This includes building multiple racks of custom servers coloed at multiple DCs, networking, private clouds, server admin, hardware maintenance, upgrades, deploys, etc... and saving me 100k+ per year over AWS and having to hire another.

mwcampbell 13y ago
How do you do this outside of an environment like AWS? Do you have to reboot a whole physical server with a new OS image every time you change something?
- hashtree 13y ago
  I leverage SmartOS, Ansible, along with a service oriented architecture based upon Finagle with ZooKeeper. More information on service arch approach: http://www.youtube.com/watch?v=30v5WVFvlno http://www.youtube.com/watch?v=30v5WVFvlno
  - mwcampbell 13y ago
    I've been watching Joyent and SmartOS in particular with interest for a while. A few questions to satisfy my curiosity. First, I'm assuming that you're running SmartOS on your own dedicated servers, not a public cloud like Joyent. Is that correct? If not, the rest doesn't apply. Do you use Ansible in the global zone, in individual non-global zones, or both? Do you build a new zone for each deploy of an application and each new version of a piece of infrastructure? I imagine this is the way to achieve the idea of immutable infrastructure. And one other little thing: Do you boot SmartOS via a USB key, PXE, or something else?
    - hashtree 13y ago
      I use Ansible for both global and non-global zones. You are correct, both the deploys and the new pieces get new zones (the old are destroy after the migration). The idea of raising entire new zones for simple migrations sounds like a pain in the butt on multiple fronts. In actuality, it is much easier. Zero downtime migrations too. I personally use USB keys, however it is expected that any one zone and/or server could fail at any moment. Servers are dirt cheap if you do it right. I bet some pay more in RAM costs per month than an entire node costs me with 32GB RAM, E3-1230v3, SSD, etc. There are multiple server types. The E3s are the non-persistent processing nodes... 3RU chassis 8 possible nodes in said chassis... each about 1k with the specs above and they are about 40 ECUs per. Each microcloud is typically around 10RU (1/4 rack). Place at separate datacenters and you get some interesting stuff. I have a whole different approach to the homebase DC. Stuff that REALLY reduce costs.
raphinou 13y ago
Wondering how you manage you database server. Care to share you approach?
- hashtree 13y ago
  I make use of polyglot persistence. Graph, columnar, document, relational, key-value... each used when they are the right tool for the job. Each scaling horizontally. It gets easy to blow them away. :)

simonsarris 13y ago

This is essentially what I do with my own server, but largely because I'm too dim to trust myself to upgrade things well. That being said, he's abstracting the problem a little bit: > If you absolutely know a system has been created via automation and never changed since the moment of creation, most of the problems I describe above disappear. Need to upgrade? No problem. Build a new, upgraded system and throw the old one away. New app revision? Same thing. Build a server (or image) with a new revision and throw away the old ones. The key here is created via automation. You get a choice, I think: * You can spend time/energy/worry/entropy on making sure the server is up to date * You can spend time/energy/worry/entropy on making sure the automated process is up to date The second one is a little easier, to be sure, though depending on the setup it could still involve a lot of research, and it's certainly more expensive. But hey, if you're in a position where trading some money to buy some time is an option and you have the cash to burn, I'd buy the time at every chance.

ninetenel 13y ago
Well, This is a nice thought ... for a retard

ckdarby 13y ago

So, basically this article doesn't provide any insights and or technical background on how any of this was achieved. Just promotes the author for conference speaking. Well played good sir you got on the front page of HN, well played.

wiremine 13y ago
I agree it is a bit self-promoting, but I think the concept itself, as-is, without technical details, is worthy of a HN discussion.
brudgers 13y ago
Thought experiments such as the author's are the insight. One of the distinguishing features of a thought experiment is a lack of implementation details in the domain to which the analogy is applied. Better not to become bogged down trying to build a faster than light locomotive.

krosaen 13y ago

Related piece from another Fowler: http://martinfowler.com/bliki/SnowflakeServer.html http://martinfowler.com/bliki/SnowflakeServer.html

beat 13y ago
I love that article. Now I use the term "snowflake server" all the time as a blunt criticism, and I find it's really sinking in on the management and ops people who hear it. Great analogy.
mwcampbell 13y ago
I'm conflicted on this. On the one hand, the phrase "snowflake server" encapsulates my observations about long-lived servers that are upgraded and extended in place. On the other hand: "Perfect replication is the enemy of any robust system." -- http://en.wikiquote.org/wiki/Daniel_Suarez http://en.wikiquote.org/wiki/Daniel_Suarez (read the other quotes from the same chapter for context)
- tome 13y ago
  But this "immutable infrastructure" idea doesn't mean that the infrastructure is perfectly replicated across machines. It aims for perfect replication through time. That's something quite different.
  - mwcampbell 13y ago
    You have a point. Indeed, the practice of building specialized machine or container images, which seems to be the best way of achieving immutable infrastructure, may make servers more heterogeneous, since each image will contain only the pieces necessary to perform its specific function. Done right, immutable infrastructure may be more resilient in the face of parasites.
KevinEldon 13y ago
And also PhoenixServer: http://martinfowler.com/bliki/PhoenixServer.html http://martinfowler.com/bliki/PhoenixServer.html

beat 13y ago

Conceptually, it's good. But in practice, it's just an analogy. Immutability in programming languages is enforced at compile time and run time. Something like: val x = 1; x = 2; is an actual error and the compiler/runtime will give a "you can't do that". Immutable infrastructure, on the other hand, would require read-only enforcement on all aspects of configuration. It can be done, but not easily, and someone always has root. So what you're really talking about is the ability to reconstruct an environment programatically from scratch, with no manual intervention, which is laudable but not exactly immutable. This is the sort of thing that makes me appreciate the 12 Factor App idea (http://www.12factor.net http://www.12factor.net). Rather than trying to make configuration immutable, make it impossible. Don't rely on the existence/continuity of a filesystem at all. Don't use configuration files.

peteretep 13y ago
I guess deployment to Heroku or similar is truly immutable infrastructure...
- beat 13y ago
  It's pretty close.
bryanlarsen 13y ago
Just because a property of the process isn't enforced in the code does not mean that the property doesn't exist in the process. In C, it's possible to change a const. But const is still a very valuable construct, even if it is possible to cheat.
- beat 13y ago
  Yeah, I suppose this analogy is much more like const-correctness in C than it is like a truly immutable variable.
samstokes 13y ago
There is a kind of read-only enforcement when all configuration is produced by automation - you learn, often painfully, that any writes (i.e. manual hacks) are liable to be rewritten, without warning and at arbitrary times. There's no error message telling you you can't hack an extra vhost into httpd.conf, but once you've got paged at 3am because your hack disappeared, you pretty quickly learn to supply your own error message. Heroku's an extreme example of this - the dyno filesystem is perfectly writable, but it's definitely going away within 24 hours. (The large-company version of "all configuration is produced by automation", which produces a surprisingly similar result, is "all configuration is produced by another team".)

JulianMorrison 13y ago

So you deploy a new physical server to the rack? Bundle the OS image with the deploy? Ghost it with the OS image before deploying? The article is light on explaining the "how" part.

oliao 13y ago

I think the point made here is quite interesting: instead of relying on (puppet, chef,...) to do the "deltas", always generating the whole system from scratch. This has a couple of advantages: * You make sure everything is automated * You can always trust the resulting system works; if it were to be built from scratch

brudgers 13y ago
Immutable hardware requires [an analogous] garbage collector. Most of the next machine would have the same state as the old one: ; Machine -> Machine ; Makes the next machine (define (next-machine m) (make-machine (machine-name m) (machine-cpu m) (machine-storage m) (update-os CURRENT-OS) (machine-applications m)))

antientropic 13y ago

The idea of applying purely functional programming concepts to deployment is not new, see the Nix package manager and the NixOS Linux distribution built on top of it (http://nixos.org/ http://nixos.org/). This paper explicitly makes the link with functional programming: http://nixos.org/~eelco/pubs/nixos-jfp-final.pdf http://nixos.org/~eelco/pubs/nixos-jfp-final.pdf

DannoHung 13y ago

http://docker.io http://docker.io is how I'd do this as soon as they have a stable version for production. Pick a common (or at least mostly common) base OS for every server, configure your entire system on top of a LXC container, badda bing, badda boom. Quick, low profile, repeatable, redistributable, etc etc etc.

incision 13y ago
Much the same thought I have. I've been doing various things toward the same goal for a long time, but Docker helps on every count (quick, redistributable etc).

bumeye 13y ago

This idea of immutable infrastructure reminds me of NixOS [1]. A linux distribution where nearly the whole filesystem is immutable. Definitely worth a look. I'm not sure if it's fit for production usage, but i think it's the future of operating systems :). [1] http://nixos.org/nixos/ http://nixos.org/nixos/

edwintorok 13y ago
It has some features to deploy to a cluster as well: http://nixos.org/disnix/ http://nixos.org/disnix/ And I find its 'nixos-rebuild build-vm' command quite handy to test changes. On the other hand immutability of the OS image doesn't mean you can easily roll-back after an upgrade, especially if you've been running the new version for a while, because you might still have mutable state (see below). Sure if you store all your configuration in Nix, then it is possible to rebuild the old configuration as it was, but your mutable state is not managed by Nix. One such example is database files. You've upgraded to the new configuration, and perhaps even had a script to auto-upgrade from your old schema. Now you want to go back: what should happen to the database? Should it be entirely rolled back (from backups) to an old snapshot? But then you loose all the new data. Should you roll-back the schema change? Again what happens to the new data that doesn't fit the old schema? Another example of mutable state is remote nodes that you depend on for a particular service. You can roll back your local machine to an old version, but will it still be able to talk to your new remote nodes? It seems that the only way to handle this is to make sure you are always at least 1 major version backwards compatible, so you can freely upgrade/roll-back machine that are part of a cluster.
- edwintorok 13y ago
  Apparently there is also NixOps for network/cluster deployment: http://hydra.nixos.org/build/5426864/download/1/manual/manual.html http://hydra.nixos.org/build/5426864/download/1/manual/manua...

mwcampbell 13y ago

I see that Wunderlist uses EC2. So I speculate that they create a fresh machine image for each new revision of the application, and also use immutable machine images for bits of infrastructure such as the database. But how do we put the idea of immutable infrastructure into practice on bare-metal servers? One option would be to set up one's own virtualization infrastructure, such as Xen or KVM. But that would undermine the I/O performance and resource consolidation advantages of running on bare metal. Docker looks promising; we just need to discover the best practices for running specific kinds of services (e.g. web applications) on top of it.

Ixiaus 13y ago
Jails have been around for a while in FreeBSD and they are often used for this exact purpose; the concept has only, really, entered the mainstream due to widespread usage of EC2/OpenStack and now Docker (which is basically FreeBSD Jails + ezjail but for Linux and a little different on some other fronts). Nothing beats a bare metal machine with FreeBSD + ZFS pool + ezjail; the really cool thing about ZFS is that you can build a "base jail" then just use ZFS to make snapshots of the base jail when creating new jails. So with that you can create versioned base jails that are running an upgraded ports tree, or more generally, an upgraded world build! It also makes transition between application versions easy because you can run a separate database jail(s) (maybe different versions of the db?) and have a jail for app v1 running while a jail for app v2 is running at the same time. What I would love to see is a tool for abstracting this deployment pattern on-top of ezjail. I've been considering doing this myself but my time is stretched too thin :(
- mwcampbell 13y ago
  Interesting. The one core OS feature that FreeBSD seems to be missing is a server-class virtualization technology, like KVM on Linux. Please correct me if I'm wrong on that. Of course, Illumos has zones, ZFS, KVM, and DTrace. But it's much less widely used than even FreeBSD.
- derefr 13y ago
  > the really cool thing about ZFS is that you can build a "base jail" then just use ZFS to make snapshots of the base jail when creating new jails btrfs also supports this, and that support is built into some LXC commands (lxc-clone, for instance.) Docker just relies on filesystem overlays and bind-mounts instead, though, so it doesn't really have need for full persisted-to-disk subtrees of the instantiated version (it's more like it holds the base version and the persisted delta on disk, and only brings them together in memory.)

awj 13y ago

We've been doing this for ages where I work. New EC2 AMIs are built inside a disk image. We can pretty easily use all of the standard tools through chroot, so installing packages is relatively simple and flexible. Instances are ready for use immediately after booting, and deploying new code is as simple as rebuilding the image and booting it. We can also easily scale up most forms of servers by simply booting more and adding them to the proxy as they come up. The only downside here is that firefighting still kind of remains a two step process. Even after investing a decent bit of time optimizing the process, it still takes 5-10 minutes to build and upload an image. From there it takes another 5-30 minutes to get EC2 to boot it, so there's a strong temptation to tweak things on the server. I've taken to patching, testing, and committing the fix on the tag for that release so I can simply apply a diff to get the server up to date. Not perfect, but it goes a long way to prevent regressions.

mwcampbell 13y ago
Do you mount the root file system read-only in these images? That seems like it would be a great way to lock things down further, if it's practical.
- awj 13y ago
  No, mostly because it's inconvenient. Making the root filesystem read only means having to track down everything that tries to write to it and fix mounts for them. After a few years of working with this, I kind of wish we'd bitten that bullet at the start. The AMIs we produce have a root filesystem that's only like 2 gigs so it goes across the network faster. 2 gigs isn't a lot of space when something decides it wants to log or write out temp files in weird places.

mitchellh 13y ago

I just want to note that Packer (http://www.packer.io http://www.packer.io) fits perfectly into the model of immutable infrastructure. Packer is an open source tool for automatically creating machine images (perhaps for multiple platforms). The idea of quickly and easily creating these master images in a way that doesn't slow down agility to change infrastructure is crucial, and Packer enables that. Disclaimer: I wrote Packer.

edwintorok 13y ago
How does it compare to DisNix? http://nixos.org/disnix/ http://nixos.org/disnix/
- mitchellh 13y ago
  As far as I know, disnix is just for Nix. Packer creates machine images for multiple platforms. It would make sense, if you were building a machine image for Nix, perhaps, that you would use this. But for an Ubuntu machine, you can't really. Unless I'm mistaken.

AsymetricCom 13y ago

The level of discussion here is about one level separated from buzzword bingo.

tieTYT 13y ago

I think I must be missing something obvious here, but how can you do this if you use something with state like a database? How do you create a production system from scratch without losing all the data?