4 ms·
The article seems to be tying two ideas together - immutability and infrastructure as code. I’m unsure why the author decided to link the two ideas. Immutabili
by sargun 5y ago
The article seems to be tying two ideas together - immutability and infrastructure as code. I’m unsure why the author decided to link the two ideas.
Immutability, IMHO, is a terrible idea for infrastructure as a general purpose rule. Making it more complicated for developers to deploy systems, and requiring deployments to go through many steps (as immutability typically requires a full blue / green deploy), means MTTR will be longer.
Also, trying to fix a problem in situ takes longer (because you don’t have the tools for it).
Lastly, immutability is a lie. If you’re using something like v8, over time, the JIT and GC may change behavior. The cloud vendor may deploy new software.
If your software can be deployed as a fully self contained, stateless application, perhaps you can ignore these problems, especially by constantly recycling infrastructure. Now, the real problems begin to occur when you stop recycling - that’s when really interesting longevity problems appear.
/rant
- candiddevmike 5y agoIMO immutability is a byproduct of adopting Terraform--it doesn't manage day 2 stuff like patching, so the solution in some people's mind is "immutability" instead of bringing in another tool to manage the day 2 stuff. I personally see immutability as a sign of organizational immaturity around system administration, if you're concerned about drift on production systems you don't have the right config management and RBAC in place. It's never more efficient to replace a fleet vs patching it in place, especially from a time standpoint.
- Spivak 5y agoPatching is relatively easy to solve with immutable infrastructure. Build a new fully patched AMI every night and when you release you pick (version of code, version of AMI) to deploy.
- candiddevmike 5y agoI don't think the benefits of this outweigh the complexity and time costs to create and operate it. Having a scheduled task patch the servers regularly with extremely limited "break the glass" RBAC on your servers is better for 99% of environments.
- Spivak 5y agoThere's really no complexity. Our deploys are blue-green (i.e two ASGs and a load balancer, no need to make it sound fancy). You need to do this anyway because instances in the cloud are inherently unreliable. Deploying new code: publish a new container, scale up green, switch, whack blue at your leisure. Patching: publish a new AMI, scale up green, switch, whack blue at your leisure.
- ehutch79 5y agoIf you can patch it, doesn't that mean it's mutable?
- Spivak 5y agoNope, because once it's deployed to production it will never be changed without another deploy.
- ehutch79 5y agoThen you're not patching it?
- Spivak 5y agoHow our system works: 1. Every night we build a new freshly patched Ubuntu image. 2. Three times a week we run a deploy that rebuilds our entire infrastructure in place with the latest Ubuntu image as the base [1]. 3. Once the deploy runs we never run apt install or apt update again. So rather than server.update() we do server = server.update() [1] This sounds fancy but I promise it's stupid simple, all the complexity is in the auto scaling groups and the load balancers which we don't manage. And it's the exact same procedure to do code deploys which we do multiple times a day.
- tilolebo 5y agoHow do you handle stateful resources? Say for instance you have multiple elasticsearch clusters with terabytes of data on each cluster.
- nwmcsween 5y agoWell the NixOS idea is the opposite of what you describe, minus the implementation which does have mutable data.