9 ms·

Infrastructure as code is not the answer (2022)

44 points by scapecast 3y ago

bilekas 3y ago
I'll say straight away, IaC for things like AWS in my company at least is invaluable.. the thought of even trying to deploy services with user policies etc, let alone the mess people would make of that is a disaster. Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.
blueflow 3y ago
Infrastructure as Code is fine, its that the current implementations are too whacky and too manpower-intensive to make it worth it. Good automation would be where you need less manpower afterwards..
logro 3y ago
With the same logic you can say that contraceptives are not the answer because they aren't 100% effective. Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.
ramoz 3y ago
A lot of people work in tech these days. A lot of people write blogs. Anyway... back to scale, consistency, and speed - IAC.
kryptiskt 3y ago
The big risk is that you'll outgrow the GUIs and end up with what amounts to code but all the logic is in YAML files instead.
- verdverm 3y ago
  Use CUE to tame the Yaml, you can generate yaml or json, and you get a bunch of tools and techniques for managing large config
jacksonhacker 3y ago
[dead]
baridbelmedar 3y ago
Imho, I think we've gone a bit overboard with Terraform at my company. Sure, it'll automate a bunch of stuff, but only a handful of people will really "get" it. I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case. I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.
- scaryclam 3y ago
  I often see people making what could be very simple terraform setups into incompehensible monolithic monsters as well. I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
  - sgarland 3y ago
    The opposite can be true as well. I’ve seen TF environments that were designed with so many layers of abstractions that unraveling a single piece of infra was an incredibly long endeavor. Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
- yourapostasy 3y ago
  The network effects benefits of a large population using a common domain specific language and abstraction are easily overlooked when one on the inside of that echo chamber dealing with the warts of that context frame. I’m also not overly fond of the current state of IaC but we have a good enough foundation upon which I have a rough line of sight on a future where we develop better representations going forward, and that alone is enough for me to avoid wholesale tossing it out or halting/slowing its use.
pquki4 3y ago
The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these. > I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late. That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.
- jpdb 3y ago
  And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs. You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
  - jansommer 3y ago
    Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
  - zaphar 3y ago
    The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production. Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense. This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
    - zzyzxd 3y ago
      GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race). To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra. If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline. It should have never been called infra-as-code, but infra-as-data. See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/
web3-is-a-scam 3y ago
We also went from - click around in the gui -> automate everything -> click around in the gui. Zero regrets. We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk. Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too. To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost. YMMV of course.
- rad_gruchalski 3y ago
  We stopped hosting our website and send real post by horse carriage and, when weather is good, post pigeons. Zero regrets here too!
  - Xylakant 3y ago
    At least you get less SEO spam emails than I do!
    - rad_gruchalski 3y ago
      And those savings. A bunch of pigeons instead of SREs! It’s a steal!
  - web3-is-a-scam 2y ago
    It’s a good thing we have tons of server cattle to feed all that straw to
- sumtechguy 3y ago
  When you have 10 boxes to manage doing it manually is within the realm of 1 person doing it and maybe even correctly! When you have 500... You will want consistency and repeatability. To this day I am still 'fixing' things someone did in the 10 machine area. 'oh yeah that machine is set up this way and the others are that way'. But I can not get rid of those 10 boxes because they became key infrastructure.
VHRanger 3y ago
Besides the point, but Medium.com is not the answer if you want to publish text on the internet. As for the article he's not proposing anything better. Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.
- klabb3 3y ago
  Decent GUIs typically have sanity checks and validation built in. Yaml otoh has an extremely relaxed structure, which to me sounds dangerously easy to mess up. Terraform isn’t my domain, but do senior yaml engineers these days have proper guardrails against typos, wrong indentation, type errors etc?
angarg12 3y ago
All of the downsides are 10x worse when doing infra manually. Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.
- Wheaties466 3y ago
  not to mention the potential security implications of doing it manually. not being consistent across the whole deployment. Sounds like a nightmare.
davedx 3y ago
I'm going to agree - because there's also the complexity of keeping all the state in sync too. Give me Fly.io or something similar any day. I'll pay the premium gratefully.
- verdverm 3y ago
  Works fine if you have a minimal scope, but try that with multiple teams of engineers who need to each deploy their piece of org level infra and services, and it will be a mess
  - davedx 2y ago
    People used to do this on Heroku. What’s the problem exactly?
    - verdverm 2y ago
      Coordination and fat finger mistakes
seper8 3y ago
Dumb article, written by someone who probably has never had to setup dev>acceptance>prd environments. GL clicking your way through that buddy.
jeroenvlek 3y ago
All of the things mentioned in the article seem more a result of bad IaC, instead of general IaC... > Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available. This is true for all code and we have tried and tested methods/principles for dealing with that. > Someone’s user account got built in, and they don’t work here any more. No comment needed. > He was in the Prod directory — and starting nuking production. Say it with me: Elevate Permissions
- scaryclam 3y ago
  > Say it with me: Elevate Permissions I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc. But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important. I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.
orwin 3y ago
I disagree with a lot of this, but here are the bit i really disagree with: > The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there. This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS. Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood. > It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read. Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.
foobarian 3y ago
Honestly a lot of the arguments in that post seem in favor of IaC, not against it.
sgarland 3y ago
Letting teams self-serve XaaS is how you get 50 dramatically mis-sized and ill-configured DB instances in prod. This team thought they’d have 1000 QPS, and have an instance that’s 50x as large as necessary. This team doesn’t know what IOPS are but figured they’d better have a lot of them, so they’re paying thousands per month for an io2 drive when the entire dataset fits comfortably into memory. This team has no idea (fair – there are a million knobs) how to configure Postgres, and the defaults suck, so they’re getting 1/4 the performance that they’re paying for. IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.
morsecodist 3y ago
IAC is one technology I will defend to the hilt. IAC is difficult because the underlying problem it is trying to solve is hard. Different people over time are making updates to a mission-critical, stateful, system that need to be coordinated with each other. IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change. If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.
ThinkBeat 3y ago
Setting up solutions in AWS is a most painful experience. Even AWS consultants do not find it easy and trivial. However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture. The best way to address it is to decrease the complexity of the system. It will have great rewards across the board. Most systems composed these days, have far more moving parts than what is needed. Simplify and shrink. Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.
k8sToGo 3y ago
IaC, like DevOps, is a mindset.
indymike 3y ago
reproducible - IAC is a quantum leap over manual on this one. self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook prevents mistakes - maybe. but at least the mistakes are documented and visible. prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close
- Wheaties466 3y ago
  tbh this is the only sound comment in this whole thread. IaC is literally a million times better than the alternative. sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
  - indymike 3y ago
    Maybe it is that IaC is usually imposed on developers, injecting a forced learning curve. In this respect, IaC is a lot like JavaScript, and I suspect that, much like JavaScript, people are just mad about being forced to learn it and haven’t realized all the “good parts” yet.
cess11 3y ago
I read the addendum. I think it was a good, understandable rant. When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself. If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code. I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease. But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.
- sgarland 3y ago
  Terraform isn’t Ansible, and vice-versa. I’ve seen both being shoehorned to do the job of the other, and it isn’t pretty. Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications. Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
  - cess11 3y ago
    Sure, there are many tools in this area, and they can be used in many ways. We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz. As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.
opentokix 3y ago
Regardless of what the author himself is saying, this is a clear example of someone that did not actually get passed intermediate level in knowledge and expertise.
noop_joe 3y ago
IaC is undeniably a good pattern for dealing with complex systems. The bigger problem is that it's not a solution for delineating responsibilities between software engineers and infrastructure engineers. All the specifics about an environment that need to make their way from software->infra (and in some cases the other way) still need to be worked out, and that's often a bigger problem than getting code to a box that always has access to a database (oversimplification). I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications. 1. https://noop.dev/blog/build-apps-not-infrastructure/ https://noop.dev/blog/build-apps-not-infrastructure/
eezing 3y ago
Infrastructure as code can be useful for complex systems. These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.
TeamDman 3y ago
To avoid data loss in prod, you best be using lifecycle { prevent_destroy = true } https://developer.hashicorp.com/terraform/tutorials/state/resource-lifecycle#prevent-resource-deletion https://developer.hashicorp.com/terraform/tutorials/state/re... > trying to get some old module to work in a new cloud environment where a bunch of assumptions about how things are done no longer apply Square hole type energy. Just because you can coerce the code to run doesn't mean it's a good idea. New environment, new assumptions, new code. Copy-paste what's relevant, but in a new project. > getting totally confused because what they see in the code doesn’t match reality Terraform compares state to detect drift? When reality doesn't match code, you clobber reality or you update the code. > It Prevents Drift! > This is true if and only if, you only ever, ever, ever, create code to change things in your environment. It at least makes drift obvious, even if it requires attention for it to be addressed. > code is, in my opinion, really not good at being documentation Unlike comments, IaC "can't" be inaccurate after applying. > But in my experience, the effort to keep your code up-to-date is a sink for engineering time. If the code is out of date, it's because you made a click change and didn't update the code. When this happens, terraform tells you exactly what changed in simple scenarios, and in more complex scenarios you can use import blocks to rein things in. > Yes it’s a clickbait headline. You clicked on it, didn’t you? Got me good. The article is a good launchboard for discussion. Overall, the IaC scene has plenty of room for growth. Terraform import blocks and code generation is helping reduce the problem of "I clicked in the portal now I gotta update the code" because you can sync both ways. Code updates infra, infra can be used to generate code.
Jenkins2000 3y ago
Back in the olden days, I had a junior dev ask me if he could learn ops, I said sure, I was just a hacker myself. He wrote a script to make sure the hard drives in all the servers were good. A month or so later we rebooted a server and it wouldn't come back up. We finally rebooted another to see if we could learn something about the dead server. But it also wouldn't boot. Then to our horror we learned that the disk check script was writing and reading to the boot sector to make sure it was good. All of our servers were affected. Good times.
xorcist 3y ago
I've done IaC is various forms over a decade and I believe the term does more harm than good. It's not "Infrastructure", and it's not really "Code" either. "Infrastructure Configuration as Text" perhaps would be more descriptive. Or just stick with the samewhat older term "Configuration Management". These days, however, it's not really in vogue. Nobody thinks they need it anymore. They have Kubernetes and containers now. When you ask enough questions about how those are managed, it usually turns out some state is being managed somewhere outside of version control. I believe the pendulum will swing back in time, but in the meantime there will be K8s on my resume, not IaC.
moribvndvs 3y ago
Truly cursed take. Loved the “uh… ha ha, got’em!” walk-back in the addendum. Here’s an another anecdote for you, if that’s what we’re basing our arguments on: My prior team had dozens of services to deploy and they were all done in the console. It took hours to roll out deployments, people constantly fucked something up (wrong version, missed dependency, config variable, etc), you had spend sometimes up to an hour to get a picture of how a service was currently stood up, was almost completely opaque and thus extremely expensive to get new hires up to speed, obviated the ability to perform post deploy automation for smoke tests and such, required giving leads broad access to environments that needed to be locked down, and the best part was you had to do it all over again for each environment. All of this went completely away when I rolled it into CloudFormation, hooked it into CI/CD pipeline, and let a maintainer kick it off with the click of a button. I guess I must have been doing something wrong /s
zbentley 3y ago
A significant benefit of IaC is the work-savings it yields for recovery from a total-site-loss event, or the migration to multi-geo/multi-site. Sure, very few IaC environments are going to make that sort of thing seamless--bootstrapping new infra from nothing in a hurry is always going to turn up every skeleton in every closet and require a lot of manual work. But having even 80% IaC automation yields huge time savings in those rare-but-impactful "start from zero" cases. Having even just the easy/copy-paste parts of the infra encoded in automation can be the difference between hours and days for disaster recovery, or days and weeks for creating a new geo/staging/whatever environment.
whatindaheck 3y ago
Almost all the listed downsides of IaC are also downsides of _any_ code. I was starting to address the points one-by-one but that’s the gist of the entire article. The alternative is we all start using Alice to click and drag pieces of logic together to build software. This is the equivalent of building infra through a web UI. I don’t see the author’s point. Sure IaC isn’t a magical solution with zero problems but was anybody claim that?

44 points by scapecast 3y ago

I'll say straight away, IaC for things like AWS in my company at least is invaluable.. the thought of even trying to deploy services with user policies etc, let alone the mess people would make of that is a disaster. Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.

blueflow 3y ago

Infrastructure as Code is fine, its that the current implementations are too whacky and too manpower-intensive to make it worth it. Good automation would be where you need less manpower afterwards..

logro 3y ago

With the same logic you can say that contraceptives are not the answer because they aren't 100% effective. Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.

ramoz 3y ago

A lot of people work in tech these days. A lot of people write blogs. Anyway... back to scale, consistency, and speed - IAC.

kryptiskt 3y ago

The big risk is that you'll outgrow the GUIs and end up with what amounts to code but all the logic is in YAML files instead.

verdverm 3y ago
Use CUE to tame the Yaml, you can generate yaml or json, and you get a bunch of tools and techniques for managing large config

jacksonhacker 3y ago

[dead]

baridbelmedar 3y ago

Imho, I think we've gone a bit overboard with Terraform at my company. Sure, it'll automate a bunch of stuff, but only a handful of people will really "get" it. I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case. I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.

scaryclam 3y ago
I often see people making what could be very simple terraform setups into incompehensible monolithic monsters as well. I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
- sgarland 3y ago
  The opposite can be true as well. I’ve seen TF environments that were designed with so many layers of abstractions that unraveling a single piece of infra was an incredibly long endeavor. Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
yourapostasy 3y ago
The network effects benefits of a large population using a common domain specific language and abstraction are easily overlooked when one on the inside of that echo chamber dealing with the warts of that context frame. I’m also not overly fond of the current state of IaC but we have a good enough foundation upon which I have a rough line of sight on a future where we develop better representations going forward, and that alone is enough for me to avoid wholesale tossing it out or halting/slowing its use.

pquki4 3y ago

The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these. > I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late. That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.

jpdb 3y ago
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs. You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
- jansommer 3y ago
  Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
- zaphar 3y ago
  The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production. Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense. This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
  - zzyzxd 3y ago
    GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race). To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra. If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline. It should have never been called infra-as-code, but infra-as-data. See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/

web3-is-a-scam 3y ago

We also went from - click around in the gui -> automate everything -> click around in the gui. Zero regrets. We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk. Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too. To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost. YMMV of course.

rad_gruchalski 3y ago
We stopped hosting our website and send real post by horse carriage and, when weather is good, post pigeons. Zero regrets here too!
- Xylakant 3y ago
  At least you get less SEO spam emails than I do!
  - rad_gruchalski 3y ago
    And those savings. A bunch of pigeons instead of SREs! It’s a steal!
- web3-is-a-scam 2y ago
  It’s a good thing we have tons of server cattle to feed all that straw to
sumtechguy 3y ago
When you have 10 boxes to manage doing it manually is within the realm of 1 person doing it and maybe even correctly! When you have 500... You will want consistency and repeatability. To this day I am still 'fixing' things someone did in the 10 machine area. 'oh yeah that machine is set up this way and the others are that way'. But I can not get rid of those 10 boxes because they became key infrastructure.

VHRanger 3y ago

Besides the point, but Medium.com is not the answer if you want to publish text on the internet. As for the article he's not proposing anything better. Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.

klabb3 3y ago
Decent GUIs typically have sanity checks and validation built in. Yaml otoh has an extremely relaxed structure, which to me sounds dangerously easy to mess up. Terraform isn’t my domain, but do senior yaml engineers these days have proper guardrails against typos, wrong indentation, type errors etc?

angarg12 3y ago

All of the downsides are 10x worse when doing infra manually. Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.

Wheaties466 3y ago
not to mention the potential security implications of doing it manually. not being consistent across the whole deployment. Sounds like a nightmare.

davedx 3y ago

I'm going to agree - because there's also the complexity of keeping all the state in sync too. Give me Fly.io or something similar any day. I'll pay the premium gratefully.

verdverm 3y ago
Works fine if you have a minimal scope, but try that with multiple teams of engineers who need to each deploy their piece of org level infra and services, and it will be a mess
- davedx 2y ago
  People used to do this on Heroku. What’s the problem exactly?
  - verdverm 2y ago
    Coordination and fat finger mistakes

seper8 3y ago

Dumb article, written by someone who probably has never had to setup dev>acceptance>prd environments. GL clicking your way through that buddy.

jeroenvlek 3y ago

All of the things mentioned in the article seem more a result of bad IaC, instead of general IaC... > Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available. This is true for all code and we have tried and tested methods/principles for dealing with that. > Someone’s user account got built in, and they don’t work here any more. No comment needed. > He was in the Prod directory — and starting nuking production. Say it with me: Elevate Permissions

scaryclam 3y ago
> Say it with me: Elevate Permissions I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc. But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important. I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.

orwin 3y ago

I disagree with a lot of this, but here are the bit i really disagree with: > The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there. This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS. Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood. > It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read. Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.

foobarian 3y ago

Honestly a lot of the arguments in that post seem in favor of IaC, not against it.

sgarland 3y ago

Letting teams self-serve XaaS is how you get 50 dramatically mis-sized and ill-configured DB instances in prod. This team thought they’d have 1000 QPS, and have an instance that’s 50x as large as necessary. This team doesn’t know what IOPS are but figured they’d better have a lot of them, so they’re paying thousands per month for an io2 drive when the entire dataset fits comfortably into memory. This team has no idea (fair – there are a million knobs) how to configure Postgres, and the defaults suck, so they’re getting 1/4 the performance that they’re paying for. IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.

morsecodist 3y ago

IAC is one technology I will defend to the hilt. IAC is difficult because the underlying problem it is trying to solve is hard. Different people over time are making updates to a mission-critical, stateful, system that need to be coordinated with each other. IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change. If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.

ThinkBeat 3y ago

Setting up solutions in AWS is a most painful experience. Even AWS consultants do not find it easy and trivial. However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture. The best way to address it is to decrease the complexity of the system. It will have great rewards across the board. Most systems composed these days, have far more moving parts than what is needed. Simplify and shrink. Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.

k8sToGo 3y ago

IaC, like DevOps, is a mindset.

indymike 3y ago

reproducible - IAC is a quantum leap over manual on this one. self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook prevents mistakes - maybe. but at least the mistakes are documented and visible. prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close

Wheaties466 3y ago
tbh this is the only sound comment in this whole thread. IaC is literally a million times better than the alternative. sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
- indymike 3y ago
  Maybe it is that IaC is usually imposed on developers, injecting a forced learning curve. In this respect, IaC is a lot like JavaScript, and I suspect that, much like JavaScript, people are just mad about being forced to learn it and haven’t realized all the “good parts” yet.

cess11 3y ago

I read the addendum. I think it was a good, understandable rant. When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself. If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code. I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease. But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.

sgarland 3y ago
Terraform isn’t Ansible, and vice-versa. I’ve seen both being shoehorned to do the job of the other, and it isn’t pretty. Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications. Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
- cess11 3y ago
  Sure, there are many tools in this area, and they can be used in many ways. We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz. As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.

opentokix 3y ago

Regardless of what the author himself is saying, this is a clear example of someone that did not actually get passed intermediate level in knowledge and expertise.

noop_joe 3y ago

IaC is undeniably a good pattern for dealing with complex systems. The bigger problem is that it's not a solution for delineating responsibilities between software engineers and infrastructure engineers. All the specifics about an environment that need to make their way from software->infra (and in some cases the other way) still need to be worked out, and that's often a bigger problem than getting code to a box that always has access to a database (oversimplification). I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications. 1. https://noop.dev/blog/build-apps-not-infrastructure/ https://noop.dev/blog/build-apps-not-infrastructure/

eezing 3y ago

Infrastructure as code can be useful for complex systems. These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.

TeamDman 3y ago

To avoid data loss in prod, you best be using lifecycle { prevent_destroy = true } https://developer.hashicorp.com/terraform/tutorials/state/resource-lifecycle#prevent-resource-deletion https://developer.hashicorp.com/terraform/tutorials/state/re... > trying to get some old module to work in a new cloud environment where a bunch of assumptions about how things are done no longer apply Square hole type energy. Just because you can coerce the code to run doesn't mean it's a good idea. New environment, new assumptions, new code. Copy-paste what's relevant, but in a new project. > getting totally confused because what they see in the code doesn’t match reality Terraform compares state to detect drift? When reality doesn't match code, you clobber reality or you update the code. > It Prevents Drift! > This is true if and only if, you only ever, ever, ever, create code to change things in your environment. It at least makes drift obvious, even if it requires attention for it to be addressed. > code is, in my opinion, really not good at being documentation Unlike comments, IaC "can't" be inaccurate after applying. > But in my experience, the effort to keep your code up-to-date is a sink for engineering time. If the code is out of date, it's because you made a click change and didn't update the code. When this happens, terraform tells you exactly what changed in simple scenarios, and in more complex scenarios you can use import blocks to rein things in. > Yes it’s a clickbait headline. You clicked on it, didn’t you? Got me good. The article is a good launchboard for discussion. Overall, the IaC scene has plenty of room for growth. Terraform import blocks and code generation is helping reduce the problem of "I clicked in the portal now I gotta update the code" because you can sync both ways. Code updates infra, infra can be used to generate code.

Jenkins2000 3y ago

Back in the olden days, I had a junior dev ask me if he could learn ops, I said sure, I was just a hacker myself. He wrote a script to make sure the hard drives in all the servers were good. A month or so later we rebooted a server and it wouldn't come back up. We finally rebooted another to see if we could learn something about the dead server. But it also wouldn't boot. Then to our horror we learned that the disk check script was writing and reading to the boot sector to make sure it was good. All of our servers were affected. Good times.

xorcist 3y ago

I've done IaC is various forms over a decade and I believe the term does more harm than good. It's not "Infrastructure", and it's not really "Code" either. "Infrastructure Configuration as Text" perhaps would be more descriptive. Or just stick with the samewhat older term "Configuration Management". These days, however, it's not really in vogue. Nobody thinks they need it anymore. They have Kubernetes and containers now. When you ask enough questions about how those are managed, it usually turns out some state is being managed somewhere outside of version control. I believe the pendulum will swing back in time, but in the meantime there will be K8s on my resume, not IaC.

moribvndvs 3y ago

Truly cursed take. Loved the “uh… ha ha, got’em!” walk-back in the addendum. Here’s an another anecdote for you, if that’s what we’re basing our arguments on: My prior team had dozens of services to deploy and they were all done in the console. It took hours to roll out deployments, people constantly fucked something up (wrong version, missed dependency, config variable, etc), you had spend sometimes up to an hour to get a picture of how a service was currently stood up, was almost completely opaque and thus extremely expensive to get new hires up to speed, obviated the ability to perform post deploy automation for smoke tests and such, required giving leads broad access to environments that needed to be locked down, and the best part was you had to do it all over again for each environment. All of this went completely away when I rolled it into CloudFormation, hooked it into CI/CD pipeline, and let a maintainer kick it off with the click of a button. I guess I must have been doing something wrong /s

zbentley 3y ago

A significant benefit of IaC is the work-savings it yields for recovery from a total-site-loss event, or the migration to multi-geo/multi-site. Sure, very few IaC environments are going to make that sort of thing seamless--bootstrapping new infra from nothing in a hurry is always going to turn up every skeleton in every closet and require a lot of manual work. But having even 80% IaC automation yields huge time savings in those rare-but-impactful "start from zero" cases. Having even just the easy/copy-paste parts of the infra encoded in automation can be the difference between hours and days for disaster recovery, or days and weeks for creating a new geo/staging/whatever environment.

whatindaheck 3y ago

Almost all the listed downsides of IaC are also downsides of _any_ code. I was starting to address the points one-by-one but that’s the gist of the entire article. The alternative is we all start using Alice to click and drag pieces of logic together to build software. This is the equivalent of building infra through a web UI. I don’t see the author’s point. Sure IaC isn’t a magical solution with zero problems but was anybody claim that?