8 ms·

Infrastructure as Code, Part One

145 points by based2 8y ago

mverwijs 8y ago
Nice piece. Looking forward to Part II. What I am missing (often, in these type of articles as well as in actual production environments) is the fact that if you develop (infrastructure) code, you also need to test your (infrastructure) code. Which means you need actual infrastructure to test on. In my case, this means network equipment, storage equipment and actual physical servers. If you're in a cloud, this means you need a seperate account on a seperate creditcard and start from there to build up the infra that Dev and Ops can start deploying their infra-as-code on. And this test-infrastructure is not the test environments other teams run their tests on. If that is not available, automating your infrastructure can be dangerous at best. Since you cannot properly test your code. And your code will rot.
- kylek 8y ago
  Part 2 is linked at the bottom :) looking forward to part _3_!
- stingraycharles 8y ago
  I found that Kubernetes + minikube (or a variant of that) is a fairly straightforward way to handle this. Teams / developers can easily set up a local testing environment, product owners can QA that way, etcetera. This of course depends on your level of lock-in with various cloud environments.
  - mverwijs 8y ago
    Yup - that works pretty well. And gives developers some insight into what is required to get stuff working. It does assume no hardware or complex networking needs to be handled. And there is the point of observability. When there is a proper testing ground for developers that is as-production, it enables developers to dig into and mess with logging, tracing, debugging of all sorts. This adds value by providing developers insight into what a reliability engineer (or whatever they call sysadmins these days) needs to provide whatever service it is that the developers' code is part of.
  - pm90 8y ago
    IaC tools often handle more than kubernetes, but agreed that k8s is a fantastic way to get reproducible behavior which is absolutely imperative for testing. This is kinda why I love Google Cloud and don't see myself moving to another cloud provider until they match GKE. I want all developers to throw everything into GKE, and Operations manages only the VPC's, firewalls etc. Developers get complete ownership over compute (and networking within the cluster) while broader network management can still be managed by an operations team.
- scarface74 8y ago
  Why would you need a separate credit card? It’s easy enough to set up separate accounts under an Organization with shared billing and with rules that work across accounts.
  - mverwijs 8y ago
    Because I want to limit the impact of maxing out a creditcard to one environment. And I want engineers to be able to futz about with all cloud services available, without having to worry about any negative impact on production. And finally: What happens when $cloud_provider makes changes to the accounts interface and you want to mess around with those new features, without hitting production? Give your future-self a break, and make sure you can futz around on any and every layer. Another common practise is using seperate domainnames. Don't use 'dev.news.ycombinator.com'. Instead, use 'news.ycombinator.dev'. This frees you up for messing around with the API of the DNS provider. And when switching DNS provider, test whatever automation you have in place for this.
    - scarface74 8y ago
      Because I want to limit the impact of maxing out a creditcard to one environment. Just because you maxed out the credit card doesn’t mean that you don’t still owe the money if you go over. That’s what billing alerts are for. And I want engineers to be able to futz about with all cloud services available, without having to worry about any negative impact on production. And finally: What happens when $cloud_provider makes changes to the accounts interface and you want to mess around with those new features, without hitting production? Give your future-self a break, and make sure you can futz around on any and every layer. That’s what the separate accounts are for but you don’t need a separate card and you still should be using an organization. This frees you up for messing around with the API of the DNS provider. And when switching DNS provider, test whatever automation you have in place for this. Why isn’t your DNS provider AWS with Route 53 where the separate domains would be in separate accounts with separate permissions and separate access keys/roles per account?
      - mverwijs 8y ago
royjacobs 8y ago
What I find unfortunate about infrastructure-as-code tooling is that a lot of the tooling isn't actually using code, but instead uses esoteric configuration languages. Indeed, the article refers to Terraform with its custom syntax. Imho tools should use actual code (whether it's TypeScript or Kotlin or whatever) instead of reinventing constructs like loops and string interpolation. Thankfully these tools are getting more popular, because frankly I can't stand configuring another Kubernetes or GCP resource using a huge block of copy/pasted YAML.
- robbyt 8y ago
  This was called Chef, and every time I've encountered a Chef infrastructure it's been a complete mess.
- amq 8y ago
  Could you name some of those code-based tools?
  - ryanworl 8y ago
    https://www.pulumi.com/ https://www.pulumi.com/
  - closeparen 8y ago
    Start with hitting cloud provider APIs directly, then looking for value-add on top of that.
  - cle 8y ago
    AWS CDK: https://github.com/awslabs/aws-cdk https://github.com/awslabs/aws-cdk
  - rad_gruchalski 8y ago
    One of the oldest examples of this: https://github.com/infochimps-labs/ironfan https://github.com/infochimps-labs/ironfan. But yeah, Chef. Examples: https://github.com/infochimps-labs/ironfan-pantry/blob/master/cookbooks/volumes/README.md https://github.com/infochimps-labs/ironfan-pantry/blob/maste.... I used to use this heavily back in 2013, 2014. Infochimps got picked up by eBay, afair. Hence why this was never developed further.
cpach 8y ago
Nice article! What Crate advocates here might be common sense to a lot of people here on HN, but I can assure you that there are lots of people in leading positions out there who has no clue about this paradigm. Hopefully some of them will read this article.
- 1_player 8y ago
  I might be one of those people that doesn't completely grok IaC yet. I understand the necessity of configuration tools such as Ansible, I wouldn't live without it, but in the case of my company (~15 apps, 20 servers) I still don't understand the use case of orchestration tools such as Terraform. All our servers configuration is managed through Ansible, apps are containerised and run on Kubernetes (on CoreOS, so even less configuration required), apps are deployed automatically with CI scripts. Why would I need to describe the hardware/infrastructure as code? I create VPSes manually, once in a blue moon, and they just get added to our K8S cluster, using the same Ansible template. It takes 2 minutes from VPS creation to adding the new node to the cluster. It'd be nice to describe in code the exact hardware provisioning, but that can be done easily with a couple lines in the internal documentation. And actually, having heterogenous server sizes might be a feature, to be able to schedule less demanding apps on less powerful and cheaper servers. What benefit would Terraform give me? Not being snarky, I just don't know how to fit it in our process.
  - zedpm 8y ago
    Terraform really shines when you have a complex environment that depends on various external services. I don't know enough about your architecture to comment on the benefits for you, but for a generic web services environment on, say, AWS, you have network configuration (VPC, subnets, security groups), EC2, load balancers, S3 buckets, RDS instances (managed databases), DNS, and possibly a number of other services which need to be set up for everything to work. Explicitly defining what services your product depends on and precisely how they're to be configured is extremely valuable. You certainly can write Ansible playbooks to create all of those resources if that works better for your team, but generally it's better to draw a line between configuring your VM/container/server and provisioning all the infrastructure it depends on.
LaserToy 8y ago
I’m sorry, but infrastructure as a code is a fake news. Most popular tools offer infrastructure as a config with some horrible and limited scripting language (in some cases even JSP doesn’t look like a horrible idea - hello helm). Declarative VS Imperative Holywar is getting a bit old (like any other tech war).
- alkonaut 8y ago
  The best language is “the one you know” which is why I think pulumi looks like the best answer so far.
oogali 8y ago
You spend the rest of the day investigating what went wrong. Eventually, you figure it out. Somebody logged on to the third machine last week and manually updated some of the software. These changes were never propagated to the other servers, or back to the staging environment, or dev environments. I'm sorry -- how many people are still doing this in 2019?
- detaro 8y ago
  Many, especially for tweaks after installation.

145 points by based2 8y ago

Nice piece. Looking forward to Part II. What I am missing (often, in these type of articles as well as in actual production environments) is the fact that if you develop (infrastructure) code, you also need to test your (infrastructure) code. Which means you need actual infrastructure to test on. In my case, this means network equipment, storage equipment and actual physical servers. If you're in a cloud, this means you need a seperate account on a seperate creditcard and start from there to build up the infra that Dev and Ops can start deploying their infra-as-code on. And this test-infrastructure is not the test environments other teams run their tests on. If that is not available, automating your infrastructure can be dangerous at best. Since you cannot properly test your code. And your code will rot.

kylek 8y ago
Part 2 is linked at the bottom :) looking forward to part _3_!
stingraycharles 8y ago
I found that Kubernetes + minikube (or a variant of that) is a fairly straightforward way to handle this. Teams / developers can easily set up a local testing environment, product owners can QA that way, etcetera. This of course depends on your level of lock-in with various cloud environments.
- mverwijs 8y ago
  Yup - that works pretty well. And gives developers some insight into what is required to get stuff working. It does assume no hardware or complex networking needs to be handled. And there is the point of observability. When there is a proper testing ground for developers that is as-production, it enables developers to dig into and mess with logging, tracing, debugging of all sorts. This adds value by providing developers insight into what a reliability engineer (or whatever they call sysadmins these days) needs to provide whatever service it is that the developers' code is part of.
- pm90 8y ago
  IaC tools often handle more than kubernetes, but agreed that k8s is a fantastic way to get reproducible behavior which is absolutely imperative for testing. This is kinda why I love Google Cloud and don't see myself moving to another cloud provider until they match GKE. I want all developers to throw everything into GKE, and Operations manages only the VPC's, firewalls etc. Developers get complete ownership over compute (and networking within the cluster) while broader network management can still be managed by an operations team.
scarface74 8y ago
Why would you need a separate credit card? It’s easy enough to set up separate accounts under an Organization with shared billing and with rules that work across accounts.
- mverwijs 8y ago
  Because I want to limit the impact of maxing out a creditcard to one environment. And I want engineers to be able to futz about with all cloud services available, without having to worry about any negative impact on production. And finally: What happens when $cloud_provider makes changes to the accounts interface and you want to mess around with those new features, without hitting production? Give your future-self a break, and make sure you can futz around on any and every layer. Another common practise is using seperate domainnames. Don't use 'dev.news.ycombinator.com'. Instead, use 'news.ycombinator.dev'. This frees you up for messing around with the API of the DNS provider. And when switching DNS provider, test whatever automation you have in place for this.
  - scarface74 8y ago
    Because I want to limit the impact of maxing out a creditcard to one environment. Just because you maxed out the credit card doesn’t mean that you don’t still owe the money if you go over. That’s what billing alerts are for. And I want engineers to be able to futz about with all cloud services available, without having to worry about any negative impact on production. And finally: What happens when $cloud_provider makes changes to the accounts interface and you want to mess around with those new features, without hitting production? Give your future-self a break, and make sure you can futz around on any and every layer. That’s what the separate accounts are for but you don’t need a separate card and you still should be using an organization. This frees you up for messing around with the API of the DNS provider. And when switching DNS provider, test whatever automation you have in place for this. Why isn’t your DNS provider AWS with Route 53 where the separate domains would be in separate accounts with separate permissions and separate access keys/roles per account?
    - mverwijs 8y ago

royjacobs 8y ago

What I find unfortunate about infrastructure-as-code tooling is that a lot of the tooling isn't actually using code, but instead uses esoteric configuration languages. Indeed, the article refers to Terraform with its custom syntax. Imho tools should use actual code (whether it's TypeScript or Kotlin or whatever) instead of reinventing constructs like loops and string interpolation. Thankfully these tools are getting more popular, because frankly I can't stand configuring another Kubernetes or GCP resource using a huge block of copy/pasted YAML.

robbyt 8y ago
This was called Chef, and every time I've encountered a Chef infrastructure it's been a complete mess.
amq 8y ago
Could you name some of those code-based tools?
- ryanworl 8y ago
  https://www.pulumi.com/ https://www.pulumi.com/
- closeparen 8y ago
  Start with hitting cloud provider APIs directly, then looking for value-add on top of that.
- cle 8y ago
  AWS CDK: https://github.com/awslabs/aws-cdk https://github.com/awslabs/aws-cdk
- rad_gruchalski 8y ago
  One of the oldest examples of this: https://github.com/infochimps-labs/ironfan https://github.com/infochimps-labs/ironfan. But yeah, Chef. Examples: https://github.com/infochimps-labs/ironfan-pantry/blob/master/cookbooks/volumes/README.md https://github.com/infochimps-labs/ironfan-pantry/blob/maste.... I used to use this heavily back in 2013, 2014. Infochimps got picked up by eBay, afair. Hence why this was never developed further.

cpach 8y ago

Nice article! What Crate advocates here might be common sense to a lot of people here on HN, but I can assure you that there are lots of people in leading positions out there who has no clue about this paradigm. Hopefully some of them will read this article.

1_player 8y ago
I might be one of those people that doesn't completely grok IaC yet. I understand the necessity of configuration tools such as Ansible, I wouldn't live without it, but in the case of my company (~15 apps, 20 servers) I still don't understand the use case of orchestration tools such as Terraform. All our servers configuration is managed through Ansible, apps are containerised and run on Kubernetes (on CoreOS, so even less configuration required), apps are deployed automatically with CI scripts. Why would I need to describe the hardware/infrastructure as code? I create VPSes manually, once in a blue moon, and they just get added to our K8S cluster, using the same Ansible template. It takes 2 minutes from VPS creation to adding the new node to the cluster. It'd be nice to describe in code the exact hardware provisioning, but that can be done easily with a couple lines in the internal documentation. And actually, having heterogenous server sizes might be a feature, to be able to schedule less demanding apps on less powerful and cheaper servers. What benefit would Terraform give me? Not being snarky, I just don't know how to fit it in our process.
- zedpm 8y ago
  Terraform really shines when you have a complex environment that depends on various external services. I don't know enough about your architecture to comment on the benefits for you, but for a generic web services environment on, say, AWS, you have network configuration (VPC, subnets, security groups), EC2, load balancers, S3 buckets, RDS instances (managed databases), DNS, and possibly a number of other services which need to be set up for everything to work. Explicitly defining what services your product depends on and precisely how they're to be configured is extremely valuable. You certainly can write Ansible playbooks to create all of those resources if that works better for your team, but generally it's better to draw a line between configuring your VM/container/server and provisioning all the infrastructure it depends on.

LaserToy 8y ago

I’m sorry, but infrastructure as a code is a fake news. Most popular tools offer infrastructure as a config with some horrible and limited scripting language (in some cases even JSP doesn’t look like a horrible idea - hello helm). Declarative VS Imperative Holywar is getting a bit old (like any other tech war).

alkonaut 8y ago
The best language is “the one you know” which is why I think pulumi looks like the best answer so far.

oogali 8y ago

You spend the rest of the day investigating what went wrong. Eventually, you figure it out. Somebody logged on to the third machine last week and manually updated some of the software. These changes were never propagated to the other servers, or back to the staging environment, or dev environments. I'm sorry -- how many people are still doing this in 2019?

detaro 8y ago
Many, especially for tweaks after installation.