3 ms·

Yea I think I would love if Terraform supported modern Kubernetes concepts. The plan-apply model is IMO the best. I don't really grok why people don't think Ter

Yea I think I would love if Terraform supported modern Kubernetes concepts. The plan-apply model is IMO the best. I don't really grok why people don't think Terraform should be used this way as I've seen in other comments here and outside of HN. At the end of the day I fail to see how managing Kubernetes resources with Terraform would be any different than AWS or GCP or whathaveyou resources.

antoncohen 9y ago
Terraform is pretty awful for Continuous Delivery. I inherited a deployment system were new app versions were deployed with Terraform, and I did quite a lot of automation to make it not totally suck from a workflow perspective, not it's still not the right tool. Devs should be able to merge a PR, and have it safely rollout to production (e.g., canary), possibly with a staging step and manual promotion if desired. They should have a simple ability to rollback to a previous version. Terraform typically would involve editing TF files to update a version, committing that TF change, opening and merge a PR for the change. The running plan, reading the output of plan, then applying the plan. If you have staging and stable environments, repeat the steps. If you want to rollback, repeat the steps. And it gets oh so much more complex. TF binary versions need to be globally consistent, because the version is stored in the state file. You basically need to only run TF from the CI/CD system, people can't be allowed to run apply locally because they might use a newer version of TF which would then break CI/CD which has the older version. So then you wrap TF commands with a Makefile that checks TF version, or you pin the TF version within the TF code. How many resources are you managing? Hundreds? Thousands? How many apps? Are they all deployed using the same state file? How long does it take to run a plan and apply? 10 minutes? 20 minutes? Same for a rollback... What if Bob want to deploy version 13 of app 'foo', but when he runs plan it says it will also update app 'bar' to version 27. But Bob doesn't know anything about bar, his team only works on foo. So you break up the TF files so every app has its own state file. Cool. But now you need to upgrade TF itself. And it is a major upgrade. Now you need to update the TF version pinned in the source files, and update the build servers to have the new TF, and run a TF command to upgrade on every state file. Then you start adding symlinks into the TF source tree, because TF doesn't have any sane way to reuse code. But there are slight changes that need to be made for each app, and TF doesn't allow variables in some parts of the code, and modules are OK be still too verbose, and you still end up with places you can't use variables. So no you start using templates to write you TF code, so you can reuse code, because 99% is the same for each app. Ugh. Terraform is great for defining infrastructure that rarely changes, and is managed by an infrastructure team. When you are trying to use it for app deployments, which happen multiple times a day, and are handles by app devs, it is a royal PITA.
- heavenlyhash 9y ago
  I'm not of a dissenting opinion on most of these experience reports, but > TF binary versions need to be globally consistent, honestly, shouldn't that kind of be your baseline goal for all tools that are on multiple hosts and taking part of your CI or development processes?
  - antoncohen 9y ago
    Yeah, sorry if I wasn't clear. I meant that the CI build servers do have a consistent version installed. But it is pretty hard to control what devs install on their laptops, and it is a little hard to write TF code without running `terraform plan` to test it, so a lot of devs may end up with TF installed. If they run `terraform apply` (and have access to write state) it could mess up the state file version. Of course that is solved by using `terraform { required_version = "= x.x.x" }`, but that plays into my point about having tons of state files, not being able to use variables, editing tons of files, and updating tons of states.
- p3llin0r3 9y ago
  Oh man I FORGOT about how bad code sharing is in terraform! Great point!
philsnow 9y ago
> The plan-apply model is IMO the best. What would be even better: a begin-commit model where you tell AWS (or $PROVIDER) that you're beginning a transactional change, you issue a bunch of API commands, and along the way the provider is modeling what that would cause the deployment to look like, and it's also checking subsequent actions' invariants / constraints against the "running total" of all the changes (the model) instead of the snapshot of how the deployment was before the "begin". I ended up having to create a "staging environment" at $WORK that's just another AWS account. The deploy flow for this one terraform project is: 1. run 'terraform plan' against production; this shows you syntax errors and _some_ semantic errors. 2. run 'terraform apply' against the staging AWS account; this will shake out more semantic errors, like an ec2 security group having too many rules. this step helps prevent you partially applying whatever change you're working on to production, since the production deploy would have failed when it gets to the ec2 security group change. 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. None of this would be necessary if AWS had some way of modeling deployments and checking invariants/conditions against models instead of checking them in situ. edit: something like: aws change create-changeset aws ec2 authorize-security-group-ingress --changeset-id c-12345abc ... aws ... --changeset-id c-12345abc aws change commit-changeset --changeset-id c-12345abc
- antoncohen 9y ago
  > 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. Oh, I've seen a few... 1. Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. 2. Access that is different between accounts. The oddest one was a bug in Packer that caused an AMI to get built with an "AWS Marketplace" version of Ubuntu. To use an AMI based off a Marketplace AMI you need to "buy" the Marketplace AMI per account (even though it is free). Someone had clicked the button in the staging account to "buy" the AMI, but then it fails to deploy to productions. 3. Random other failures. Like hitting naming length limits because "prod" is longer than "qa". Or like API rate limiting slowing down deploys in one account, causing them to timeout and the TF apply to fail, even thought plan passed and the apply worked in staging. This is can then land you in an odd state where re-running apply will destroy all of a resource, instead of updating it (taking down the whole service). Which then leads you to automate `terraform taint`... TF just wasn't designed to do app deployments.

antoncohen 9y ago

Terraform is pretty awful for Continuous Delivery. I inherited a deployment system were new app versions were deployed with Terraform, and I did quite a lot of automation to make it not totally suck from a workflow perspective, not it's still not the right tool. Devs should be able to merge a PR, and have it safely rollout to production (e.g., canary), possibly with a staging step and manual promotion if desired. They should have a simple ability to rollback to a previous version. Terraform typically would involve editing TF files to update a version, committing that TF change, opening and merge a PR for the change. The running plan, reading the output of plan, then applying the plan. If you have staging and stable environments, repeat the steps. If you want to rollback, repeat the steps. And it gets oh so much more complex. TF binary versions need to be globally consistent, because the version is stored in the state file. You basically need to only run TF from the CI/CD system, people can't be allowed to run apply locally because they might use a newer version of TF which would then break CI/CD which has the older version. So then you wrap TF commands with a Makefile that checks TF version, or you pin the TF version within the TF code. How many resources are you managing? Hundreds? Thousands? How many apps? Are they all deployed using the same state file? How long does it take to run a plan and apply? 10 minutes? 20 minutes? Same for a rollback... What if Bob want to deploy version 13 of app 'foo', but when he runs plan it says it will also update app 'bar' to version 27. But Bob doesn't know anything about bar, his team only works on foo. So you break up the TF files so every app has its own state file. Cool. But now you need to upgrade TF itself. And it is a major upgrade. Now you need to update the TF version pinned in the source files, and update the build servers to have the new TF, and run a TF command to upgrade on every state file. Then you start adding symlinks into the TF source tree, because TF doesn't have any sane way to reuse code. But there are slight changes that need to be made for each app, and TF doesn't allow variables in some parts of the code, and modules are OK be still too verbose, and you still end up with places you can't use variables. So no you start using templates to write you TF code, so you can reuse code, because 99% is the same for each app. Ugh. Terraform is great for defining infrastructure that rarely changes, and is managed by an infrastructure team. When you are trying to use it for app deployments, which happen multiple times a day, and are handles by app devs, it is a royal PITA.

heavenlyhash 9y ago
I'm not of a dissenting opinion on most of these experience reports, but > TF binary versions need to be globally consistent, honestly, shouldn't that kind of be your baseline goal for all tools that are on multiple hosts and taking part of your CI or development processes?
- antoncohen 9y ago
  Yeah, sorry if I wasn't clear. I meant that the CI build servers do have a consistent version installed. But it is pretty hard to control what devs install on their laptops, and it is a little hard to write TF code without running `terraform plan` to test it, so a lot of devs may end up with TF installed. If they run `terraform apply` (and have access to write state) it could mess up the state file version. Of course that is solved by using `terraform { required_version = "= x.x.x" }`, but that plays into my point about having tons of state files, not being able to use variables, editing tons of files, and updating tons of states.
p3llin0r3 9y ago
Oh man I FORGOT about how bad code sharing is in terraform! Great point!

philsnow 9y ago

> The plan-apply model is IMO the best. What would be even better: a begin-commit model where you tell AWS (or $PROVIDER) that you're beginning a transactional change, you issue a bunch of API commands, and along the way the provider is modeling what that would cause the deployment to look like, and it's also checking subsequent actions' invariants / constraints against the "running total" of all the changes (the model) instead of the snapshot of how the deployment was before the "begin". I ended up having to create a "staging environment" at $WORK that's just another AWS account. The deploy flow for this one terraform project is: 1. run 'terraform plan' against production; this shows you syntax errors and _some_ semantic errors. 2. run 'terraform apply' against the staging AWS account; this will shake out more semantic errors, like an ec2 security group having too many rules. this step helps prevent you partially applying whatever change you're working on to production, since the production deploy would have failed when it gets to the ec2 security group change. 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. None of this would be necessary if AWS had some way of modeling deployments and checking invariants/conditions against models instead of checking them in situ. edit: something like: aws change create-changeset aws ec2 authorize-security-group-ingress --changeset-id c-12345abc ... aws ... --changeset-id c-12345abc aws change commit-changeset --changeset-id c-12345abc

antoncohen 9y ago
> 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. Oh, I've seen a few... 1. Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. 2. Access that is different between accounts. The oddest one was a bug in Packer that caused an AMI to get built with an "AWS Marketplace" version of Ubuntu. To use an AMI based off a Marketplace AMI you need to "buy" the Marketplace AMI per account (even though it is free). Someone had clicked the button in the staging account to "buy" the AMI, but then it fails to deploy to productions. 3. Random other failures. Like hitting naming length limits because "prod" is longer than "qa". Or like API rate limiting slowing down deploys in one account, causing them to timeout and the TF apply to fail, even thought plan passed and the apply worked in staging. This is can then land you in an odd state where re-running apply will destroy all of a resource, instead of updating it (taking down the whole service). Which then leads you to automate `terraform taint`... TF just wasn't designed to do app deployments.