3 ms·

The complexity of cloud deployments tends to grow wildly over time. What starts as an ECR repo with a single ECS deployment turns into Route 53 zones, S3 bucket

The complexity of cloud deployments tends to grow wildly over time. What starts as an ECR repo with a single ECS deployment turns into Route 53 zones, S3 buckets, ELBs, multiple deployments of ECS, security policies, the odd EC2 instance (there's always one somehwere), etc, etc. Terraform gives you a commmon language to make sense of it all that can grow as your cloud infra does. When combined with git and CI/CD it's also an amazing self-service experince. For example you can put the Terraform code that describes your environment in a git repo, and allow any employee to open pull requests, and deploy changes on merge automaically, and require IT approval to merge. Now any engineer can self-service request access to a prod environment (by modifying IAM in Terraform), or configure a production deployment without ever needing actual access to prod. IT gets an audit log, they get a control gate (the code review), and engineers get to self-service changes which reduces the load on IT.

dolni 4y ago
Is self-service Terraform as you describe actually successful? I use Terraform frequently and I have zero confidence that this process works except for the simplest changes. For one, building out anything of complexity in Terraform often involves some trial and error to make sure all the resources are wired together correctly. For two, there are some Terraform changes that can incur downtime if you just let Terraform destroy and create resources. The solution I employ for this is what I not-so-affectionately call "Terraform surgery". It usually entails standing up the new resource(s), cutting over to them, and then destroying the old ones. Sometimes I will manually remove some stuff from the state file and delete it once Terraform has performed the cutover. That stuff is possible to do in CI, but it would be inefficient and painful. To me, the biggest benefit of Terraform is managing changes to cloud resources. You know what resources are added when, why, and by whom. You have a mechanism for promoting changes between environments confidently. And if your infra ever blew up, you would be able to put stuff back.
- cybrexalpha 4y ago
  I've seen it work best with an embedded plan. So engineers open a PR with terraform changes, the CI runs `terraform plan`, and comments on the PR with that diff. So you can see if your terraform is valid, if you're going to be destroying things you shouldn't, etc. Larger, more complex testing is generally done by locally invoking your terraform on a testing environment. How this is setup kind of depends, but I've seen this work well for testing out complex changes before going to the PR & Plan stage.
- acdha 4y ago
  > For one, building out anything of complexity in Terraform often involves some trial and error to make sure all the resources are wired together correctly. This seems like misattribution: you do that everywhere when you’re doing something new - the problem is the underlying gap between the understanding of the problem which you started with and what the underlying services actually require. You can avoid most of that by using modules to reduce the amount of new structure you need to develop and static linters (tflint, tfsec, etc.) to flag things in advance. > For two, there are some Terraform changes that can incur downtime if you just let Terraform destroy and create resources. This should never be necessary due to Terraform - that’s one of the major advantages over older tools like AWS CloudFormation. If you’re seeing this as a problem where it’s not imposed by the platform (e.g. many AWS resources won’t allow you to change the name after creation) you can use lifecycle rules to avoid forcing re-creation where that makes sense or to create the replacement first & switch over before deleting the old resource. If you relocated the definitions in your code, you can tell Terraform how to migrate old state non-destructively.

dolni 4y ago

Is self-service Terraform as you describe actually successful? I use Terraform frequently and I have zero confidence that this process works except for the simplest changes. For one, building out anything of complexity in Terraform often involves some trial and error to make sure all the resources are wired together correctly. For two, there are some Terraform changes that can incur downtime if you just let Terraform destroy and create resources. The solution I employ for this is what I not-so-affectionately call "Terraform surgery". It usually entails standing up the new resource(s), cutting over to them, and then destroying the old ones. Sometimes I will manually remove some stuff from the state file and delete it once Terraform has performed the cutover. That stuff is possible to do in CI, but it would be inefficient and painful. To me, the biggest benefit of Terraform is managing changes to cloud resources. You know what resources are added when, why, and by whom. You have a mechanism for promoting changes between environments confidently. And if your infra ever blew up, you would be able to put stuff back.

cybrexalpha 4y ago
I've seen it work best with an embedded plan. So engineers open a PR with terraform changes, the CI runs `terraform plan`, and comments on the PR with that diff. So you can see if your terraform is valid, if you're going to be destroying things you shouldn't, etc. Larger, more complex testing is generally done by locally invoking your terraform on a testing environment. How this is setup kind of depends, but I've seen this work well for testing out complex changes before going to the PR & Plan stage.
acdha 4y ago
> For one, building out anything of complexity in Terraform often involves some trial and error to make sure all the resources are wired together correctly. This seems like misattribution: you do that everywhere when you’re doing something new - the problem is the underlying gap between the understanding of the problem which you started with and what the underlying services actually require. You can avoid most of that by using modules to reduce the amount of new structure you need to develop and static linters (tflint, tfsec, etc.) to flag things in advance. > For two, there are some Terraform changes that can incur downtime if you just let Terraform destroy and create resources. This should never be necessary due to Terraform - that’s one of the major advantages over older tools like AWS CloudFormation. If you’re seeing this as a problem where it’s not imposed by the platform (e.g. many AWS resources won’t allow you to change the name after creation) you can use lifecycle rules to avoid forcing re-creation where that makes sense or to create the replacement first & switch over before deleting the old resource. If you relocated the definitions in your code, you can tell Terraform how to migrate old state non-destructively.