3 ms·

> The plan-apply model is IMO the best. What would be even better: a begin-commit model where you tell AWS (or $PROVIDER) that you're beginning a transactiona

> The plan-apply model is IMO the best. What would be even better: a begin-commit model where you tell AWS (or $PROVIDER) that you're beginning a transactional change, you issue a bunch of API commands, and along the way the provider is modeling what that would cause the deployment to look like, and it's also checking subsequent actions' invariants / constraints against the "running total" of all the changes (the model) instead of the snapshot of how the deployment was before the "begin". I ended up having to create a "staging environment" at $WORK that's just another AWS account. The deploy flow for this one terraform project is: 1. run 'terraform plan' against production; this shows you syntax errors and _some_ semantic errors. 2. run 'terraform apply' against the staging AWS account; this will shake out more semantic errors, like an ec2 security group having too many rules. this step helps prevent you partially applying whatever change you're working on to production, since the production deploy would have failed when it gets to the ec2 security group change. 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. None of this would be necessary if AWS had some way of modeling deployments and checking invariants/conditions against models instead of checking them in situ. edit: something like: aws change create-changeset aws ec2 authorize-security-group-ingress --changeset-id c-12345abc ... aws ... --changeset-id c-12345abc aws change commit-changeset --changeset-id c-12345abc

antoncohen 9y ago
> 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. Oh, I've seen a few... 1. Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. 2. Access that is different between accounts. The oddest one was a bug in Packer that caused an AMI to get built with an "AWS Marketplace" version of Ubuntu. To use an AMI based off a Marketplace AMI you need to "buy" the Marketplace AMI per account (even though it is free). Someone had clicked the button in the staging account to "buy" the AMI, but then it fails to deploy to productions. 3. Random other failures. Like hitting naming length limits because "prod" is longer than "qa". Or like API rate limiting slowing down deploys in one account, causing them to timeout and the TF apply to fail, even thought plan passed and the apply worked in staging. This is can then land you in an odd state where re-running apply will destroy all of a resource, instead of updating it (taking down the whole service). Which then leads you to automate `terraform taint`... TF just wasn't designed to do app deployments.
- philsnow 9y ago
  > Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. ah yeah, I had forgotten that I had to go through our TAM to get the staging account set up with all the same limits so that it was an accurate reflection. > "prod" is longer than "qa" Ugh, painful. > TF just wasn't designed to do app deployments agreed, I've gotten more and more interested in cloudformation over TF (for AWS-only deployments) over the last two years of using TF.

antoncohen 9y ago

> 3. finally, run 'terraform apply' on production. I've yet to find an error that made it to this step that didn't get caught in one of the previous two steps. Oh, I've seen a few... 1. Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. 2. Access that is different between accounts. The oddest one was a bug in Packer that caused an AMI to get built with an "AWS Marketplace" version of Ubuntu. To use an AMI based off a Marketplace AMI you need to "buy" the Marketplace AMI per account (even though it is free). Someone had clicked the button in the staging account to "buy" the AMI, but then it fails to deploy to productions. 3. Random other failures. Like hitting naming length limits because "prod" is longer than "qa". Or like API rate limiting slowing down deploys in one account, causing them to timeout and the TF apply to fail, even thought plan passed and the apply worked in staging. This is can then land you in an odd state where re-running apply will destroy all of a resource, instead of updating it (taking down the whole service). Which then leads you to automate `terraform taint`... TF just wasn't designed to do app deployments.

philsnow 9y ago
> Account limits that are different between accounts, e.g., number of ELB, ASGs, EIPs, etc. allowed. ah yeah, I had forgotten that I had to go through our TAM to get the staging account set up with all the same limits so that it was an accurate reflection. > "prod" is longer than "qa" Ugh, painful. > TF just wasn't designed to do app deployments agreed, I've gotten more and more interested in cloudformation over TF (for AWS-only deployments) over the last two years of using TF.