3 ms·

Basically agree with the article. I've used direct cloud formation, AWS SAM, Ansible, terraform, and AWS CDK to spin up infrastructure... My hard line opinion

Basically agree with the article. I've used direct cloud formation, AWS SAM, Ansible, terraform, and AWS CDK to spin up infrastructure... My hard line opinion is that if something NEEDS state management to exist and update, it's a pet, treat it like a pet. Don't mix pets with the rest of your automated machinery except to the minimum extent required, when absolutely necessary. We had to rewrite an Ansible role because a patch level version upgrade on the recommended community package started destroying security groups... Another time we had to roll back our deployment Ansible image and update 30 repos because a minor level version bump in another Ansible recommended package suddenly required log groups to have an expiration value set in AWS or the module fell over with an null reference on the AWS lookup/comparison... So we couldn't use the latest version to fix it. I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... I also encourage modular designs, any of which (except cdn/db, and dns related stuff, in my use cases) can be torn down and re-built with only the brief outage nonexistence causes.. we only have done that once in 3 years, and we believe the issue was actually on AWS's internal side anyways. I've spun up over 260,000 vms over 4-5 years with one cloud formation template and the basic SDK call, and we've never bothered to convert it to another tool because it's never broken.. we occasionally tweak it, use gp3 instead of gp2, etc, but it's never needed us to unexpectedly side track a sprint for 1-3 days

dangus 4y ago
> I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... This is also what you do with Terraform: that's what the Terraform Cloud product is about (or you can just build a CI pipeline in your tool of choice with a blog post's amount of work). It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world. Terraform state also comes in handy as an audit trail. Through versions of your state file, you can see how your infrastructure changed in the past (You could use S3 versioning or you could have Terraform Cloud manage the state file automatically). Terraform Cloud turns that concept into a relatively powerful feature. It becomes a compliance record: who changed what and how it changed, along with diff files. As another commenter pointed out, it really seems like the alternative tool you're using (CloudFormation) has its own concept of state, but it's just hiding it from you. Pulumi also has a concept of state. I haven't really seen an infrastructure automation tool without that concept. Ultimately, these tools need to have some way to track what resource the code is referencing. Call that metadata or state, you have to track something. And if you come up with something that somehow avoids it, I have to ask why? It seems like trying really hard to make a car without doors that also prevents rain from coming inside: why not just leave the doors on the car?
- ldoughty 4y ago
  > It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world Certainly, it's hard to start it from day one. But it's not that difficult to move into it. If you have buy-in from your developer/operations team. We started with API and CLI calls for our beta version. Our next migration was to use partial Ansible control and we morph that into almost a monolith because of interconnected pieces that were typically required with this design. But it really didn't need to be a monolith, we just wanted to link things together and building a giant monolith was easier to make those references. So we then split up into smaller Ansible playbooks, and we did lookups to create the linkage, which roughly broke the monolithic pattern and allowed us to do smaller deploys. But we still ran into breaking changes unexpectedly. So we decided to abandon the months of effort we put into Ansible and we started looking at terraform, because one of their salespeople promised our management team that it is cloud agnostic and we would only have to write once. After a week or two looking into that, we realized that we were basically just going to have to remake Ansible modules and we rarely weren't saving anything by migrating. Granted this was two and a half to three years ago. Things might have changed. We then switch to SAM, and as we did that we extracted the Ansible side out of our deployment and we started redeploying brand new small SAM stacks, and started treating almost everything of our infrastructure as sheep instead of cattle, we completely redeployed our launch configurations our cluster are lambda functions, basically everything except the database, DNS, and the CDN with every deploy. This basically removed state is being an issue because the state is only needed for the first deploy, and we don't technically change the stacks afterwards since we simply replace them every time. For us, this also meant we could easily test and roll back if needed. Since we don't need to change the state of the stack back to its previous state, we simply changed the pointer to the previous stack, which typically was a DNS state change. But like I said, we focused on our states being only around DNS, etc. Which, almost isn't stateful, because our SAM deploys would insert zero weighted DNS records, and The state management is really just adjusting the weighted values.
erikerikson 4y ago
Right here with you on this, I can add ARM templates (in complete mode) to the list. I might suggest the Serverless framework (no serverless needed) which lets you write CFTs in most formats you prefer, use variables (including pulling config from S3), provides the canned scripts you otherwise add, and so on. I wish MS would unbreak it's plugin to Serverless to support full ARM. As others have written, there is state in the CFT/ARM/etc. services via named deploys. As you note, that allows for the deletion of removed things (including things for which you've specified another name). That's great. The challenge with Terraform state use is that it is often too tetchy about what it doesn't know about and often reacts by tearing down and rebuilding assets, causing unavailability and data loss. It was even observed doing this for an entire stack due to a transitory network failure at PayPal. I loved --auto-approve because I don't want to reintroduce human failure and friction into the mix but Terraform's models and dependency on "community or when we get to it" means it just isn't safe the way the native provisioning services are. As you note, alignment of incentives and all.
klooney 4y ago
Cloudformation needs state management to exist and update, and you can get it wedged, even if you aren't making any changes out of band.
- ldoughty 4y ago
  That's a fair point, but it shifts the management to the entity that is best equipped to manage it.. and at least with SAM, which makes to the bulk of our deploys, we've not had issues as long as the first deploy didn't error (since you can't fix a broken initial deploy) When I've used CDK, I actually strip the boot strap and avoid the functions that add state, and it makes great templates... But as I mentioned before, my use case might be different and less prone to issues.. I'm mostly using ec2 servers, lambda, and the typical supporting services (iam, cloud watch, s3, CloudFront, sqs, etc.)
rwiggins 4y ago
That mirrors my experience too. At my last job, we had... a lot of annoyances with Ansible version upgrades. There is a lot of churn there. Stuff unexpectedly broke pretty often, and even when it didn't, it felt like a conveyor belt of constant deprecated features and warnings.
m_mueller 4y ago
IMO CF (and dependent APIs that use CF servers internally) is strictly worse since it has state, but that state lives server side and the only way to influence it seems to be to mock around with destroy and rebuild and hope for the best (in my experience it can easily get into a deadlock). TF at least directly talks to the provider APIs like S3 and EC2 and you get their error messages when something goes wrong. When TF locks itself up I know how to deal with it myself without needing anyone‘s support (delete the respective key in the state backing KVS). That being said I agree with OP - making it stateless would be even better.

dangus 4y ago

> I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... This is also what you do with Terraform: that's what the Terraform Cloud product is about (or you can just build a CI pipeline in your tool of choice with a blog post's amount of work). It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world. Terraform state also comes in handy as an audit trail. Through versions of your state file, you can see how your infrastructure changed in the past (You could use S3 versioning or you could have Terraform Cloud manage the state file automatically). Terraform Cloud turns that concept into a relatively powerful feature. It becomes a compliance record: who changed what and how it changed, along with diff files. As another commenter pointed out, it really seems like the alternative tool you're using (CloudFormation) has its own concept of state, but it's just hiding it from you. Pulumi also has a concept of state. I haven't really seen an infrastructure automation tool without that concept. Ultimately, these tools need to have some way to track what resource the code is referencing. Call that metadata or state, you have to track something. And if you come up with something that somehow avoids it, I have to ask why? It seems like trying really hard to make a car without doors that also prevents rain from coming inside: why not just leave the doors on the car?

ldoughty 4y ago
> It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world Certainly, it's hard to start it from day one. But it's not that difficult to move into it. If you have buy-in from your developer/operations team. We started with API and CLI calls for our beta version. Our next migration was to use partial Ansible control and we morph that into almost a monolith because of interconnected pieces that were typically required with this design. But it really didn't need to be a monolith, we just wanted to link things together and building a giant monolith was easier to make those references. So we then split up into smaller Ansible playbooks, and we did lookups to create the linkage, which roughly broke the monolithic pattern and allowed us to do smaller deploys. But we still ran into breaking changes unexpectedly. So we decided to abandon the months of effort we put into Ansible and we started looking at terraform, because one of their salespeople promised our management team that it is cloud agnostic and we would only have to write once. After a week or two looking into that, we realized that we were basically just going to have to remake Ansible modules and we rarely weren't saving anything by migrating. Granted this was two and a half to three years ago. Things might have changed. We then switch to SAM, and as we did that we extracted the Ansible side out of our deployment and we started redeploying brand new small SAM stacks, and started treating almost everything of our infrastructure as sheep instead of cattle, we completely redeployed our launch configurations our cluster are lambda functions, basically everything except the database, DNS, and the CDN with every deploy. This basically removed state is being an issue because the state is only needed for the first deploy, and we don't technically change the stacks afterwards since we simply replace them every time. For us, this also meant we could easily test and roll back if needed. Since we don't need to change the state of the stack back to its previous state, we simply changed the pointer to the previous stack, which typically was a DNS state change. But like I said, we focused on our states being only around DNS, etc. Which, almost isn't stateful, because our SAM deploys would insert zero weighted DNS records, and The state management is really just adjusting the weighted values.

erikerikson 4y ago

Right here with you on this, I can add ARM templates (in complete mode) to the list. I might suggest the Serverless framework (no serverless needed) which lets you write CFTs in most formats you prefer, use variables (including pulling config from S3), provides the canned scripts you otherwise add, and so on. I wish MS would unbreak it's plugin to Serverless to support full ARM. As others have written, there is state in the CFT/ARM/etc. services via named deploys. As you note, that allows for the deletion of removed things (including things for which you've specified another name). That's great. The challenge with Terraform state use is that it is often too tetchy about what it doesn't know about and often reacts by tearing down and rebuilding assets, causing unavailability and data loss. It was even observed doing this for an entire stack due to a transitory network failure at PayPal. I loved --auto-approve because I don't want to reintroduce human failure and friction into the mix but Terraform's models and dependency on "community or when we get to it" means it just isn't safe the way the native provisioning services are. As you note, alignment of incentives and all.

klooney 4y ago

Cloudformation needs state management to exist and update, and you can get it wedged, even if you aren't making any changes out of band.

ldoughty 4y ago
That's a fair point, but it shifts the management to the entity that is best equipped to manage it.. and at least with SAM, which makes to the bulk of our deploys, we've not had issues as long as the first deploy didn't error (since you can't fix a broken initial deploy) When I've used CDK, I actually strip the boot strap and avoid the functions that add state, and it makes great templates... But as I mentioned before, my use case might be different and less prone to issues.. I'm mostly using ec2 servers, lambda, and the typical supporting services (iam, cloud watch, s3, CloudFront, sqs, etc.)

rwiggins 4y ago

That mirrors my experience too. At my last job, we had... a lot of annoyances with Ansible version upgrades. There is a lot of churn there. Stuff unexpectedly broke pretty often, and even when it didn't, it felt like a conveyor belt of constant deprecated features and warnings.

m_mueller 4y ago

IMO CF (and dependent APIs that use CF servers internally) is strictly worse since it has state, but that state lives server side and the only way to influence it seems to be to mock around with destroy and rebuild and hope for the best (in my experience it can easily get into a deadlock). TF at least directly talks to the provider APIs like S3 and EC2 and you get their error messages when something goes wrong. When TF locks itself up I know how to deal with it myself without needing anyone‘s support (delete the respective key in the state backing KVS). That being said I agree with OP - making it stateless would be even better.