8 ms·
Two years with CloudFormation: lessons learned
- alecbenzer 8y agoHow does CloudFormation compare with using something like ansible to manage AWS environments?
- kokey 8y agoMy opinion is that just because you can interact with AWS APIs with Ansible it means that you should. I think it's good to use AWS interaction with Ansible for things like dynamic inventory and the orchestration of certain tasks (e.g. stop a group of instances, switch load balancer config between blue/green deployments, etc.). That said I don't think it's that much worse than Cloudformation because it suffers from the same lack of statelessness and idempotency that you need to engineer around.
- unkoman 8y agoCloudformation is infrastructure management, not configuration management. Both Ansible and Cloudformation can be used for both in different ways, but usually you have your configuration management (such as docker containers) in one step of your pipeline and cloudformation templates as another. That way you can test your infrastructure (by deploying cloudformation templates and tearing them down) as well as your code without them being too dependant.
- jbergknoff 8y agoThis infrastructure/configuration distinction is very hazy when it comes to services like Lambda or Fargate, where you just specify your code artifact and there's essentially nothing more to do. It's not clear that it's a net benefit to introduce additional tooling beyond CloudFormation/Terraform for deploying to these services. It's certainly not strictly necessary.
- scarface74 8y agoWhat do you mean by “configuration management” - I don’t use Docker. I use CF for managing configuration with Parameter Store.
- auslander 8y ago> using something like ansible to manage AWS You may use Ansible for top level orchestration, deploying CF stacks and providing CF parameters. Just leave all resource creation to CF itself.
- dkhenry 8y agoI have used ansible to manage large production deployments of AWS infrastructure and what I will say is that it is very good at doing it, but it requires a lot of work compared to using something like terraform or cloud formation. Its not hard to have one playbook to provision all your infrastructure and make sure its up and running it just takes a good amount of lookups and facts calls. The only reason I advocate for doing it is if a team will have a small infrastructure complexity ( like a basic ELB -> ASG -> RDS/EC2/S3 ) and they don't want to bring in more complex tools. Using ansible means you can use one tool to manage both your AMI's for immutable infrastructure and the infrastructure its self ( and can easily script your continuous deployment ). Once you start to really get a complex footprint getting a dedicated tool for infrastructure makes a lot of sense.
- kokey 8y agoThe only thing that spending time with Cloudformation teaches me is how much it makes me prefer doing things with Terraform. I think Cloudformation is considerably better than nothing and it was great when there were no alternatives, but that was a while ago.
- Dunedan 8y agoWhat I don't like about Terraform is that the state of your resources is stored locally, which might lead to consistency problems depending on your setup. With CloudFormation state is handled by CloudFormation itself, so you can be confident that stack updates operate on the latest state.
- artellectual 8y agoYou can store the state remotely there are many options for state storage too. You can use s3 or dynamo db or even create your own web service that accepts web hooks from terraform.
- Dunedan 8y agoOf course you can implement/set up syncing of state on your own, but that's something you don't have to worry about when using CloudFormation. I'm not sure how the options for Terraform handle race conditions when trying to deploy updates in parallel, but again, something you don't have to worry about with CloudFormation.
- olafalo 8y agoIMO the limits of CloudFormation are a bigger pain point than they're made out to be here. The limit of 200 resources per stack is easy to hit, and so is the 450KB template size limit (well, it's possible at least). It's frustrating to need to spread a single service across three stacks because it has a lot of API Gateway endpoints. The real answer is nested stacks, but those still count towards the (raisable) total stack limit of 200.
- Dunedan 8y agoAt least there isn't the limit of 50KB for SAM templates anymore [1]. That was a ridiculous limit, especially as nested templates aren't possible with SAM yet [2]. [1]: https://github.com/awslabs/serverless-application-model/issues/151 https://github.com/awslabs/serverless-application-model/issu... [2]: https://github.com/awslabs/serverless-application-model/issues/90 https://github.com/awslabs/serverless-application-model/issu...
- auslander 8y ago> The limit of 200 resources per stack is easy to hit Never managed to get even close to that number, doing CF for 5 years :)
- olafalo 8y agoI'm guessing it's heavily dependent on what you're using it for, what kind of resources are you describing in CFN? In my case, it's Lambda + API Gateway that's the main culprit. For each endpoint in an API, there's a resource for the lambda itself, and a managed policy, role, log group, subscription filter, API resource, method, lambda permission, model, and additional OPTIONS method for CORS purposes. With a setup like that, you can hit the limit with a moderately-sized API.
- kesor 8y agoor you can utilize exported resources, like the article describes, and append your API endpoints to the "master" API Gateway resource that is not even in the same stack.
- jillesvangurp 8y agoWe've been using CF for a few years as well. IMHO it is very complicated to manage and getting stuff working involves a lot of trial and error. Also you end up waiting a lot. Waiting for things to spin up, waiting for things to become available, waiting for things to rollback, etc. On top of that the failure modes can be ugly and hard to figure out. My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps. So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one. If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.
- stingraycharles 8y ago> My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild One of the more common scenarios where CF gets into a broken state is: 1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update) 2) something else fails, causing rollback 3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data) 4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates. Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
- SanderKnape 8y agoYou can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again. Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- jonthepirate 8y agoTerraform does everything CloudFormation does but in a simpler way where you have more control over what's happening.
- zaphar 8y agoCloudformation has better atomicity guarantees though. It's not perfect but in general if a change to a stack fails it will get rolled back to a known good state. Terraform doesn't give you the same guarantee. You'll have to push a rollback or fix yourself leaving your AWS resources in a potentially broken state while you do.
- Illniyar 8y agoIs there anyone here who used both amazon CF's and azure's ARM and can comment on the benefits and problems of each? When I used CF a few years back (when it started) it was a pain (for those things it actually supported). I'm now using azure and ARM's integration with azure's cloud seems better to me.
- kesor 8y agoLast time I checked ARM did not have any support for custom resources. The first thing mentioned in the article. Also last time I checked it was JSON only, while CloudFormation moved to YAML which is shorter and allows for proper comments.
- ghayes 8y agoI’d also love any experiences people have with Google’s Google Deployment Manager. For me, the product felt like it had many flaws that also had previously tended me away from CloudFormation (specifically, not full support for beta or alpha features and questions about inconsistent states during failures). I decided to go with Terraform since it feels like the industry standard and had full support for even quite new GCP features.
- kesor 8y agoExcellent advice! I would also advise creating a couple of scripts that upload to S3 and run the update-stack commands automagically. Every advice in the article is gold though.
- bevel 8y agoMy biggest inconsistency with Cloud Formation is with smaller AWS offerings. If I need to build a VPC with some EC2 capacity it works well. If I want to create a load balancer and use R53 to do DNS based certificate validation with their in house SSL provider, I'm out of luck. It looks like internal products need to work with cloud formation to enable support, and aws doesn't have a consistent model here. It seems that they are fine with some products cutting corners and not offering support (like DNS based certificate validation) Inconsistency within aws isn't all that surprising.
- gazoakley 8y agoTerraform can do that: https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html https://www.terraform.io/docs/providers/aws/r/acm_certificat... That said, one irritating omission I've had to deal with is not being able to add email subscriptions to SNS topics. The underlying AWS API is a bit odd - I don't think it provides an ARN until the subscription is confirmed.
- renke1 8y agoCloudFormation is pretty cool. In a rather short amount of time I was able to create a reproducible deployment (based on any commit in my Git repo) that deploys a Lambda, makes it accesible via API Gatway, creates a DynamoDB table for storage, sets up Cognito user pool for user management, creates CloudFront distribution that securely serves my SPA and the API Gateway and lastly adds a record to my domain such that is accesible at `${commit}.mydomain.com`.
- technics256 8y agoAwesome. Care to share or point to best resources to learn? Thanks!
- Cpoll 8y agoI recommend using Troposphere instead of vanilla CF. It's a Python library that generates CF templates. It doesn't abstract out anything, so the structure ends up looking very similar to a json or yml template, but with all the conveniences of working with objects in Python. The biggest gripe I have with CF is that it's impossible to introduce existing components into a CloudFormation stack, so any legacy infrastructure has to remain manually managed.
- djstein 8y agoI was unaware of the ability to create custom CF resources! This is great. I will try to make a config to create a AWS Aurora Serverless RDS instance. It went GR Friday, and the team says CF support won’t be available until the end of the month.