5 ms·
We've been using CF for a few years as well. IMHO it is very complicated to manage and getting stuff working involves a lot of trial and error. Also you end up
by jillesvangurp 8y ago
We've been using CF for a few years as well. IMHO it is very complicated to manage and getting stuff working involves a lot of trial and error.
Also you end up waiting a lot. Waiting for things to spin up, waiting for things to become available, waiting for things to rollback, etc. On top of that the failure modes can be ugly and hard to figure out.
My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.
So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.
If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.
- stingraycharles 8y ago> My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild One of the more common scenarios where CF gets into a broken state is: 1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update) 2) something else fails, causing rollback 3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data) 4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates. Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
- SanderKnape 8y agoYou can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again. Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- cle 8y agoYou better do so before deploying again, because the roll forward will break since the resource already exists. This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.
- SanderKnape 8y agoTrue, but it beats the alternative where CloudFormation deletes objects that you didn't want deleted. The underlying issue is that the S3 objects are outside of the CloudFormation scope, thus it takes no risk and doesn't delete your objects. A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable. Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.
- cle 8y agoMy preferred behavior would be for CFN to not barf when rolling forward. In other words, to be able to assume control over a resource that already exists.
- deleted 8y ago[deleted]
- scarface74 8y agoSpecifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template. There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway. As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.
- auslander 8y ago> DeletionPolicy=Retain with named resources Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb"). Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned. Speaking of S3, its better not to include bucket resource in CF until you know what you're doing. > stuff like RDS databases which you absolutely never ever want to have destroyed That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.
- tangentspace 8y agoI agree with splitting up CF stacks this way. It reduces the the blast radius, that saved me on a number of occasions when stack updates went sideways. The problem introduced by that approach is how to manage a large number of CF stacks. First I used a homegrown Python library to manage them, then switched to having Terraform manage CF. At first Terraform on CF was just intended to be an expedient measure to facilitate migrating everything to Terraform, and eventually we did migrate to pure Terraform. But then we started hitting all the rough edges in Terraform. In hindsight, the hybrid approach had actually been more stable and manageable than using either tool in isolation.
- deboflo 8y agoYou should take a look at AWS Code Pipeline, which natively support creating/updating cloudformation stacks.
- Rapzid 8y agoSame waiting with Terraform. If you aren't automating windows installs and/or imaging, count yourself lucky :) Terraform also lacks a lot of the extra "smarts" that CF has; like rolling updates of any kind and some other higher level automation across different services. They take very much a blue/green approach which is beyond limiting for some services. Ran into a few broken states using Cloud Formation via Serverless project. Luckily had been in the habbit of keeping "statefull" AWS resources such as queues, databases, and other stuff in their own stacks(or Terraform) to keep their stacks complication to a minimum and mitigate the impact of the more complicated stacks needing to be deleted.. Ultimately I started going with and promote a hybrid approach where Terraform makes sense; Terraform for base infrastructure(including most stateful resources) and CF for stuff like autoscale groups and etc.
- plumeria 8y agoUpdating GSIs for a DynamoDB table has been difficult for us. We ended up recreating the table in order to change the indexes.