6 ms·

Change Sets for AWS CloudFormation

45 points by wallflower 11y ago

fleitz 11y ago
Call me when they get rid of UPDATE_ROLLBACK_FAILED until then it's practically useless. Whoever thought it was ever acceptable to leave your production stack in state that requires deleting it is an idiot. Even if they fixed it I doubt I'd ever trust a service that ever acted like that with my production infrastructure.
- andrewguenther 11y ago
  It isn't gone, but you can at least recover from it now. https://blogs.aws.amazon.com/application-management/post/Tx11YT4MHFDZMK6/Continue-Rolling-Back-an-Update-for-AWS-CloudFormation-stacks-in-the-UPDATE-ROLL https://blogs.aws.amazon.com/application-management/post/Tx1...
  - justinsb 11y ago
    Is that just a retry though? It sounds like the onus is still on you to fix the problem manually so that CloudFormation can complete its rollback, rather than e.g. being able to request a different desired CloudFormation state that might be reachable.
- deleted 11y ago
  [deleted]
meddlepal 11y ago
CloudFormation is awful. I used it extensively for about a year and will never go back.
- pmoriarty 11y ago
  Could you please elaborate on your experience? Very interested. Also, would you mind talking about what you use instead?
  - meddlepal 11y ago
    I use Terraform right now but I'm looking at some other tech as well currently because Terraform is a bit too low-level for my current employers needs (but a fantastic product overall). CF problems: * JSON format is awful (practically speaking this can be solved with Troposphere...) * So slow. * UPDATE_ROLLBACK_FAILED ... It apparently was fixed very very recently, but what a garbage design decision. * Debugging CF errors is a nightmare. * Nested templates have to be stashed in S3.
    - lox 11y ago
      Cfoo (https://github.com/drrb/cfoo https://github.com/drrb/cfoo) is great, it's simply a yaml-to-json converter with some syntactic sugar for cloudformation. Troposphere seemed like too heavy an approach.
      - eropple 11y ago
        YMMV, but I'd much rather Cfer (https://github.com/seanedwards/cfer https://github.com/seanedwards/cfer), which I found useful enough to contribute back to. A real DSL instead of relying on comparatively inexpressive YAML is very appealing to me.
    - xorgar831 11y ago
      Likewise, CF is the leading cause of outages in our infrastructure. We've completely abandon it in favor of Terraform. * CF gets deadlocked if resources are changed outside of it, and there's no way to prevent people from doing that either * No concept of a rolling update to a fleet or any warning it's going to cause an outage with out having to dig through fine print in the docs on every single change, just terminates all the EC2 instances in an ASG/ELB in some cases * If you have a template, remember any change you make to that template has to be applied to every single stack you have, otherwise you'll either damage something or deadlock the stack when updating it later, it can't figure out what's deployed on it's own and maintains its own state that can be different from the real state. We had to stop using templates at all and just store a copy of each template in our local git repo. * If you have more than 100 stacks searching the list of stacks requires clicking a handful of times on the "show more stacks" button before it'll show the one you typed in * Stupid slow, esp. compared to Terraform * Doesn't delete Route53 DNS records even after saying it does in the log outout * Can break ELB configurations, such as sticky sessions and cause outages * Doesn't actually support deploying all AWS resources, SNS/SQS perms/rules aren't complete for example.
- SteveNuts 11y ago
  Were you using cloudformation only or did you use something like troposphere
  - meddlepal 11y ago
    CloudFormation only.
- buremba 11y ago
  +1 1. We can't test the templates in local environment, the only way to test a template is to create a real cluster which usually takes ~15 minutes. 2. The error messages are not helpful enough. It usually fails with "SETUP_FAILED" error with no proper error cause. 3. The rollback may not work if you change the configuration. You need to delete the resources manually or even need to get help from Amazon since the resources stuck at some failed status. 4. The JSON template is hard to write and read. We have a fat single file template that has 2k lines and scared to change it since we might broke something. 5. Not all AWS actions are supported. For example we need to add custom resources (ELB and RDS instances) to a Opsworks stack but it's not possible with Cloudformation.
  - zwily 11y ago
    I'm not a CloudFormation superfan, but... 1. Yup. 2. In my experience, the error message is usually fine, but there are many times where it's completely obtuse. 3. They just added a new feature to help with this. When you get ROLLBACK_FAILED, you can now tweak things and retry the rollback. It has helped me a few times not have to contact support recently. 4. 100% agree. I write all my templates in yaml, which helps a lot. I really hope they just add native yaml support. Example[1] 5. You can now create custom resources using Lambda from within your templates, which is pretty cool. Example[1] edit: I should add that at my company, we mostly use Terraform now. I still like to use CF for projects that will be open-sourced though, as I don't want step 1 of my instructions to be "install terraform". [1] https://github.com/zwily/sqs-to-lambda-via-lambda https://github.com/zwily/sqs-to-lambda-via-lambda
  - leg100 11y ago
    Can't counter these points other than that what else is there to automate AWS infrastructure? Terraform suffers all these problems, too.
    - buremba 11y ago
      I don't have experience with Terraform but it should be possible to find a way to fix these issues except the first one.
  - 11y ago
- the_arun 11y ago
  What do you use instead?
kenbreeman 11y ago
This is a step in the right direction but the lack of consistent error handling and having some resources created that can't be referenced are still huge pain points (e.g. the default security group and route table). I've had the most success using CloudFormation to create an entirely new stack, test it, cut over, then delete the old one... definitely not optimal.
- mryan 11y ago
  References to existing resources, such as the default SG, can be handled with stack parameters. You can pass in its ID as a parameter to the stack, and refer to this parameter in your launch configs or ingress rules. An example from one of my client's stacks: "Parameters": { "DefaultSG": { "Type": "AWS::EC2::SecurityGroup::Id", "Default": "sg-abc123" }, Personally I prefer to create a new SG to replace the default one as it means all of my infrastructure is part of a CF stack, but the parameter method can be used to partially manage (some) non-CF resources.
leg100 11y ago
This is effectively the 'dry-run' feature that has been sought for a long long time [1], and led me to switch to Terraform (which does do this amongst other key differences) . Why did it take so damn long to implement? I do appreciate it has come about eventually. Terraform has its foibles (and stupid half-declarative language) but it would take further improvements to consider going back to CF. [1] https://forums.aws.amazon.com/thread.jspa?messageID=563929&#563929 https://forums.aws.amazon.com/thread.jspa?messageID=563929&#...
- zwily 11y ago
  Actually, this isn't quite dry-run. What this does is calculate the diff between two templates, and just provide the operations from that diff. It does not inspect the current state of resources to see if there were any out-of-band changes that would be unwound. So it gets you closer to a dry-run, but is not as complete as terraform's.

45 points by wallflower 11y ago

Call me when they get rid of UPDATE_ROLLBACK_FAILED until then it's practically useless. Whoever thought it was ever acceptable to leave your production stack in state that requires deleting it is an idiot. Even if they fixed it I doubt I'd ever trust a service that ever acted like that with my production infrastructure.

andrewguenther 11y ago
It isn't gone, but you can at least recover from it now. https://blogs.aws.amazon.com/application-management/post/Tx11YT4MHFDZMK6/Continue-Rolling-Back-an-Update-for-AWS-CloudFormation-stacks-in-the-UPDATE-ROLL https://blogs.aws.amazon.com/application-management/post/Tx1...
- justinsb 11y ago
  Is that just a retry though? It sounds like the onus is still on you to fix the problem manually so that CloudFormation can complete its rollback, rather than e.g. being able to request a different desired CloudFormation state that might be reachable.
deleted 11y ago
[deleted]

meddlepal 11y ago

CloudFormation is awful. I used it extensively for about a year and will never go back.

pmoriarty 11y ago
Could you please elaborate on your experience? Very interested. Also, would you mind talking about what you use instead?
- meddlepal 11y ago
  I use Terraform right now but I'm looking at some other tech as well currently because Terraform is a bit too low-level for my current employers needs (but a fantastic product overall). CF problems: * JSON format is awful (practically speaking this can be solved with Troposphere...) * So slow. * UPDATE_ROLLBACK_FAILED ... It apparently was fixed very very recently, but what a garbage design decision. * Debugging CF errors is a nightmare. * Nested templates have to be stashed in S3.
  - lox 11y ago
    Cfoo (https://github.com/drrb/cfoo https://github.com/drrb/cfoo) is great, it's simply a yaml-to-json converter with some syntactic sugar for cloudformation. Troposphere seemed like too heavy an approach.
    - eropple 11y ago
      YMMV, but I'd much rather Cfer (https://github.com/seanedwards/cfer https://github.com/seanedwards/cfer), which I found useful enough to contribute back to. A real DSL instead of relying on comparatively inexpressive YAML is very appealing to me.
  - xorgar831 11y ago
    Likewise, CF is the leading cause of outages in our infrastructure. We've completely abandon it in favor of Terraform. * CF gets deadlocked if resources are changed outside of it, and there's no way to prevent people from doing that either * No concept of a rolling update to a fleet or any warning it's going to cause an outage with out having to dig through fine print in the docs on every single change, just terminates all the EC2 instances in an ASG/ELB in some cases * If you have a template, remember any change you make to that template has to be applied to every single stack you have, otherwise you'll either damage something or deadlock the stack when updating it later, it can't figure out what's deployed on it's own and maintains its own state that can be different from the real state. We had to stop using templates at all and just store a copy of each template in our local git repo. * If you have more than 100 stacks searching the list of stacks requires clicking a handful of times on the "show more stacks" button before it'll show the one you typed in * Stupid slow, esp. compared to Terraform * Doesn't delete Route53 DNS records even after saying it does in the log outout * Can break ELB configurations, such as sticky sessions and cause outages * Doesn't actually support deploying all AWS resources, SNS/SQS perms/rules aren't complete for example.
SteveNuts 11y ago
Were you using cloudformation only or did you use something like troposphere
- meddlepal 11y ago
  CloudFormation only.
buremba 11y ago
+1 1. We can't test the templates in local environment, the only way to test a template is to create a real cluster which usually takes ~15 minutes. 2. The error messages are not helpful enough. It usually fails with "SETUP_FAILED" error with no proper error cause. 3. The rollback may not work if you change the configuration. You need to delete the resources manually or even need to get help from Amazon since the resources stuck at some failed status. 4. The JSON template is hard to write and read. We have a fat single file template that has 2k lines and scared to change it since we might broke something. 5. Not all AWS actions are supported. For example we need to add custom resources (ELB and RDS instances) to a Opsworks stack but it's not possible with Cloudformation.
- zwily 11y ago
  I'm not a CloudFormation superfan, but... 1. Yup. 2. In my experience, the error message is usually fine, but there are many times where it's completely obtuse. 3. They just added a new feature to help with this. When you get ROLLBACK_FAILED, you can now tweak things and retry the rollback. It has helped me a few times not have to contact support recently. 4. 100% agree. I write all my templates in yaml, which helps a lot. I really hope they just add native yaml support. Example[1] 5. You can now create custom resources using Lambda from within your templates, which is pretty cool. Example[1] edit: I should add that at my company, we mostly use Terraform now. I still like to use CF for projects that will be open-sourced though, as I don't want step 1 of my instructions to be "install terraform". [1] https://github.com/zwily/sqs-to-lambda-via-lambda https://github.com/zwily/sqs-to-lambda-via-lambda
- leg100 11y ago
  Can't counter these points other than that what else is there to automate AWS infrastructure? Terraform suffers all these problems, too.
  - buremba 11y ago
    I don't have experience with Terraform but it should be possible to find a way to fix these issues except the first one.
- 11y ago
the_arun 11y ago
What do you use instead?

kenbreeman 11y ago

This is a step in the right direction but the lack of consistent error handling and having some resources created that can't be referenced are still huge pain points (e.g. the default security group and route table). I've had the most success using CloudFormation to create an entirely new stack, test it, cut over, then delete the old one... definitely not optimal.

mryan 11y ago
References to existing resources, such as the default SG, can be handled with stack parameters. You can pass in its ID as a parameter to the stack, and refer to this parameter in your launch configs or ingress rules. An example from one of my client's stacks: "Parameters": { "DefaultSG": { "Type": "AWS::EC2::SecurityGroup::Id", "Default": "sg-abc123" }, Personally I prefer to create a new SG to replace the default one as it means all of my infrastructure is part of a CF stack, but the parameter method can be used to partially manage (some) non-CF resources.

leg100 11y ago

This is effectively the 'dry-run' feature that has been sought for a long long time [1], and led me to switch to Terraform (which does do this amongst other key differences) . Why did it take so damn long to implement? I do appreciate it has come about eventually. Terraform has its foibles (and stupid half-declarative language) but it would take further improvements to consider going back to CF. [1] https://forums.aws.amazon.com/thread.jspa?messageID=563929&#563929 https://forums.aws.amazon.com/thread.jspa?messageID=563929&#...

zwily 11y ago
Actually, this isn't quite dry-run. What this does is calculate the diff between two templates, and just provide the operations from that diff. It does not inspect the current state of resources to see if there were any out-of-band changes that would be unwound. So it gets you closer to a dry-run, but is not as complete as terraform's.