3 ms·

> I'm over here laughing in my "over an hour of CloudFormation only to have a run fail, and then a rollback fail, and have to contact AWS support" life. Your e

> I'm over here laughing in my "over an hour of CloudFormation only to have a run fail, and then a rollback fail, and have to contact AWS support" life. Your experience must be over four years out of date, because CloudFormation has supported the ability to continue updating a rollback since February 2016 [1]. [1] https://aws.amazon.com/blogs/devops/continue-rolling-back-an-update-for-aws-cloudformation-stacks-in-the-update_rollback_failed-state/ https://aws.amazon.com/blogs/devops/continue-rolling-back-an...

bird_monster 6y ago
My experience is from 2019, friend.
- wgjordan 6y ago
  Needing to contact support for this issue in 2019 only means that the user didn't know how to use the 'continue update rollback' feature properly, which was a feature added back in 2016 specifically to support the rollback-failed scenarios.
  - _5yoy 6y ago
    It's weird the frequency with which the AWS Support Reps agree the issues we find aren't those that would've been resolvable on our own, as each time we file a ticket we follow up with our reps for appropriate training and support. I understand that you like this tool, but you do not need to fight everyone suggesting it doesn't work as well as you want it to.
    - throwaway894345 6y ago
      I’ve also run into these from time to time, at an old gig where we didn’t have a support contract, and thus we just had borked stacks here and there and prayed that we would never similarly bork prod. On the other hand, we worked hard to get our CloudFormation templates to a state that we could rebuild prod from scratch fairly easily if it came down to it.
pqb 6y ago
Yes, it is true. However, I am hesitant to say the AWS CloudFormation is a bulletproof tool that does not need help from the support side. If somebody was too impatient and manually removed a resource that was managed by CFN, then he might end with a deadlock, where only help was actually a support team member. Been there, don't repeat that error - please do not create (not auto-detectable) drift changes on a purpose (it could be hard to modify/remove the stack in future). Edit: My memories are from Q1 2020.
- wgjordan 6y ago
  > If somebody was too impatient and manually removed a resource that was managed by CFN, then he might end with a deadlock, where only help was actually a support team member. This specific situation can be resolved by doing a 'continue update rollback' and skipping the already-deleted resources- see the troubleshooting guide [1] for more detailed info. [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-update-rollback-failed https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
acdha 6y ago
Their experience is accurate. If you use CloudFormation for anything non-trivial you’ll be rudely corrected from thinking that this works and then spend time migrating to Terraform so you never go through that again, plus you’ll get support for most new AWS features faster. I spent 2019 & early 2020 helping people make that switch after getting burned by the CF “impossible” situations happening.

bird_monster 6y ago

My experience is from 2019, friend.

wgjordan 6y ago
Needing to contact support for this issue in 2019 only means that the user didn't know how to use the 'continue update rollback' feature properly, which was a feature added back in 2016 specifically to support the rollback-failed scenarios.
- _5yoy 6y ago
  It's weird the frequency with which the AWS Support Reps agree the issues we find aren't those that would've been resolvable on our own, as each time we file a ticket we follow up with our reps for appropriate training and support. I understand that you like this tool, but you do not need to fight everyone suggesting it doesn't work as well as you want it to.
  - throwaway894345 6y ago
    I’ve also run into these from time to time, at an old gig where we didn’t have a support contract, and thus we just had borked stacks here and there and prayed that we would never similarly bork prod. On the other hand, we worked hard to get our CloudFormation templates to a state that we could rebuild prod from scratch fairly easily if it came down to it.

pqb 6y ago

Yes, it is true. However, I am hesitant to say the AWS CloudFormation is a bulletproof tool that does not need help from the support side. If somebody was too impatient and manually removed a resource that was managed by CFN, then he might end with a deadlock, where only help was actually a support team member. Been there, don't repeat that error - please do not create (not auto-detectable) drift changes on a purpose (it could be hard to modify/remove the stack in future). Edit: My memories are from Q1 2020.

wgjordan 6y ago
> If somebody was too impatient and manually removed a resource that was managed by CFN, then he might end with a deadlock, where only help was actually a support team member. This specific situation can be resolved by doing a 'continue update rollback' and skipping the already-deleted resources- see the troubleshooting guide [1] for more detailed info. [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-update-rollback-failed https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

acdha 6y ago

Their experience is accurate. If you use CloudFormation for anything non-trivial you’ll be rudely corrected from thinking that this works and then spend time migrating to Terraform so you never go through that again, plus you’ll get support for most new AWS features faster. I spent 2019 & early 2020 helping people make that switch after getting burned by the CF “impossible” situations happening.