4 ms·
Something funny (well, kind of sad) about CloudFormation I noticed this summer was that if you deploy a CloudFormation stack which updates a ECS service and dep
by easton 5y ago
Something funny (well, kind of sad) about CloudFormation I noticed this summer was that if you deploy a CloudFormation stack which updates a ECS service and deploys tasks which then fail health checks, CloudFormation will do nothing about this and just let ECS keep killing and restarting tasks for.. well, at least several hours. You have to know to go into ECS and drain the tasks manually and then initiate a rollback from CF to get your service back into a good state. The bug reports about this I found were going back years.
The upside is that I got really well acquainted with how ECS worked.
- tkahnoski 5y ago100x this. Prior company committed to doing Infrastructure as Code and CloudFormation worked well except for this hiccup. We didn't even have that many services on ECS but we probably had 1 ticket a week asking support to help us with a 'stuck' stack. Our commitment to CloudFormation was doubled down on that we could do containers, Lambda, and 95% of any other AWS Services.... However, in hidsight using SAM and the ECS CLI probably would have resulted in a more predictable CI/CD process as we weren't fighting deploy semantics through CloudFormation abstraction.
- viraptor 5y agoWhat we ended up doing is mostly decoupling CF and tasks. CF manages the infrastructure. Software deployments / tasks are done by another system which understands issues like this one.
- fictionfuture 5y agoI had this same bug!! Cost us like $1000 before we fixed it;
- misiti3780 5y agoWhy did it cost so much money ?
- deleted 5y ago[deleted]
- knodi 5y agoThis support for CloudFormation across AWS services is crazy bad. For the longest time you couldn't have an ALB rules with it. To fill this gap you have to create dreaded custom resources... which can get suck for 4 hours.
- hahkerman 5y agoCFn supports stack rollback alarms. All you need to do is create an alarm for failing ECS tasks.
- easton 5y agoIIRC, the problem was that rollbacks couldn’t start until it agreed that the service deployment had failed, which wouldn’t happen until I manually drained the task (or waited an unknown period for CF to timeout). You could click the rollback button but nothing would happen. I don’t think the alarm would’ve helped, but maybe I’m misunderstanding.
- hahkerman 5y agoI'm referring specifically to this feature: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-rollback-triggers.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... CFn will allow you to specify up to 5 alarms to monitor during stack deployment as well as a duration to monitor them for up to 3 hours. During deployment, the stack does not transition into the COMPLETE state until the monitor duration has completed without the alarms being triggered. If one of the alarms is triggered, the stack immediately rolls back. Since failed ECS tasks create CW Events, you can set up alarms on them and use them as the rollback monitor trigger.
- theteapot 5y agoYour updating the service to a broken image? I would say ensuring it's healthy falls outside of the scope of CF responsibilities. `aws ecs update-service` won't even do that. ECS should wait till your new image is healthy before draining previous image versions. If you deployed a bad image, probably needs someone to look at it anyway.
- eranation 5y agoYes! It drove me nuts! I had a task that I forgot to give a / 200 ok and it just thrashed like crazy. I think once I have the cluster auto rollback it solved the issue though (other than ensuring a reliable health check). They should fix this!
- jamesfinlayson 5y agoAgreed - very painful. I think there was a recent improvement to how CloudFormation handles failed ECS deployments but it's not on by default.