5 ms·
The ECS default deployment model is quite frankly a disaster. I've written many Python/Go scripts over the years to wrangle it into a sensible form for CI/CD.
by ciguy 6y ago
The ECS default deployment model is quite frankly a disaster. I've written many Python/Go scripts over the years to wrangle it into a sensible form for CI/CD.
Good to see that they're working on it, but I don't know why they don't fix the underlying paradigm instead of making it Cloudformation exclusive.
- justicezyx 6y agoCould you elaborate a bit on the problem of "ECS default deployment model"?
- ciguy 6y agoIt's just very rigid and not easy to extend. As soon as you hit a certain scale or need to do something slightly non-standard it's a mess.
- vazamb 6y agoI would love to know what the problem is. We do dozen of deployments every week with a ALB + ECS + Fargate setup. We upload a new container image, create a new task and launch as many tasks as desired (so if we want 2 containers running we launch 2, for a total of 4). ALB calls the /health endpoints on the new containers and if they pass the healthchecks it drains connections to the old containers and stops the tasks. This has worked seamlessly for a long time now without any downtime during deployments. EDIT: I should mentioned that we are using AWS CDK for all of this. All it does is register a new task as the default task for a service and ECS/ALB does the rest.
- Nk26 6y agoI would like to know the same, we have moved almost everything to fargate and ecs and have had zero issues.
- x3n0ph3n3 6y agoSame here -- I perform the exact same kind of deployment you mentioned using CloudFormation. My only grief is poor rollback detection / control.
- ciguy 6y agoThe default model works for most use cases, but it's not really flexible for non-standard cases and it fails completely at larger scale. If you have a few dozen containers taking 100,000 requests per second and you want to slow warm them with a segment of traffic for example, it's much easier to do on Kube than on ECS. ECS is also possible but it's just not as transparent or easy to work with.
- WatchDog 6y agoI find ECS particularly, ECS on EC2, to be really painful for small deployments. I just want a cluster that scales in and out to the amount of memory my tasks need, I'm not very concerned about CPU. Until capacity providers it was more or less impossible to do so without having a bunch of excess capacity provisioned. Even with capacity providers, I can't seem to get a cluster to scale in to 0 instances when no tasks are needed. I have one ECS service that requires an EBS volume mount, which means that when the task definition is updated, I need the service to stop, so that the new task can mount the same volume. This deployment model is essentially impossible without implementing a custom deployment strategy. Fargate makes things a bit easier, and now that you can use EFS volumes with it it might make more sense. But overall, everything in ECS just seems poorly designed, clunky and hacky, like many AWS products.
- friend-monoid 6y agoYou can’t do EFS on Fargate in CloudFormation yet though.
- scarface74 6y agoTrue. But you can do it with a custom resource. https://gist.github.com/guillaumesmo/4782e26500a3ac768888daab3c55b139 https://gist.github.com/guillaumesmo/4782e26500a3ac768888daa...
- WatchDog 6y agoYou also cant use capacity providers with cloudformation yet. Likely due to the fact, that you also cant delete capacity providers, nor change a cluster's default capacity provider.
- scarface74 6y agoThat’s not a Blue Green deployment.....
- ganstyles 6y agoI don't think they are saying it was, unless I'm misreading. They're talking about standard ECS deploys. Yo add my anecdata, I do lots of ECS deploys via terraform into production and it works pretty seamlessly.
- scarface74 6y agoI don’t have a problem with deploys with CF. But it only let’s you configure a minimum healthy percentage. Which is good enough if you only need to validate that an instance is in an acceptable state via a health check.
- sandGorgon 6y agoExactly this - I think internally there is a tussle on what is the strategic way forward. There are so many - Elastic Beanstalk, ECS, EKS, ECS-on-Fargate, EKS-on-Fargate..and of course the huge marketing push for Serverless. They could have the sensible way out and built EKS as the foundation of everything - makes total sense given the massive ecosystem around kubernetes. ECS and Fargate should be killed off. https://cdk8s.io/ https://cdk8s.io/ replaces Elastic Beanstalk ....but still basically runs on top of EKS. The pairing of CDK8S and EKS are fundamentally enough for all usecases that AWS basically sells.
- solidasparagus 6y agok8s is not for everyone. It has a high administration and complexity burden. It would be a mistake to make k8s a requirement for core AWS services. Fargate is a low-level compute platform - it is parallel to ec2 and lambda. It does not compete with kubernetes (as can be seen by the eks-on-fargate offering). There is probably some tension between ECS and k8s - AWS built a container orchestration platform based on what they think the world (and Amazon) needs and then k8s became madly popular. And it's not clear that AWS was wrong because k8s is essentially too complex for many use cases. It makes total sense that they would support both fully.
- harpratap 6y agoVanilla K8s is not meant to be directly used by everyone. Watch https://www.youtube.com/watch?v=ZqQTEdHVaCw https://www.youtube.com/watch?v=ZqQTEdHVaCw for insight on how K8s team expect the project to move forward. Frameworks like OpenDeis and Knative are meant to be used by developers. IMO K8s will become akin to Linux Kernel. Almost no one uses bare mainline Linux kernels, you choose a distro based on your needs. Companies like RedHat and Canonical will pop up and provide their own packaged "distros" of K8s and you will choose which philosophy best suits your needs.
- Androider 6y agoCDK8S doesn't replace Elastic Beanstalk, because the folks using EB are going to continue to use EB. If it ain't broke... ECS predates EKS, and is better integrated than EKS with their other services. If you want a painless container experience on AWS, ECS for Fargate is what'll you'll want to use today. So both existing folks who use ECS today as well as new customers are going to keep using ECS. Folks who like K8, perhaps from prior experience, are going to use EKS despite it having some rough corners. See, Amazon is perfectly happy to support everything under the sun as long as there are paying users. See their database offering for very much the same strategy, and it would be silly to say that they should drop DB x since they now offer DB y.
- bdcravens 6y agoWe haven't had any problems, but our workload is mostly async background work. Our CI pushes master when it passes, running a simple script to push to ECR with a git SHA tag, update the task to use the latest image, and then update the service to the latest task definition. It takes 2 lines of bash to update the task definition, and 2 lines to run the AWS commands.
- snypox 6y agoReally? At the last company we had like a 200 lines PowerShell script for all of that.
- scarface74 6y agoYou can do all of it by pushing the container to ECR, tagging it with your build number and running CloudFormation. You pass the build number in as a parameter and you specify your image as image: !subst “image:${Tag}”
- ciguy 6y agoYeah for the standard vanilla case it's ok. As soon as you hit a certain scale or need to do anything non-standard it becomes much harder to work with.
- tbrock 6y agoReally? We’ve been using it for 3 years and it works fine. Make sure you pay attention to min/max task settings. There are basically three ways to do it: Min less than max, max is 100. Max greater than min, min is 100. Some combination of the two. For the first you take down running tasks first and then backfill with new (green) tasks. For the second you add green tasks and once stable take down blue. Make sure that if you only have two tasks that min/max move in 50% increments: I.e you can’t scale up/down to 125%/75% but can to 150%/50%.