14 ms·
(Disclaimer: I work for AWS, but any opinions expressed here are solely my own and not necessarily the company's.) I don't know about that. ECS works fine at
by otterley 7y ago
(Disclaimer: I work for AWS, but any opinions expressed here are solely my own and not necessarily the company's.)
I don't know about that. ECS works fine at large scale, but it's not going to replace 4000 tasks immediately. And doing so could potentially be a huge shock to your customers if you tried. (You have to take the time to gracefully drain existing connections, etc.) Nor did the author desire to implement a blue-green type deployment which would use AWS's elasticity to its fullest potential out of unspecified cost concerns.
It's not clear to me that Kubernetes would fare significantly better here. It's not without its own particular performance bottlenecks; and the issues about safely draining connections and blue/green deployments would remain the same there. The issues aren't really related to the particular deployment orchestrator, as much as the fact that 4000 containers is pretty large number by any measure for a single unit of deployment.
- maerF0x0 7y agoActually OP gave the exact reason ECS does not work for them: > The rate at which we can start tasks restricts the parallelism of our deploy. Despite us setting the MaximumPercent parameter to 200%, the ECS start-task API call has a hard limit of 10 tasks per call, and it is rate-limited. We need to call it 400 times to place all our containers in production. That call limit needs to scale with the cluster size
- otterley 7y agoThat sounds different from the usual way customers perform an ECS deployment, which is to replace the task definition of the ECS Service with a new one and let the control plane manage it. Only a single API call is needed, and the control plane can launch replacement tasks pretty quickly itself.
- ahmcb 7y agoWe don't actually call the startTask API ourselves, but when we tell an ECS service (using cloudformation) to use a new task definition (which is basically just a new ECR image tag), ECS calls startTask and various other APIs on our behalf.
- otterley 7y agoThat is true. FWIW, as an experiment I built a 50-node ECS cluster and launched 4000 tasks on it. I couldn't build a single service with 4000 tasks due to soft limits, so I built 4 services with 1000 tasks each (nginx, 256MB, 128 CPU units). No load balancer. Overall task replacement time was about 15 minutes -- which is pretty respectable, in my view. Obviously not as fast as doing a hot code swap, though. Tradeoffs abound.
- AYBABTME 7y agoWhy exactly is it an expensive task? Where's the bottleneck? I worked on this problem before, so curious how it's handled (or not) at AWS.
- pojzon 7y agoCan I have a question about the bottlenecks you have mentioned in case of Kubernetes ? Im really curious about this one. What were the bottlenecks you have faced till now.