5 ms·

>When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features. Could you elabora

>When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features. Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it.

paulddraper 8y ago
If you're comparing to Kubernetees, "everything". But I'm using ECS and Fargate right now (orchestrated with Terraform). https://rivethealth.com https://rivethealth.com Probably the biggest thing I would like is native secrets management. "Security groups" are limited as it depends on having an AWS interface per container. ANd I don't know how much longer our time on Fargate will last. It's expensive, but more significantly there are limitations in not having access to the host. Having to install an SSH server in every docker container to be able to debug (simple things, like top) is annoying.
- ratling 8y ago
  We nixed ECS for the same reason, no secrets through ConfigSet.
ekidd 8y ago
[Update: See below, many of these features have been added to ECS since the last time I touched this portion of our infrastructure.] > Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it. Some typical examples: - Kubernetes allows you to run a monitoring container on every single node using something called a "DaemonSet". On ECS, you'll have to build all your monitoring tools into your base image, or use cloud-init to spawn an ECS task on each machine. - You're probably going to end up writing a bunch of scripts to generate ECS task definition JSON and to update running services, and you'll need to integrate this into your CI system somehow. With Kubernetes, you can get away with "kubectl apply -f" for a fairly long time. - Kubernetes makes it relatively easy to allocate and manage persistent disk volumes. I wouldn't necessarily use them for a production database, but they're great for smaller things. - Kubernetes has autoscaling support, plus the ability to control which containers run on which types of servers. - Kubernetes has basic secret management built-in. It's nothing as nice as Vault, but it's good enough to get started. - Kubernetes has support for a whole bunch of useful minor things that would typically wind up as Terraform scripts on AWS. And so on. None of these is very hard individually, but there's a ton of things like this. So we're slowly migrating pieces of ECS infrastructure over to Kubernetes so that we can stop reinventing so many wheels. Again, for those things which can run on Heroku, either choice is overkill. And I have to admit that ECS is very reliable at the things it does. If you do decide to look at Kubernetes, I highly recommend skimming the O'Reilly books, which provide a solid overview of how it all fits together.
- lentil 8y ago
  ECS already has all the things you mentioned here. Some, like daemon set deployments, are built in. Others, like secrets management, are provided as integrations with other AWS services. But they are all there.
- Thaxll 8y ago
  All the things you mentioned are available on ECS: - https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ecs-adds-daemon-scheduling/ https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ec... - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... - ECS runs on ASG / EC2 so there is auto scalling - https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/ https://aws.amazon.com/blogs/compute/managing-secrets-for-am... I think you don't understand all the glue between AWS services, off course ECS doesn't have everything that's why there is EKS, but all the thing above exists on ECS.
  - ekidd 8y ago
    Nice! I'd been busy with another project and missed a couple of those announcements. The DaemonSet equivalent will allow me to rip up some particularly annoying cruft the next time I need to touch it. Thank you. I've looked at various guides for setting up cluster autoscaling on ECS, and so far, everything I've found looks far more complicated than Kubernetes cluster autoscaling on Google. Is there a nice guide?
    - deboflo 8y ago
      Look no further: https://github.com/awslabs/ecs-refarch-continuous-deployment https://github.com/awslabs/ecs-refarch-continuous-deployment https://github.com/aws-samples/ecs-refarch-cloudformation/blob/master/README.md https://github.com/aws-samples/ecs-refarch-cloudformation/bl...
  - joseph 8y ago
    The secrets example isn't very comparable. With Kubernetes, the secrets are injected into the container as environment variables or files, whereas the ECS example requires assigning an IAM role to the task and doing a Parameter Store lookup from within the container. This usually requires a custom docker image with an entrypoint to handle that part. The same is true for ConfigMaps, which I believe ECS lacks.
privateSFacct 8y ago
FWIW did the fargate approach very happily. I know k8s getting the love but found fargate easy to get going on and set and forget once deployed

paulddraper 8y ago

If you're comparing to Kubernetees, "everything". But I'm using ECS and Fargate right now (orchestrated with Terraform). https://rivethealth.com https://rivethealth.com Probably the biggest thing I would like is native secrets management. "Security groups" are limited as it depends on having an AWS interface per container. ANd I don't know how much longer our time on Fargate will last. It's expensive, but more significantly there are limitations in not having access to the host. Having to install an SSH server in every docker container to be able to debug (simple things, like top) is annoying.

ratling 8y ago
We nixed ECS for the same reason, no secrets through ConfigSet.

ekidd 8y ago

[Update: See below, many of these features have been added to ECS since the last time I touched this portion of our infrastructure.] > Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it. Some typical examples: - Kubernetes allows you to run a monitoring container on every single node using something called a "DaemonSet". On ECS, you'll have to build all your monitoring tools into your base image, or use cloud-init to spawn an ECS task on each machine. - You're probably going to end up writing a bunch of scripts to generate ECS task definition JSON and to update running services, and you'll need to integrate this into your CI system somehow. With Kubernetes, you can get away with "kubectl apply -f" for a fairly long time. - Kubernetes makes it relatively easy to allocate and manage persistent disk volumes. I wouldn't necessarily use them for a production database, but they're great for smaller things. - Kubernetes has autoscaling support, plus the ability to control which containers run on which types of servers. - Kubernetes has basic secret management built-in. It's nothing as nice as Vault, but it's good enough to get started. - Kubernetes has support for a whole bunch of useful minor things that would typically wind up as Terraform scripts on AWS. And so on. None of these is very hard individually, but there's a ton of things like this. So we're slowly migrating pieces of ECS infrastructure over to Kubernetes so that we can stop reinventing so many wheels. Again, for those things which can run on Heroku, either choice is overkill. And I have to admit that ECS is very reliable at the things it does. If you do decide to look at Kubernetes, I highly recommend skimming the O'Reilly books, which provide a solid overview of how it all fits together.

lentil 8y ago
ECS already has all the things you mentioned here. Some, like daemon set deployments, are built in. Others, like secrets management, are provided as integrations with other AWS services. But they are all there.
Thaxll 8y ago
All the things you mentioned are available on ECS: - https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ecs-adds-daemon-scheduling/ https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ec... - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... - ECS runs on ASG / EC2 so there is auto scalling - https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/ https://aws.amazon.com/blogs/compute/managing-secrets-for-am... I think you don't understand all the glue between AWS services, off course ECS doesn't have everything that's why there is EKS, but all the thing above exists on ECS.
- ekidd 8y ago
  Nice! I'd been busy with another project and missed a couple of those announcements. The DaemonSet equivalent will allow me to rip up some particularly annoying cruft the next time I need to touch it. Thank you. I've looked at various guides for setting up cluster autoscaling on ECS, and so far, everything I've found looks far more complicated than Kubernetes cluster autoscaling on Google. Is there a nice guide?
  - deboflo 8y ago
    Look no further: https://github.com/awslabs/ecs-refarch-continuous-deployment https://github.com/awslabs/ecs-refarch-continuous-deployment https://github.com/aws-samples/ecs-refarch-cloudformation/blob/master/README.md https://github.com/aws-samples/ecs-refarch-cloudformation/bl...
- joseph 8y ago
  The secrets example isn't very comparable. With Kubernetes, the secrets are injected into the container as environment variables or files, whereas the ECS example requires assigning an IAM role to the task and doing a Parameter Store lookup from within the container. This usually requires a custom docker image with an entrypoint to handle that part. The same is true for ConfigMaps, which I believe ECS lacks.

privateSFacct 8y ago

FWIW did the fargate approach very happily. I know k8s getting the love but found fargate easy to get going on and set and forget once deployed