4 ms·

Unfortunately, my enterprise scale employer forced me onto ECS on EC2 for some of my apps for a very specific reason: Reserved instance pricing. I think there’s

Unfortunately, my enterprise scale employer forced me onto ECS on EC2 for some of my apps for a very specific reason: Reserved instance pricing. I think there’s reserved instance-like pricing for Fargate now. For one particular set of containers (Java application we license from a vendor and then customize with plugins somewhat), the CPU and RAM requirements are fairly large so the savings of ECS on EC2 with the longest Reserved Instance Contract means that I will forever be dealing with the idiocy of ASG based Capacity Providers. For those not in the know, ASG based Capacity Providers are hard to work with because they are very immutable so you end up having to create-then-delete any changes that touch the capacity provider. A capacity provider cannot be deleted if the ASG has any instances. Many tools like terraform’s AWS provider will refuse to delete the ASG till the capacity provider is deleted. The terraform provider just cannot properly reason about the process of discovering and scaling in the ECS tasks on the provider, scaling in the ASG, waiting for 0 instances, and then deleting the Capacity Provider. It’s honestly beyond how providers are supposed to work. TL;DR: The flow chart is somewhat correct: Do everything in your power to run on ECS Fargate. It’s mature enough and has excellent VPC and IAM support these days. Stay as far away from ECS on EC2 as you can. As for EKS, I like it but this company runs on the whole “everyone can do whatever” so each team would have to run its own EKS cluster. If we had a centralized team providing a base k8s cluster with monitoring and what not built in for us to deploy on, I’d be more amenable to it. As it stands, I would have to learn both the development and ops AND security sides of running EKS for a handful of apps. ECS while seeming similar on the surface is much simpler and externalizes concepts like ingresses and load balancing and persistence into the AWS concepts and tooling (CDK, CloudFormation, Terraform) you already know (one hopes).

jpgvm 5y ago
I would go as far to say avoid ECS as much as necessary. Using ECS heavily either means using CloudFormation or Terraform heavily, both of which are shitty tools (TF is probably the best tool in it's class, haven't tried Pulumi yet but doesn't stop it from being shit). Importantly both of which are almost impossible for "normal" developers to use with any level of competency. This leads to 2 inevitable outcomes, a) snowflakes and all the special per-app ops work that entails and b) app teams pushing everything back to infrastructure teams because they either don't want to work with TF or can't be granted sufficient permissions to use it effectively. k8s solves these challenges much more effectively assuming your ops team is capable of setting it up, managing the cluster(s), namespaces, RBAC, etc and any base-level services like external-dns, some ingress provider, cert-manager, etc. Once you do this then app teams are able to deploy directly, they can use helm (eww but it works) to spin up whatever off-the-shelf software they want and are able to easily write manifests in a way that they can't fuck up horribly as easily. Best for both teams, ops and devs. Downside? Requires competent ops team (hard to find) and also some amount of taste in tooling (use things like tanka to make manifests less shit), not to mention the time to actually spin all this up in peace without being pushed to do tons of ad-hoc garbage continually (i.e competent org). So in summary, k8s is generally the right solution for larger orgs because it enforces better split of responsibilities and establishes a powerful (relatively) easy to use API that can support practically everything. Also in the future there is things like ACK (https://aws.amazon.com/blogs/containers/aws-controllers-for-kubernetes-ack/ https://aws.amazon.com/blogs/containers/aws-controllers-for-...) coming which will further reduce the need for app teams to interact with TF or Cloudformation.
- twalla 5y ago
  People focus way too much on the orchestration aspect of Kubernetes when the real draw for larger, more mature organizations is that it provides a good API for doing pretty much anything infrastructure or operations related. You can take the API primitives and build whatever abstractions you want on top of them to match up with how your org deploys and operates software. Once you get into stuff like Open Application Model and start marrying it to operators that can deploy AWS resources and whatever GitOps tool you want to use, you end up with a really nice, consistent interface for developers that removes your infrastructure folks as a bottleneck in the "Hi I want to deploy this new thing everywhere" process.
  - jpgvm 5y ago
    Couldn't agree more. It's the model that is good and most of the benefit is derived organisationally rather than technically.
- krinchan 5y ago
  > So in summary, k8s is generally the right solution for larger orgs because it enforces better split of responsibilities and establishes a powerful (relatively) easy to use API that can support practically everything. And therein lies the problem, my employer won't consolidate the enterprise like that. They're still throwing 100 or so teams around and telling them BIRI (build it, run it) and making everyone write their own infra and application code because "DEVOPS".
  - jpgvm 5y ago
    Yeah that is pretty unfortunate. Makes me think there is probably a market for a k8s w/batteries distribution + ops as a service available via AWS VPC peering/transit GW if such a thing doesn't already exist. i.e pre-setup with good patterns, pluggable/easy OIDC auth, cert-manager and friends. Essentially to cater to the "I need k8s but I don't have the people for it and this makes me sad" crowd.
DandyDev 5y ago
Even without using reserved instance pricing, ECS on EC2 is much cheaper, isn't it? At work we use Hasura, which is written in Haskell and cannot be (easily?) run as a Lambda. Our alternative solution is to run it as a container on ECS. Given that it's a permanently running service, with Fargate we'd pay just to have it sit idle for half of the time, and Fargate is not cheap. Even when running non-reserved EC2 instances to make up our ECS cluster, it is cheaper than using Fargate.

jpgvm 5y ago

I would go as far to say avoid ECS as much as necessary. Using ECS heavily either means using CloudFormation or Terraform heavily, both of which are shitty tools (TF is probably the best tool in it's class, haven't tried Pulumi yet but doesn't stop it from being shit). Importantly both of which are almost impossible for "normal" developers to use with any level of competency. This leads to 2 inevitable outcomes, a) snowflakes and all the special per-app ops work that entails and b) app teams pushing everything back to infrastructure teams because they either don't want to work with TF or can't be granted sufficient permissions to use it effectively. k8s solves these challenges much more effectively assuming your ops team is capable of setting it up, managing the cluster(s), namespaces, RBAC, etc and any base-level services like external-dns, some ingress provider, cert-manager, etc. Once you do this then app teams are able to deploy directly, they can use helm (eww but it works) to spin up whatever off-the-shelf software they want and are able to easily write manifests in a way that they can't fuck up horribly as easily. Best for both teams, ops and devs. Downside? Requires competent ops team (hard to find) and also some amount of taste in tooling (use things like tanka to make manifests less shit), not to mention the time to actually spin all this up in peace without being pushed to do tons of ad-hoc garbage continually (i.e competent org). So in summary, k8s is generally the right solution for larger orgs because it enforces better split of responsibilities and establishes a powerful (relatively) easy to use API that can support practically everything. Also in the future there is things like ACK (https://aws.amazon.com/blogs/containers/aws-controllers-for-kubernetes-ack/ https://aws.amazon.com/blogs/containers/aws-controllers-for-...) coming which will further reduce the need for app teams to interact with TF or Cloudformation.

twalla 5y ago
People focus way too much on the orchestration aspect of Kubernetes when the real draw for larger, more mature organizations is that it provides a good API for doing pretty much anything infrastructure or operations related. You can take the API primitives and build whatever abstractions you want on top of them to match up with how your org deploys and operates software. Once you get into stuff like Open Application Model and start marrying it to operators that can deploy AWS resources and whatever GitOps tool you want to use, you end up with a really nice, consistent interface for developers that removes your infrastructure folks as a bottleneck in the "Hi I want to deploy this new thing everywhere" process.
- jpgvm 5y ago
  Couldn't agree more. It's the model that is good and most of the benefit is derived organisationally rather than technically.
krinchan 5y ago
> So in summary, k8s is generally the right solution for larger orgs because it enforces better split of responsibilities and establishes a powerful (relatively) easy to use API that can support practically everything. And therein lies the problem, my employer won't consolidate the enterprise like that. They're still throwing 100 or so teams around and telling them BIRI (build it, run it) and making everyone write their own infra and application code because "DEVOPS".
- jpgvm 5y ago
  Yeah that is pretty unfortunate. Makes me think there is probably a market for a k8s w/batteries distribution + ops as a service available via AWS VPC peering/transit GW if such a thing doesn't already exist. i.e pre-setup with good patterns, pluggable/easy OIDC auth, cert-manager and friends. Essentially to cater to the "I need k8s but I don't have the people for it and this makes me sad" crowd.

DandyDev 5y ago

Even without using reserved instance pricing, ECS on EC2 is much cheaper, isn't it? At work we use Hasura, which is written in Haskell and cannot be (easily?) run as a Lambda. Our alternative solution is to run it as a container on ECS. Given that it's a permanently running service, with Fargate we'd pay just to have it sit idle for half of the time, and Fargate is not cheap. Even when running non-reserved EC2 instances to make up our ECS cluster, it is cheaper than using Fargate.