4 ms·

This blog post doesn't account for an extremely important detail. Right off the bat it mentions "Fargate is now faster than EC2" but this is not the case for p

by nickjj 4y ago

This blog post doesn't account for an extremely important detail. Right off the bat it mentions "Fargate is now faster than EC2" but this is not the case for practical usage on pre-sized clusters, at least not on EKS. As of today you can't avoid a ~30 second hard delay on starting a container on EKS Fargate. It's the time it takes for Amazon to understand you want to run a workload and provide the Fargate driven compute resources into your cluster so it's available to run your workload. Then on top of that, you will wait however long it takes to download your image from a registry because images aren't cached anywhere. In practice this usually means 40-60 seconds to just start a container on EKS Fargate, and if you happen to perform jobs before your main app runs such as a database migration it means you're paying this penalty twice (even if your migration is a no-op since it spawns a pod to run the command), so it's really a ~90-120 second delay to start a container every single time you deploy your application, which you need to wait for each replica too. Compare that to having an EC2 based cluster where you have the resources provisioned ahead of time. Starting your containers will happen as fast as your process will start, there's 0 seconds waiting for the compute resources to be available and since it's just an EC2 node images are saved on disk so you only need to download the diff (often very small for app code changes). In practice the difference in night and day, I'm so happy we moved away from Fargate to managed EC2 nodes on EKS. Things are SO much faster, everything from being able to develop and test things on EKS to every single release of all of our services.

jifucboc 4y ago
> having an EC2 based cluster where you have the resources provisioned ahead of time. That means keeping unused or partially used EC2s up which means $$$.
- nickjj 4y ago
  > That means keeping unused or partially used EC2s up which means $$$. It does but having a pre-sized cluster that's overprovisioned can end up being cheaper than using Fargate because Fargate is really expensive. This isn't accounting for using spot instances on EC2 too. Of course your mileage may vary but as long as you're not dealing with massively irregular spikes where you're spinning up hundreds or thousands of containers up / down based on load you can very easily just run a few [dozen] replicas of your apps on a pre-sized cluster with enough room to let Kubernetes horizontally scale your pods as needed. For scaling the cluster's size there's Karpenter which is mentioned in the post. It can quite quickly give you new resources in a Fargate-like manner without needing to use Fargate. The benefit of Karpenter is it's faster than auto-scaling groups and you can spin up uniquely sized instances based on your workload.
  - ollyculverhouse 4y ago
    Fargate isn't really expensive? It does have a premium but I wouldn't say 'really expensive'. https://www.trek10.com/blog/fargate-pricing-vs-ec2 https://www.trek10.com/blog/fargate-pricing-vs-ec2
    - nickjj 4y ago
      > It does have a premium but I wouldn't say 'really expensive' EKS Fargate costs $29.55 per 1 vCPU and $3.25 per 1GB of memory per month. This is listed in their official docs: https://aws.amazon.com/fargate/pricing/ https://aws.amazon.com/fargate/pricing/ Let's say you plan to run 5 web apps, 3 replicas of each and they all need 4gb of memory. Fargate demands you reserve 2 vCPUs for 4gb even if your app only needs 20% of 1 CPU, so you're wasting 1.8 CPUs per replica of each app. Often times web apps use a lot more memory than CPU, even with Ruby or Python your web app might be humming along at like 5% CPU load while using potentially gigs of memory. That leaves you paying: 5 * 3 * 4gb of memory = 60gb of memory = $195.00 5 * 3 * 2 cpus = 30 cpu cores = $886.50 = $1,081.50 You're wasting a ton of CPU here because of AWS' rules on EKS Fargate memory / cpu size combos. To run your workload on EC2 you can grab yourself (4) m6i.xlarge instances which has 16 CPU cores and 64gb of memory of total capacity which is $560.64 (grand total for 4). In this case Fargate is double the cost, to me that is a lot. You do have less CPU capacity but it doesn't matter, your apps aren't coming close to that in CPU usage. If your work load supported it (stateless web apps that can finish a request within 2 minutes) you could use EC2 spot instances too which would be $122.35 total instead of $560.64. Both regular instance and spot instance prices were taken from AWS' official docs, I just multiplied their hourly values by 730 which is how many hours AWS uses to calculate 1 month.

by nickjj 4y ago

jifucboc 4y ago

> having an EC2 based cluster where you have the resources provisioned ahead of time. That means keeping unused or partially used EC2s up which means $$$.

nickjj 4y ago
> That means keeping unused or partially used EC2s up which means $$$. It does but having a pre-sized cluster that's overprovisioned can end up being cheaper than using Fargate because Fargate is really expensive. This isn't accounting for using spot instances on EC2 too. Of course your mileage may vary but as long as you're not dealing with massively irregular spikes where you're spinning up hundreds or thousands of containers up / down based on load you can very easily just run a few [dozen] replicas of your apps on a pre-sized cluster with enough room to let Kubernetes horizontally scale your pods as needed. For scaling the cluster's size there's Karpenter which is mentioned in the post. It can quite quickly give you new resources in a Fargate-like manner without needing to use Fargate. The benefit of Karpenter is it's faster than auto-scaling groups and you can spin up uniquely sized instances based on your workload.
- ollyculverhouse 4y ago
  Fargate isn't really expensive? It does have a premium but I wouldn't say 'really expensive'. https://www.trek10.com/blog/fargate-pricing-vs-ec2 https://www.trek10.com/blog/fargate-pricing-vs-ec2
  - nickjj 4y ago
    > It does have a premium but I wouldn't say 'really expensive' EKS Fargate costs $29.55 per 1 vCPU and $3.25 per 1GB of memory per month. This is listed in their official docs: https://aws.amazon.com/fargate/pricing/ https://aws.amazon.com/fargate/pricing/ Let's say you plan to run 5 web apps, 3 replicas of each and they all need 4gb of memory. Fargate demands you reserve 2 vCPUs for 4gb even if your app only needs 20% of 1 CPU, so you're wasting 1.8 CPUs per replica of each app. Often times web apps use a lot more memory than CPU, even with Ruby or Python your web app might be humming along at like 5% CPU load while using potentially gigs of memory. That leaves you paying: 5 * 3 * 4gb of memory = 60gb of memory = $195.00 5 * 3 * 2 cpus = 30 cpu cores = $886.50 = $1,081.50 You're wasting a ton of CPU here because of AWS' rules on EKS Fargate memory / cpu size combos. To run your workload on EC2 you can grab yourself (4) m6i.xlarge instances which has 16 CPU cores and 64gb of memory of total capacity which is $560.64 (grand total for 4). In this case Fargate is double the cost, to me that is a lot. You do have less CPU capacity but it doesn't matter, your apps aren't coming close to that in CPU usage. If your work load supported it (stateless web apps that can finish a request within 2 minutes) you could use EC2 spot instances too which would be $122.35 total instead of $560.64. Both regular instance and spot instance prices were taken from AWS' official docs, I just multiplied their hourly values by 730 which is how many hours AWS uses to calculate 1 month.