4 ms·

I agree that ECS works great for stateless containerized workloads. But you will need other AWS-managed services for state (RDS), caching (ElastiCache), and que

by pmig 2y ago

I agree that ECS works great for stateless containerized workloads. But you will need other AWS-managed services for state (RDS), caching (ElastiCache), and queueing (SQS). So your application is now suddenly spread across multiple services, and you'll need an IaC tool like Terraform, etc. The beauty (and the main reason we use K8s) is that everything is inside our cluster. We use cloudnative-pg, Redis pods, and RabbitMQ if needed, so everything is maintained in a GitOps project, and we have no IaC management overhead. (We do manually provision S3 buckets for backups and object storage, though.)

placardloop 2y ago
Mentioning “no IaC management overhead” is weird. If you’re not using IaC, you’re doing it wrong. However, GitOps is IaC, just by another name, so you actually do have IaC “overhead”.
- Lucasoato 2y ago
  Exactly, not only because Flux/ArgoCD are inherently some sort of IaC themselves, but also because on top of those tools you’ll need to have Terraform to manage the K8s cluster as well as a good practice.
- VectorLock 2y ago
  And most GitOps workflows have considerably more complexity and overhead than your typical Terraform "IaC" workflow.
williamdclt 2y ago
Many companies run k8s for compute and use rds/sqs/redis outside of it. For example RDS is not just hosted PG, it has a whole bunch of features that don’t come out of the box (you do pay for it, I’m not giving an opinion as to whether it’s worth the price)
- murukesh_s 2y ago
  Yea RDS makes your life easy, notifications and easy application of security patches both OS and DB level (minor version upgrades). Easy upgrade of major versions, easy upgrade of storage, RAM and compute (but not so easy to downgrade), easy options for replication, Blue/Green deployments etc to name a few.
- master_crab 2y ago
  Yup. We do that. Anything stateful is not allowed inside the cluster. PVs are annoying enough without having to manage a DB bolted onto what was originally designed for stateless web services.
  - 8n4vidtmkvmk 2y ago
    My db is my cluster. It's been stable for years but I'm afraid to touch it. There's a long outstanding issue in k8s that makes PVs harder to resize than it should be. And they're just more complicated. Trying to move to managed MySQL now. It'll cost me a bunch more but at least I get a fail over node which I don't know how to set up myself... Still no master-master though, apparently that's not an option.
    - pmig 2y ago
      Actually resizing PVC depends on the CSI drive. Some support easy resizing, some require the volume to be detached. You can double check your CSI driver and might just need to patch your storage class. Agreed running databases without operators that can handle replication, master promotion backups and PIT restore is super scary. Most of the modern operators support all of these operations.
whalesalad 2y ago
You make a great point that when everything is on kube it’s easier to manage. But… if you are maintaining storage buckets and stuff elsewhere (to avoid accidental deletion etc, a worthy cause) then you are using terraform regardless. So adding RDS etc to the mix is not as tough as you make it sound. I see both sides of the fence and both have their pros and cons. If you have great operational experience with kube though I’d go all in on that. AWS bends you over with management fees… it’s far more affordable to run a DB, RMQ, etc on your own versus RDS, AMQ
- pmig 2y ago
  AWS Controller for Kubernetes (ACK)[1] provides resources for creating S3 buckets as CR. Also in combination with Pod Identities there is no need for tf. [1]https://aws-controllers-k8s.github.io/community/docs/user-docs/resource-crud/ https://aws-controllers-k8s.github.io/community/docs/user-do...
- esotericimpl 2y ago
  [dead]
huksley 2y ago
How do you run all this on developer's machine?
politelemon 2y ago
In everything you've listed, my conclusion is the opposite. The spread across multiple managed services is not a bad thing, that's actually better considering that using them reduces operational overhead. That is, the spread is irrelevant if the services are managed. The ugliness of k8s is that you're bringing your points of failure together into one, mega point of failure and complexity. Final aside - you absolutely should be using IaC for any serious deployments. If you're using clickops or CLI then the context of the discussion is different and the same critera do not apply.
- pmig 2y ago
  Yes, but our GitOps repository heavily utilizes Kustomize and Flux, allowing us to reuse a significant amount of code across multiple deployment stages and clusters, which has proven to be very effective. We have worked with Terraform modules before, but they quickly became difficult to manage. Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection.
  - placardloop 2y ago
    > Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection. No they aren’t. All of the major IaC solutions (TF, CDK, etc) do ECS deployments directly through their own API, including with drift detection and updates. Good for you for finding something that works, but it sounds like your advice related to IaC solutions is based on a misunderstanding of the benefits of IaC and the tools available.
klysm 2y ago
You’ve replaced IaC overhead with k8s overhead

by pmig 2y ago

placardloop 2y ago

Mentioning “no IaC management overhead” is weird. If you’re not using IaC, you’re doing it wrong. However, GitOps is IaC, just by another name, so you actually do have IaC “overhead”.

Lucasoato 2y ago
Exactly, not only because Flux/ArgoCD are inherently some sort of IaC themselves, but also because on top of those tools you’ll need to have Terraform to manage the K8s cluster as well as a good practice.
VectorLock 2y ago
And most GitOps workflows have considerably more complexity and overhead than your typical Terraform "IaC" workflow.

williamdclt 2y ago

Many companies run k8s for compute and use rds/sqs/redis outside of it. For example RDS is not just hosted PG, it has a whole bunch of features that don’t come out of the box (you do pay for it, I’m not giving an opinion as to whether it’s worth the price)

murukesh_s 2y ago
Yea RDS makes your life easy, notifications and easy application of security patches both OS and DB level (minor version upgrades). Easy upgrade of major versions, easy upgrade of storage, RAM and compute (but not so easy to downgrade), easy options for replication, Blue/Green deployments etc to name a few.
master_crab 2y ago
Yup. We do that. Anything stateful is not allowed inside the cluster. PVs are annoying enough without having to manage a DB bolted onto what was originally designed for stateless web services.
- 8n4vidtmkvmk 2y ago
  My db is my cluster. It's been stable for years but I'm afraid to touch it. There's a long outstanding issue in k8s that makes PVs harder to resize than it should be. And they're just more complicated. Trying to move to managed MySQL now. It'll cost me a bunch more but at least I get a fail over node which I don't know how to set up myself... Still no master-master though, apparently that's not an option.
  - pmig 2y ago
    Actually resizing PVC depends on the CSI drive. Some support easy resizing, some require the volume to be detached. You can double check your CSI driver and might just need to patch your storage class. Agreed running databases without operators that can handle replication, master promotion backups and PIT restore is super scary. Most of the modern operators support all of these operations.

whalesalad 2y ago

You make a great point that when everything is on kube it’s easier to manage. But… if you are maintaining storage buckets and stuff elsewhere (to avoid accidental deletion etc, a worthy cause) then you are using terraform regardless. So adding RDS etc to the mix is not as tough as you make it sound. I see both sides of the fence and both have their pros and cons. If you have great operational experience with kube though I’d go all in on that. AWS bends you over with management fees… it’s far more affordable to run a DB, RMQ, etc on your own versus RDS, AMQ

pmig 2y ago
AWS Controller for Kubernetes (ACK)[1] provides resources for creating S3 buckets as CR. Also in combination with Pod Identities there is no need for tf. [1]https://aws-controllers-k8s.github.io/community/docs/user-docs/resource-crud/ https://aws-controllers-k8s.github.io/community/docs/user-do...
esotericimpl 2y ago
[dead]

huksley 2y ago

How do you run all this on developer's machine?

politelemon 2y ago

In everything you've listed, my conclusion is the opposite. The spread across multiple managed services is not a bad thing, that's actually better considering that using them reduces operational overhead. That is, the spread is irrelevant if the services are managed. The ugliness of k8s is that you're bringing your points of failure together into one, mega point of failure and complexity. Final aside - you absolutely should be using IaC for any serious deployments. If you're using clickops or CLI then the context of the discussion is different and the same critera do not apply.

pmig 2y ago
Yes, but our GitOps repository heavily utilizes Kustomize and Flux, allowing us to reuse a significant amount of code across multiple deployment stages and clusters, which has proven to be very effective. We have worked with Terraform modules before, but they quickly became difficult to manage. Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection.
- placardloop 2y ago
  > Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection. No they aren’t. All of the major IaC solutions (TF, CDK, etc) do ECS deployments directly through their own API, including with drift detection and updates. Good for you for finding something that works, but it sounds like your advice related to IaC solutions is based on a misunderstanding of the benefits of IaC and the tools available.

klysm 2y ago

You’ve replaced IaC overhead with k8s overhead