4 ms·

> EKS My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale poli

> EKS My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale policies. There are very few things to worry about. On the other hand, using k8s has always been a big deal. We built a whole team to manage k8s. We introduce dozens of concepts of k8s or spend person-years on “platform engineering” to hide k8s concepts. We publish guidelines and sdks and all kinds of validators so people can use k8s “properly”. And we still write 10s of thousands lines of YAML plus 10s of thousands of code to implement an operator. Sometimes I wonder if k8s is too intrusive.

xyzzy_plugh 3y ago
I tend to agree that for most things on AWS, EC2 + ASG is superior. It's very polished. EKS is very bare bones. I would probably go so far as to just run Kubernetes on EC2 if I had to go that route. But in general k8s provides incredibly solid abstractions for building portable, rigorously available services. Nothing quite compares. It's felt very stable over the past few years. Sure, EC2 is incredibly stable, but I don't always do business on Amazon.
- Noumenon72 3y ago
  At first I thought your "in general" statement was contradicting your preference for EC2 + ASG. I guess AWS is such a large part of my world that "in general" includes AWS instead of meaning everything but AWS.
cedws 3y ago
K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.
- xyzzy_plugh 3y ago
  This isn't my experience at all. Maybe three or four years ago? Who exactly needs millions of lines of code?
  - Spivak 3y ago
    I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc. And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
    - mschuster91 3y ago
      We have all of that neatly extracted into a Terraform module. Write it once and now EKS clusters are essentially disposable.
      - Solvency 3y ago
        You just added yet another Thing in that huge pile of things representing millions of lines of code. That's the point.
        dvfjsdhgfv 3y ago
        Everything we run our workloads on is based on millions of LoCs, whether it's in the OS, in K8S, in is built-in or external kinds. If you decide to run K8S in AWS, you'll be better of using Karpenter, external-secrets and all these things as they will make your life easier in various ways.
        lijok 3y ago
        Why is that inherently a problem? How many LOCs in the linux kernel again?
- 3y ago
foofie 3y ago
> My contrarian view is that EC2 + ASG is so pleasant to use. Sometimes I think that managed kubernetes services like EKS are the epitome of "give the customers what they want", even when it makes absolutely no sense at all. Kubernetes is about stitching together COTS hardware to turn it into a cluster where you can deploy applications. If you do not need to stitch together COTS hardware, you have already far better tools available to get your app running. You don't need to know or care in which node your app is suppose to run and not run, what's your ingress control, if you need to evict nodes, etc. You have container images, you want to run containers out of them, you want them to scale a certain way, etc.
mr_moose 3y ago
To me, it sounds like your company went through a complex re-architecturing exercise at the same time you moved to Kubernetes, and your problems have more to do with your (probably flawed) migration strategy than the tool. Lifting and shifting an "EC2 + ASG" set-up to Kubernetes is a straightforward process unless your app is doing something very non-standard. It maps to a Deployment in most cases. The fact that you even implemented an operator (a very advanced use-case in Kubernetes) strongly suggests to me that you're doing way more than just lifting and shifting your existing set-up. Is it a surprise then that you're seeing so much more complexity?
- krab 3y ago
  Not familiar with the OP but this may have been the pitch for migration: "K8S will allow us better automation".

xyzzy_plugh 3y ago

I tend to agree that for most things on AWS, EC2 + ASG is superior. It's very polished. EKS is very bare bones. I would probably go so far as to just run Kubernetes on EC2 if I had to go that route. But in general k8s provides incredibly solid abstractions for building portable, rigorously available services. Nothing quite compares. It's felt very stable over the past few years. Sure, EC2 is incredibly stable, but I don't always do business on Amazon.

Noumenon72 3y ago
At first I thought your "in general" statement was contradicting your preference for EC2 + ASG. I guess AWS is such a large part of my world that "in general" includes AWS instead of meaning everything but AWS.

cedws 3y ago

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.

xyzzy_plugh 3y ago
This isn't my experience at all. Maybe three or four years ago? Who exactly needs millions of lines of code?
- Spivak 3y ago
  I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc. And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
  - mschuster91 3y ago
    We have all of that neatly extracted into a Terraform module. Write it once and now EKS clusters are essentially disposable.
    - Solvency 3y ago
      You just added yet another Thing in that huge pile of things representing millions of lines of code. That's the point.
      - dvfjsdhgfv 3y ago
        Everything we run our workloads on is based on millions of LoCs, whether it's in the OS, in K8S, in is built-in or external kinds. If you decide to run K8S in AWS, you'll be better of using Karpenter, external-secrets and all these things as they will make your life easier in various ways.
      - lijok 3y ago
        Why is that inherently a problem? How many LOCs in the linux kernel again?
3y ago

foofie 3y ago

> My contrarian view is that EC2 + ASG is so pleasant to use. Sometimes I think that managed kubernetes services like EKS are the epitome of "give the customers what they want", even when it makes absolutely no sense at all. Kubernetes is about stitching together COTS hardware to turn it into a cluster where you can deploy applications. If you do not need to stitch together COTS hardware, you have already far better tools available to get your app running. You don't need to know or care in which node your app is suppose to run and not run, what's your ingress control, if you need to evict nodes, etc. You have container images, you want to run containers out of them, you want them to scale a certain way, etc.

mr_moose 3y ago

To me, it sounds like your company went through a complex re-architecturing exercise at the same time you moved to Kubernetes, and your problems have more to do with your (probably flawed) migration strategy than the tool. Lifting and shifting an "EC2 + ASG" set-up to Kubernetes is a straightforward process unless your app is doing something very non-standard. It maps to a Deployment in most cases. The fact that you even implemented an operator (a very advanced use-case in Kubernetes) strongly suggests to me that you're doing way more than just lifting and shifting your existing set-up. Is it a surprise then that you're seeing so much more complexity?

krab 3y ago
Not familiar with the OP but this may have been the pitch for migration: "K8S will allow us better automation".