5 ms·

I actually know k8 as well as another Borg-alike (Tupperware) that actually runs at Borg-like scale. GP is absolutely right that it is insane overkill with no

I actually know k8 as well as another Borg-alike (Tupperware) that actually runs at Borg-like scale. GP is absolutely right that it is insane overkill with no conceivable way to pay back the complexity investment in all but a few niche cases. Working at a FAANG isn’t the only niche where full-Kube makes sense, there are others, but it’s a niche thing, with the niche being roughly: I rack and run physical gear on a limited number of SKUs at “cattle not pets” scale. If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust than any half-assed Borg-alike setup most people would come up with. They’re happy to take your money if you yo dawg like imaginary computers so I put a docker in your Xen VM so you can Kubernetes while you pay me big bucks, but that’s because they’re not running charities.

FridgeSeal 4y ago
Ok, let’s agree that K8s is overkill. What commodity tool should I be replacing it with that gives me: * super for heterogenous workloads (web servers, big batch loads, streaming workloads) * autoscaling the workloads * autoscaling the cluster * auto-restarting workloads * basically automatic networking * the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use * the ability to hire people with experience in it already * ability to run stateful workloads, and have disks managed and moved for you * the ability to do all that by just providing a config * the ability to do all that without having to let people have actual access to the machines/OS * per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code * have access to a wide range of tooling and docs * be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment. * ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications. * ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications * ability to do all this without your devs needing to be networking experts. * not locked to a specific cloud provider I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out. > If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.
- x86x87 4y ago
  How did people ever manage to do anything before k8s? I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases. Learning how to package services and have them run anywhere? A lost art. Cloud agnosticism is a red herring. No matter what you do today you are not cloud agnostic.
  - FridgeSeal 4y ago
    > How did people ever manage to do anything before k8s? Nobody ever said it wasn’t possible. Nobody is saying “you can’t run things the way you want”, the argument is “you keep saying K8s is bad, here’s some reasons why it’s maybe not”. > I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases. If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert. > Learning how to package services and have them run anywhere? A lost art. The issue isn’t my teams code-if we were only running our own stuff, and we only used a single language, packaging would likely be straightforward. But it’s the fact that we run several other applications, of varying degrees of “packaged” and the fact that I can barely get the Python devs to use a virtualenv properly that makes this statement a bit too unreasonable in my experience. Containers may be messy, but at least I can run 3 different spark pipelines and 4 Python applications without needing to dig into the specifics of “how this particular app packages it’s dependencies” because that’s always an awful experience.
    - x86x87 4y ago
      > If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert. Let's define what you're talking about here and I can show you the way. What is the difficulty in autoscalling Spark jobs? What jupyter env do they use? You say grpc api. That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work" Where are you running your workloads? Cloud? Which one?
xyzzy_plugh 4y ago
I don't agree. The complexity isn't wildly different for complicated scenarios. Scenarios falling between the simplest and complicated absolutely benefit from running on a managed Kubernetes. Rolling your own Kubernetes is useful only at scales where you can't wrangle the complexity of self-service operations for developers, and even then it's still very questionable unless your headcount is astounding. I would have agreed with you three years ago, but today there's a ton of surprisingly good off the shelf solutions, you can get very, very far in a brief amount of time with a single engineer. A ton of the functionality exposed by Kubernetes is absolutely necessary at any scale and very difficult to do yourself. "Don't let any process on this host talk to this port except these ones" is something that is so much easier in K8s than anything else I've ever encountered.
- lovehashbrowns 4y ago
  I fully agree with you that the complexity isn’t all that different from the typical scenarios. One of the scenarios I’ve seen Kubernetes replace is something like tying together Packer, Chef, Jenkins, Terraform, And Vault to get an EC2 instance propped up, but that’s just to deploy the code. What was particularly annoying to me was that the developers were already using containers to build their code and another container for unit testing. So switching to Kubernetes would’ve been far easier. I had to go in and deconstruct the work they were doing with containers to then get their app deployed to an EC2 instance. Although at this level you’re still not saving a ton by going the Kubernetes route. You get a ton of benefits with it, though. Even just tiny things like being able to get the config from a deployment using kubectl is so good. And then you can modify it, too! That’s been super helpful for troubleshooting.
- benreesman 4y ago
  I feel that you’ve kind of talked past my point, though maybe I missed yours. My point was that unless you’re running on metal, the cloud people built all this shit, and much, much better than k8s (or Tupperware for that matter). They’ll stick you Docker container inside a VM happily because people pay them to do it, but it’s redundant. Google didn’t OSS Tensorflow out of the goodness of their hearts, they did it to sell TPU hours on GCP. k8s is likewise not charitable giving: it’s to sell managed Kubernetes on GCP, and subtle nudges to get people to make it their careers are uh, just business I guess. Provisioning as code is in a sad state: Chef/Ansible/Salt/friends can’t deliver on idempotency, Nix is too complicated for most. So people use a container primitive (Docker) to work around that. This has nothing, absolutely nothing, to do with service discover and failover and auto-grow and other Kube shit. AWS does all that already. Better. This is a borderline-cynical business strategy on the part of the cloud providers, and judging from the religious fervor around k8s (someone upthread has it in their username for Christ’s sake), it’s working.
x86x87 4y ago
Well put. Unless you're on bare metal with a cetain scale k8s just adds an abstraction layer on top of whatever primitives the cloud provides.
SuperQue 4y ago
You seem to have missed the point of what makes Kubernetes a really nice system to work with. It's not the machines. It's the comosible components, the service building blocks I can get with an API call. * Want traffic into a cluster, ask for an Ingress. * Want to auto-scale? Ask for a HorizontalPodAutoscaler. * Want to manage CPU and memory rather than number of containers? VerticalPodAutoscaler. * Want to point your monitoring system at a port? PodMonitor or ServiceMonitor And of course, you can extend on the basic plugins. Want a bunch of databases? Maybe install the Vitess operator and simply make an API call to create an instance of VitessCluster. There are a dozen options, depending on what flavor of database you want. Oh, and you don't need a sysadmin/SRE to do all this for you. You can be granted rights to a Namespace and compose everything you need in your namespace.

FridgeSeal 4y ago

Ok, let’s agree that K8s is overkill. What commodity tool should I be replacing it with that gives me: * super for heterogenous workloads (web servers, big batch loads, streaming workloads) * autoscaling the workloads * autoscaling the cluster * auto-restarting workloads * basically automatic networking * the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use * the ability to hire people with experience in it already * ability to run stateful workloads, and have disks managed and moved for you * the ability to do all that by just providing a config * the ability to do all that without having to let people have actual access to the machines/OS * per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code * have access to a wide range of tooling and docs * be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment. * ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications. * ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications * ability to do all this without your devs needing to be networking experts. * not locked to a specific cloud provider I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out. > If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.

x86x87 4y ago
How did people ever manage to do anything before k8s? I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases. Learning how to package services and have them run anywhere? A lost art. Cloud agnosticism is a red herring. No matter what you do today you are not cloud agnostic.
- FridgeSeal 4y ago
  > How did people ever manage to do anything before k8s? Nobody ever said it wasn’t possible. Nobody is saying “you can’t run things the way you want”, the argument is “you keep saying K8s is bad, here’s some reasons why it’s maybe not”. > I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases. If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert. > Learning how to package services and have them run anywhere? A lost art. The issue isn’t my teams code-if we were only running our own stuff, and we only used a single language, packaging would likely be straightforward. But it’s the fact that we run several other applications, of varying degrees of “packaged” and the fact that I can barely get the Python devs to use a virtualenv properly that makes this statement a bit too unreasonable in my experience. Containers may be messy, but at least I can run 3 different spark pipelines and 4 Python applications without needing to dig into the specifics of “how this particular app packages it’s dependencies” because that’s always an awful experience.
  - x86x87 4y ago
    > If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert. Let's define what you're talking about here and I can show you the way. What is the difficulty in autoscalling Spark jobs? What jupyter env do they use? You say grpc api. That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work" Where are you running your workloads? Cloud? Which one?

xyzzy_plugh 4y ago

I don't agree. The complexity isn't wildly different for complicated scenarios. Scenarios falling between the simplest and complicated absolutely benefit from running on a managed Kubernetes. Rolling your own Kubernetes is useful only at scales where you can't wrangle the complexity of self-service operations for developers, and even then it's still very questionable unless your headcount is astounding. I would have agreed with you three years ago, but today there's a ton of surprisingly good off the shelf solutions, you can get very, very far in a brief amount of time with a single engineer. A ton of the functionality exposed by Kubernetes is absolutely necessary at any scale and very difficult to do yourself. "Don't let any process on this host talk to this port except these ones" is something that is so much easier in K8s than anything else I've ever encountered.

lovehashbrowns 4y ago
I fully agree with you that the complexity isn’t all that different from the typical scenarios. One of the scenarios I’ve seen Kubernetes replace is something like tying together Packer, Chef, Jenkins, Terraform, And Vault to get an EC2 instance propped up, but that’s just to deploy the code. What was particularly annoying to me was that the developers were already using containers to build their code and another container for unit testing. So switching to Kubernetes would’ve been far easier. I had to go in and deconstruct the work they were doing with containers to then get their app deployed to an EC2 instance. Although at this level you’re still not saving a ton by going the Kubernetes route. You get a ton of benefits with it, though. Even just tiny things like being able to get the config from a deployment using kubectl is so good. And then you can modify it, too! That’s been super helpful for troubleshooting.
benreesman 4y ago
I feel that you’ve kind of talked past my point, though maybe I missed yours. My point was that unless you’re running on metal, the cloud people built all this shit, and much, much better than k8s (or Tupperware for that matter). They’ll stick you Docker container inside a VM happily because people pay them to do it, but it’s redundant. Google didn’t OSS Tensorflow out of the goodness of their hearts, they did it to sell TPU hours on GCP. k8s is likewise not charitable giving: it’s to sell managed Kubernetes on GCP, and subtle nudges to get people to make it their careers are uh, just business I guess. Provisioning as code is in a sad state: Chef/Ansible/Salt/friends can’t deliver on idempotency, Nix is too complicated for most. So people use a container primitive (Docker) to work around that. This has nothing, absolutely nothing, to do with service discover and failover and auto-grow and other Kube shit. AWS does all that already. Better. This is a borderline-cynical business strategy on the part of the cloud providers, and judging from the religious fervor around k8s (someone upthread has it in their username for Christ’s sake), it’s working.

x86x87 4y ago

Well put. Unless you're on bare metal with a cetain scale k8s just adds an abstraction layer on top of whatever primitives the cloud provides.

SuperQue 4y ago

You seem to have missed the point of what makes Kubernetes a really nice system to work with. It's not the machines. It's the comosible components, the service building blocks I can get with an API call. * Want traffic into a cluster, ask for an Ingress. * Want to auto-scale? Ask for a HorizontalPodAutoscaler. * Want to manage CPU and memory rather than number of containers? VerticalPodAutoscaler. * Want to point your monitoring system at a port? PodMonitor or ServiceMonitor And of course, you can extend on the basic plugins. Want a bunch of databases? Maybe install the Vitess operator and simply make an API call to create an instance of VitessCluster. There are a dozen options, depending on what flavor of database you want. Oh, and you don't need a sysadmin/SRE to do all this for you. You can be granted rights to a Namespace and compose everything you need in your namespace.