4 ms·

Is the mountain of k8s code on top of your cloud of choice not strictly more complex than the cloud services it's orchestrating? Like I think k8s is good softwa

by Spivak 2y ago

Is the mountain of k8s code on top of your cloud of choice not strictly more complex than the cloud services it's orchestrating? Like I think k8s is good software but to hear that you're not choosing it because it's good engineering is surprising. To me that's the only reason you choose it, because you want or need the control-loop based infrastructure management that you can't get on say AWS alone.

marcinzm 2y ago
There’s always a mountain of code on top of the cloud provider.
- andrewstuart2 2y ago
  Furthermore, there's always a mountain of code aggregating heterogeneous systems into a uniform interface. Look at the linux kernel, or LLVM, or pretty much any other (successful) project that provides abstractions to many different backends.
- Spivak 2y ago
  Yes but that code isn't typically running in production the way k8s is. Having a lot of terraform is just a fancy method of statically provisioning infrastructure. The code is gone once it applies. Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. Your problems now include k8s networking as well as VPC networking.
  - marcinzm 2y ago
    > The code is gone once it applies Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful. > Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition. I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.
zipmapfoldright 2y ago
That's the same with.. the Linux kernel? I'd wager that most services people write have less complexity than the the kernel it runs on. We choose to do that not because we need its full feature set, but because it gives us useful abstractions.
- cjk 2y ago
  I don’t know that that’s the right analogy. End users don’t have to write a mountain of configuration in order to effectively use the Linux kernel. If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
  - cortesoft 2y ago
    You can deploy quite a bit to k8s with just a few lines of helm installs, and it will work with very little configuration.
  - Spivak 2y ago
    Exactly, I think I could have stated it better. Using k8s is strictly more complex from the perspective of the operator, and the mountain I'm referring to isn't actually from k8s itself but from all the 3rd party stuff you have to layer on to actually make it useful-- things like cluster-autoscaler, vpa, kube-prometheus, aws-load-balancer-controller, misc. operators, that fall on my side of the wall when it comes to fixing it when it's broken. I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
    - skydhash 2y ago
      If it can’t be monolithic like *BSD, at least offer distributions like the Linux Sphere. Like a ready to go LAMP stack. Or a collection of services (Database, Storage, Logs,…)
      - 2y ago

by Spivak 2y ago

marcinzm 2y ago

There’s always a mountain of code on top of the cloud provider.

andrewstuart2 2y ago
Furthermore, there's always a mountain of code aggregating heterogeneous systems into a uniform interface. Look at the linux kernel, or LLVM, or pretty much any other (successful) project that provides abstractions to many different backends.
Spivak 2y ago
Yes but that code isn't typically running in production the way k8s is. Having a lot of terraform is just a fancy method of statically provisioning infrastructure. The code is gone once it applies. Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. Your problems now include k8s networking as well as VPC networking.
- marcinzm 2y ago
  > The code is gone once it applies Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful. > Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition. I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.

zipmapfoldright 2y ago

That's the same with.. the Linux kernel? I'd wager that most services people write have less complexity than the the kernel it runs on. We choose to do that not because we need its full feature set, but because it gives us useful abstractions.

cjk 2y ago
I don’t know that that’s the right analogy. End users don’t have to write a mountain of configuration in order to effectively use the Linux kernel. If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
- cortesoft 2y ago
  You can deploy quite a bit to k8s with just a few lines of helm installs, and it will work with very little configuration.
- Spivak 2y ago
  Exactly, I think I could have stated it better. Using k8s is strictly more complex from the perspective of the operator, and the mountain I'm referring to isn't actually from k8s itself but from all the 3rd party stuff you have to layer on to actually make it useful-- things like cluster-autoscaler, vpa, kube-prometheus, aws-load-balancer-controller, misc. operators, that fall on my side of the wall when it comes to fixing it when it's broken. I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
  - skydhash 2y ago
    If it can’t be monolithic like *BSD, at least offer distributions like the Linux Sphere. Like a ready to go LAMP stack. Or a collection of services (Database, Storage, Logs,…)
    - 2y ago