3 ms·

Kubernetes has always had poor design decisions that have rarely ever been improved, because the designers care more about keeping people using their system tha

Kubernetes has always had poor design decisions that have rarely ever been improved, because the designers care more about keeping people using their system than making it simpler. The way forward is not to keep adding abstractions to get around these design problems. The way forward is to fix the design. We need something simpler than K8s. Something that does not need a million abstractions, yet accomplishes the basic goals that we want: efficiency, scalability, security, reliability, reproducibility, immutability, etc. I think the future we should be driving for is a distributed operating system. A real operating system whose primitives run distributed applications. K8s already reproduces features that exist in the kernel, and the kernel already does a lot of the heavy lifting of networking and containerization. What's left is to add userland and kernel features so apps can make a syscall or library call to tell the operating system how they'd like to scale, or deploy, or be traced, etc. This would simplify without adding abstractions. It would also be controlled by the open source world, rather than by corporations, making an incentive to finally get rid of crappy design patterns and make the right design be top priority. Those of you who have used Mosix clusters already have an idea how simple and effective this can be. SSI clusters didn't take off because they were focused on unmodified applications, and sharing memory and threads across systems is hard. But by requiring applications to design themselves for a distributed OS, we can instead allow apps to run independently yet communicate seamlessly across systems as if they were just on one.

dijit 5y ago
You're probably looking for Nomad or Mesos honestly. Kubernetes, as designed, is a framework and a rather leaky one in my not so humble and subjective opinion. it accomplishes a pretty lofty goal (bin-packing, deployment, eventual reconcilation of service and generalising the network/storage so that you don't care about underlying infra) but it does so at a rather heavy cost of complexity and because you can swap out underlying pieces: if you're not on a cloud provider then you're probably going to spend the majority of your life maintaining it. But, you probably want Nomad, if you're thinking about kubernetes seriously.
- Hallucinaut 5y ago
  > if you're not on a cloud provider... and lord help you if you are but choose to try to use the same approach across environments and eskew the markup on something like EKS. I've been thumping my head against trying to have a rather trivial HA service working on both a Hyper-V and EC2 cluster equally for weeks and it seems near impossible for reasons no leading lights in the space quite articulates.
  - dijit 5y ago
    I believe your experience in this regard would be a good blog post in of itself that I would love to read.
- throwaway984393 5y ago
  Nomad is neat, but I think it's the wrong paradigm. Say you wanted to write an application that inspects the current state of jobs running in Nomad and modify them. You need to design something specifically to do that within nomad. Essentially you need to integrate their proprietary interface into your application. In a distributed operating system you would just do ls /proc/host/*/* and then issue syscalls or write to files in /proc/ and /sys/. No proprietary interface, it's just operating system primitives. I think it's so simple that it's hard for people to grasp how insanely easier this would be than any current system. Look up Mosix/OpenMosix (but that's an SSI; I'm not saying we should do SSI exactly, it's just an example of the simplicity).

dijit 5y ago

You're probably looking for Nomad or Mesos honestly. Kubernetes, as designed, is a framework and a rather leaky one in my not so humble and subjective opinion. it accomplishes a pretty lofty goal (bin-packing, deployment, eventual reconcilation of service and generalising the network/storage so that you don't care about underlying infra) but it does so at a rather heavy cost of complexity and because you can swap out underlying pieces: if you're not on a cloud provider then you're probably going to spend the majority of your life maintaining it. But, you probably want Nomad, if you're thinking about kubernetes seriously.

Hallucinaut 5y ago
> if you're not on a cloud provider... and lord help you if you are but choose to try to use the same approach across environments and eskew the markup on something like EKS. I've been thumping my head against trying to have a rather trivial HA service working on both a Hyper-V and EC2 cluster equally for weeks and it seems near impossible for reasons no leading lights in the space quite articulates.
- dijit 5y ago
  I believe your experience in this regard would be a good blog post in of itself that I would love to read.
throwaway984393 5y ago
Nomad is neat, but I think it's the wrong paradigm. Say you wanted to write an application that inspects the current state of jobs running in Nomad and modify them. You need to design something specifically to do that within nomad. Essentially you need to integrate their proprietary interface into your application. In a distributed operating system you would just do ls /proc/host/*/* and then issue syscalls or write to files in /proc/ and /sys/. No proprietary interface, it's just operating system primitives. I think it's so simple that it's hard for people to grasp how insanely easier this would be than any current system. Look up Mosix/OpenMosix (but that's an SSI; I'm not saying we should do SSI exactly, it's just an example of the simplicity).