6 ms·

I've been a PM for Kubernetes at two of the major contributing companies and helped popularize Operators. The project that will replace Kubernetes will be some

I've been a PM for Kubernetes at two of the major contributing companies and helped popularize Operators. The project that will replace Kubernetes will be something that provides builtins that compose together to EXTEND the system. With such a foundation, you'll be given the flexibility to evolve over time and correct design flaws. Kubernetes was in a rush to beat competitors out in the early orchestration wars, so they focused on primitives that people needed to deploy "today". I think they did a good job, but the focus wasn't on extensibility. In an ideal world (and I've proposed), there be a long term SIG/WG that experimented with custom controllers to add new primitives that focused on EXTENDING Kubernetes. Most companies at the time wanted to tighten up the core APIs, not realizing that they should be skating towards where the puck is going. Now every major Kubernetes provider is past the point of just getting clusters into folks' hands and are realizing they need to add more functionality, but have no means that doesn't come at the cost of explosive complexity.

the_duke 5y ago
Out of curiosity, and particularly because I'm working on a Webassembly based alternative to Kubernetes: What would you do differently if you could start from scratch?
- readams 5y ago
  Very curious where you see the advantage of using Webassembly there. Presumably this is not running in a browser.
  - the_duke 5y ago
    Webassembly is also gaining steam for server workloads due to several advantages (less overhead, better capability based security, composability, ...). See Spin [1], Wasmcloud [2], Lunatic [3], etc. My system is based on a distributed Webassembly runtime. The reason for taking inspiration from Kubernetes is making deployment of distributed workloads on that runtime easy. A nice benefit for a Kubernetes alike system is that the equivalent to operators and controllers can be much more light-weight WASM actors that are easier to develop, deploy and scale. [1] https://github.com/fermyon/spin https://github.com/fermyon/spin [2] https://github.com/wasmCloud/wasmCloud https://github.com/wasmCloud/wasmCloud [3] https://github.com/lunatic-solutions/lunatic https://github.com/lunatic-solutions/lunatic
    - kitd 5y ago
      Even better (and I expect you're aware of this already), you can run Webassembly workloads under Kubernetes control: https://krustlet.dev/ https://krustlet.dev/
      - the_duke 5y ago
        Yeah, krustlet is great for integrating into existing Kube deployments. I'm just tackling the problem from the other side.
    - andrewallbright 5y ago
      Webassembly as a desirable server side technology... I suppose I need to recontextualize what I understand webassembly to be. Hmm.
      - traverseda 5y ago
        It's like the JVM but more languages can run on it and it has buy in from more companies. Imagine all the promises of Java in the 90s/2000s but the open source community is actually interested in it.
        Tostino 5y ago
        Is that actually true though, or is this more of an aspirational goal for WASM? The JVM runs an absolute ton of different languages.
- withinboredom 5y ago
  Please, for the love of interop, do not use GRPC. Not every language supports it (JSON is way better anyway). There are a number of times I’ve wanted to build on something in k8s. But I’m not learning go just to write an api controller. I’m not going to write a GRPC implementation just to write an api controller in my language of choice either. Full disclosure: I really hate GRPC as a wire protocol.
  - treeman79 5y ago
    When I used it for ruby. I was totally fascinated by it. Wow. It was a horrible experience. Lack of proper support made everyone hard. Loved the concepts though.
  - dboreham 5y ago
    Worse than using a binary protocol and an IDL, I think, are the myths that gRPC proponents seem to believe: e.g. you can't delete fields and all fields are optional, ergo <voila> you now have guaranteed backwards compatibility.
mountainriver 5y ago
This is exactly right. I don’t think it’ll happen within k8s because if you’ve ever worked with sig-api you’ll know they aren’t big on innovation. The biggest problem with kube is the declarative interfaces aren’t enough for most problems. You need imperative methods as well and the only way to sort of do this is with aggregated API servers (yuck!) Even the core Pod resource even has a /logs endpoint which is imperative. Also kubectl is a mess, verb->noun makes no sense at all.

the_duke 5y ago

Out of curiosity, and particularly because I'm working on a Webassembly based alternative to Kubernetes: What would you do differently if you could start from scratch?

readams 5y ago
Very curious where you see the advantage of using Webassembly there. Presumably this is not running in a browser.
- the_duke 5y ago
  Webassembly is also gaining steam for server workloads due to several advantages (less overhead, better capability based security, composability, ...). See Spin [1], Wasmcloud [2], Lunatic [3], etc. My system is based on a distributed Webassembly runtime. The reason for taking inspiration from Kubernetes is making deployment of distributed workloads on that runtime easy. A nice benefit for a Kubernetes alike system is that the equivalent to operators and controllers can be much more light-weight WASM actors that are easier to develop, deploy and scale. [1] https://github.com/fermyon/spin https://github.com/fermyon/spin [2] https://github.com/wasmCloud/wasmCloud https://github.com/wasmCloud/wasmCloud [3] https://github.com/lunatic-solutions/lunatic https://github.com/lunatic-solutions/lunatic
  - kitd 5y ago
    Even better (and I expect you're aware of this already), you can run Webassembly workloads under Kubernetes control: https://krustlet.dev/ https://krustlet.dev/
    - the_duke 5y ago
      Yeah, krustlet is great for integrating into existing Kube deployments. I'm just tackling the problem from the other side.
  - andrewallbright 5y ago
    Webassembly as a desirable server side technology... I suppose I need to recontextualize what I understand webassembly to be. Hmm.
    - traverseda 5y ago
      It's like the JVM but more languages can run on it and it has buy in from more companies. Imagine all the promises of Java in the 90s/2000s but the open source community is actually interested in it.
      - Tostino 5y ago
        Is that actually true though, or is this more of an aspirational goal for WASM? The JVM runs an absolute ton of different languages.
withinboredom 5y ago
Please, for the love of interop, do not use GRPC. Not every language supports it (JSON is way better anyway). There are a number of times I’ve wanted to build on something in k8s. But I’m not learning go just to write an api controller. I’m not going to write a GRPC implementation just to write an api controller in my language of choice either. Full disclosure: I really hate GRPC as a wire protocol.
- treeman79 5y ago
  When I used it for ruby. I was totally fascinated by it. Wow. It was a horrible experience. Lack of proper support made everyone hard. Loved the concepts though.
- dboreham 5y ago
  Worse than using a binary protocol and an IDL, I think, are the myths that gRPC proponents seem to believe: e.g. you can't delete fields and all fields are optional, ergo <voila> you now have guaranteed backwards compatibility.

mountainriver 5y ago

This is exactly right. I don’t think it’ll happen within k8s because if you’ve ever worked with sig-api you’ll know they aren’t big on innovation. The biggest problem with kube is the declarative interfaces aren’t enough for most problems. You need imperative methods as well and the only way to sort of do this is with aggregated API servers (yuck!) Even the core Pod resource even has a /logs endpoint which is imperative. Also kubectl is a mess, verb->noun makes no sense at all.