3 ms·

I _still_ think Kubernetes is insanely complex, despite all that it does. It seems less complex these days because it's so pervasive, but complex it remains. I

by nunez 1y ago

I _still_ think Kubernetes is insanely complex, despite all that it does. It seems less complex these days because it's so pervasive, but complex it remains. I'd like to see more emphasis on UX for v2 for the most common operations, like deploying an app and exposing it, then doing things like changing service accounts or images without having to drop into kubectl edit. Given that LLMs are it right now, this probably won't happen, but no harm in dreaming, right?

Pet_Ant 1y ago
Kubernetes itself contains so many layers of abstraction. There are pods, which is the core new idea, and it's great. But now there are deployments, and rep sets, and namespaces... and it makes me wish we could just use Docker Swarm. Even Terraform seems to live on just a single-layer and was relatively straight-forward to learn. Yes, I am in the middle of learning K8s so I know exactly how steep the curve is.
- nunez 1y ago
  The abstractions are fine and make a lot of sense the deeper you get into k8s. CRs for example. CRs are an amazing design concept. The fact that everything is a CR, with some exceptions (like everything in the v1 API) and their controllers are always Pods running somewhere is incredible. Debugging cluster irrops is so much easier once you internalize this. However, this exemplifies my desire for better UX. It would be great if "kubectl logs" accepted an CR kind, which would have the API server would automatically find its associated controller and work with the controller-manager to pump out its logs. Shit, I should make that PR, actually. Even better would be a (much improved) built-in UI that UIops people can use to do something like this. This will become extremely important once the ex-VMware VIadmins start descending onto Kubernetes clusters, which will absolutely happen given that k8s is probably the best vCenter alternative that exists (as non-obvious as that seems right now, though I work at Red Hat and am focused on OpenShift, so take that with two grains of salt)
- jakewins 1y ago
  The core idea isn’t pods. The core idea is reconciliation loops: you have some desired state - a picture of how you’d like a resource to look or be - and little controller loops that indefinitely compare that to the world, and update the world. Much of the complexity then comes from the enormous amount of resource types - including all the custom ones. But the basic idea is really pretty small. I find terraform much more confusing - there’s a spec, and the real world.. and then an opaque blob of something I don’t understand that terraform sticks in S3 or your file system and then.. presumably something similar to a one-shot reconciler that wires that all together each time you plan and apply?
  - vrosas 1y ago
    Someone saying "This is complex but I think I have the core idea" and someone to responding "That's not the core idea at all" is hilarious and sad. BUT ironically what you just laid out about TF is exactly the same - you just manually trigger the loop (via CI/CD) instead of a thing waiting for new configs to be loaded. The state file you're referencing is just a cache of the current state and TF reconciles the old and new state.
    - jauco 1y ago
      Always had the conceptual model that terraform executes something that resembles a merge using a three way diff. There’s the state file (base commit, what the system looked like the last time terraform succesfully executed). The current system (the main branch, which might have changed since you “branched off”) and the terraform files (your branch) Running terraform then merges your branch into main. Now that I’m writing this down, I realize I never really checked if this is accurate, tf apply works regardless of course.
      - mdaniel 1y ago
        and then the rest of the owl is working out the merge conflicts :-D I don't know how to have a cute git analogy for "but first, git deletes your production database, and then recreates it, because some attribute changed that made the provider angry"
throwaway5752 1y ago
I've come to think that it is a case of "the distinctions between types of computer programs are a human construct" problem. I agree with you on a human level. Operators and controllers remind me of COM and CORBA, in a sense. They are hightly abstract things, that are intrinsically so flexible that they allow judgement (and misjudgement) in design. For simple implementations, I'd want k8s-lite, that was more opinionated and less flexible. Something which doesn't allow for as much shooting ones' self in the foot. For very complex implementations, though, I've felt existing abstractions to be limiting. There is a reason why a single cluster is sometimes the basis for cell boundaries in cellular architectures. I sometimes wonder if one single system - kubernetes 2.0 or anything else - can encompass the full complexity of the problem space while being tractable to work with by human architects and programmers.
- nine_k 1y ago
  > I'd want k8s-lite, that was more opinionated and less flexible You seem to want something like https://skateco.github.io/ https://skateco.github.io/ (still compatible to k8s manifests). Or maybe even something like https://uncloud.run/ https://uncloud.run/ Or if you still want real certified Kubernetes, but small, there is https://k3s.io/ https://k3s.io/
  - mdaniel 1y ago
    Ah, so that explains it: https://github.com/skateco/skate#:~:text=leverages%20podman%20kube%20play%20to%20run%20pod%20manifests https://github.com/skateco/skate#:~:text=leverages%20podman%...
NathanFlurry 1y ago
[flagged]
- coderatlarge 1y ago
  where is that in the design space relative to where goog internal cluster management has converged to after the many years and the tens of thousands of engineers who have sanded it down under heavy fire since the original borg?
- mdaniel 1y ago
  I recognize that I'm biased, but you'll want to strongly consider whether https://rivet.gg/docs/config https://rivet.gg/docs/config is getting your audience where they can be successful, as compared to (e.g.) https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#pod-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap...
- stackskipton 1y ago
  Ops type here, after looking at Rivet, I've started doing The Office "Dear god no, PLEASE NO" Most people are looking for Container Management runtime with HTTP(S) frontend that will handle automatic certificate from Let's Encrypt. I don't want Functions/Actors or require this massive suite: FoundationDB: Actor state CockroachDB: OLTP ClickHouse: Developer-facing monitoring Valkey: Caching NATS: Pub/sub Traefik: Load balancers & tunnels This is just switching Kubernetes cloud lock in with KEDA and some other more esoteric operators to Rivet Cloud lock in. At least Kubernetes is slightly more portable than this. Oh yea, I don't know what Clickhouse is doing with monitoring but Prometheus/Grafana suite called, said they would love for you to come home.

by nunez 1y ago

Pet_Ant 1y ago

Kubernetes itself contains so many layers of abstraction. There are pods, which is the core new idea, and it's great. But now there are deployments, and rep sets, and namespaces... and it makes me wish we could just use Docker Swarm. Even Terraform seems to live on just a single-layer and was relatively straight-forward to learn. Yes, I am in the middle of learning K8s so I know exactly how steep the curve is.

nunez 1y ago
The abstractions are fine and make a lot of sense the deeper you get into k8s. CRs for example. CRs are an amazing design concept. The fact that everything is a CR, with some exceptions (like everything in the v1 API) and their controllers are always Pods running somewhere is incredible. Debugging cluster irrops is so much easier once you internalize this. However, this exemplifies my desire for better UX. It would be great if "kubectl logs" accepted an CR kind, which would have the API server would automatically find its associated controller and work with the controller-manager to pump out its logs. Shit, I should make that PR, actually. Even better would be a (much improved) built-in UI that UIops people can use to do something like this. This will become extremely important once the ex-VMware VIadmins start descending onto Kubernetes clusters, which will absolutely happen given that k8s is probably the best vCenter alternative that exists (as non-obvious as that seems right now, though I work at Red Hat and am focused on OpenShift, so take that with two grains of salt)
jakewins 1y ago
The core idea isn’t pods. The core idea is reconciliation loops: you have some desired state - a picture of how you’d like a resource to look or be - and little controller loops that indefinitely compare that to the world, and update the world. Much of the complexity then comes from the enormous amount of resource types - including all the custom ones. But the basic idea is really pretty small. I find terraform much more confusing - there’s a spec, and the real world.. and then an opaque blob of something I don’t understand that terraform sticks in S3 or your file system and then.. presumably something similar to a one-shot reconciler that wires that all together each time you plan and apply?
- vrosas 1y ago
  Someone saying "This is complex but I think I have the core idea" and someone to responding "That's not the core idea at all" is hilarious and sad. BUT ironically what you just laid out about TF is exactly the same - you just manually trigger the loop (via CI/CD) instead of a thing waiting for new configs to be loaded. The state file you're referencing is just a cache of the current state and TF reconciles the old and new state.
  - jauco 1y ago
    Always had the conceptual model that terraform executes something that resembles a merge using a three way diff. There’s the state file (base commit, what the system looked like the last time terraform succesfully executed). The current system (the main branch, which might have changed since you “branched off”) and the terraform files (your branch) Running terraform then merges your branch into main. Now that I’m writing this down, I realize I never really checked if this is accurate, tf apply works regardless of course.
    - mdaniel 1y ago
      and then the rest of the owl is working out the merge conflicts :-D I don't know how to have a cute git analogy for "but first, git deletes your production database, and then recreates it, because some attribute changed that made the provider angry"

throwaway5752 1y ago

I've come to think that it is a case of "the distinctions between types of computer programs are a human construct" problem. I agree with you on a human level. Operators and controllers remind me of COM and CORBA, in a sense. They are hightly abstract things, that are intrinsically so flexible that they allow judgement (and misjudgement) in design. For simple implementations, I'd want k8s-lite, that was more opinionated and less flexible. Something which doesn't allow for as much shooting ones' self in the foot. For very complex implementations, though, I've felt existing abstractions to be limiting. There is a reason why a single cluster is sometimes the basis for cell boundaries in cellular architectures. I sometimes wonder if one single system - kubernetes 2.0 or anything else - can encompass the full complexity of the problem space while being tractable to work with by human architects and programmers.

nine_k 1y ago
> I'd want k8s-lite, that was more opinionated and less flexible You seem to want something like https://skateco.github.io/ https://skateco.github.io/ (still compatible to k8s manifests). Or maybe even something like https://uncloud.run/ https://uncloud.run/ Or if you still want real certified Kubernetes, but small, there is https://k3s.io/ https://k3s.io/
- mdaniel 1y ago
  Ah, so that explains it: https://github.com/skateco/skate#:~:text=leverages%20podman%20kube%20play%20to%20run%20pod%20manifests https://github.com/skateco/skate#:~:text=leverages%20podman%...

NathanFlurry 1y ago

[flagged]

coderatlarge 1y ago
where is that in the design space relative to where goog internal cluster management has converged to after the many years and the tens of thousands of engineers who have sanded it down under heavy fire since the original borg?
mdaniel 1y ago
I recognize that I'm biased, but you'll want to strongly consider whether https://rivet.gg/docs/config https://rivet.gg/docs/config is getting your audience where they can be successful, as compared to (e.g.) https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#pod-v1-core https://kubernetes.io/docs/reference/generated/kubernetes-ap...
stackskipton 1y ago
Ops type here, after looking at Rivet, I've started doing The Office "Dear god no, PLEASE NO" Most people are looking for Container Management runtime with HTTP(S) frontend that will handle automatic certificate from Let's Encrypt. I don't want Functions/Actors or require this massive suite: FoundationDB: Actor state CockroachDB: OLTP ClickHouse: Developer-facing monitoring Valkey: Caching NATS: Pub/sub Traefik: Load balancers & tunnels This is just switching Kubernetes cloud lock in with KEDA and some other more esoteric operators to Rivet Cloud lock in. At least Kubernetes is slightly more portable than this. Oh yea, I don't know what Clickhouse is doing with monitoring but Prometheus/Grafana suite called, said they would love for you to come home.