3 ms·

Why don't you just put your Kubernetes in Kubernetes? Containers solve everything re: dependency management, right?

deleted 3y ago
[deleted]
dilyevsky 3y ago
We actually did just that with just apiserver+sqlite-based storage and it works pretty well. Ofc you have the same problem with an outer cluster now but if you’re careful with locking your entire state into the “inner” cluster you can escape that
geodel 3y ago
Fully agree. They can follow Cloud Native deployment best practices for guidance.
nonameiguess 3y ago
I think you're saying this sarcastically, but every Kubernetes component except the kubelet already can, and usually does in most distros, run as a Pod managed by Kubernetes itself. Edit the manifest, systemctl restart kubelet, and there you go, you just upgraded. Other than kube-proxy, which has a dependency on iptables, I don't even think any other component has any external dependencies, and the distroless containers they run in are pretty close to scratch. Running them in Pods is more for convenience than dependency management. The actual issue that needs to be addressed when upgrading is API deprecations and incompatibility. For better or worse, when people try to analogize this to the way Red Hat or Debian works, Kubernetes is not Linux. It may not always perfectly achieve the goal, but Linux development operates on the strict rule that no changes can ever break userspace. Kubernetes, in contrast, flat out tells you that generally available APIs will stay available, but if you use beta or alpha APIs, those may or may not exist in future releases. Use them at your own peril. In practice, this has meant virtually every cluster in existence and virtually every cluster management tool created by third parties, uses at least beta APIs. Nobody heeded the warning. So upgrades break stuff all the time. Should they have not ever released beta APIs at all? Maybe. They actually did start turning them off by default a few releases back, but the cultural norms are already there now. Everybody just turns them back on so Prometheus and cert-manager and Istio and what not will continue working.
- airocker 3y ago
  minikube does exactly that! Runs new versions of kubernetes inside a single Docker container.

deleted 3y ago

[deleted]

We actually did just that with just apiserver+sqlite-based storage and it works pretty well. Ofc you have the same problem with an outer cluster now but if you’re careful with locking your entire state into the “inner” cluster you can escape that

geodel 3y ago

Fully agree. They can follow Cloud Native deployment best practices for guidance.

nonameiguess 3y ago

I think you're saying this sarcastically, but every Kubernetes component except the kubelet already can, and usually does in most distros, run as a Pod managed by Kubernetes itself. Edit the manifest, systemctl restart kubelet, and there you go, you just upgraded. Other than kube-proxy, which has a dependency on iptables, I don't even think any other component has any external dependencies, and the distroless containers they run in are pretty close to scratch. Running them in Pods is more for convenience than dependency management. The actual issue that needs to be addressed when upgrading is API deprecations and incompatibility. For better or worse, when people try to analogize this to the way Red Hat or Debian works, Kubernetes is not Linux. It may not always perfectly achieve the goal, but Linux development operates on the strict rule that no changes can ever break userspace. Kubernetes, in contrast, flat out tells you that generally available APIs will stay available, but if you use beta or alpha APIs, those may or may not exist in future releases. Use them at your own peril. In practice, this has meant virtually every cluster in existence and virtually every cluster management tool created by third parties, uses at least beta APIs. Nobody heeded the warning. So upgrades break stuff all the time. Should they have not ever released beta APIs at all? Maybe. They actually did start turning them off by default a few releases back, but the cultural norms are already there now. Everybody just turns them back on so Prometheus and cert-manager and Istio and what not will continue working.

airocker 3y ago
minikube does exactly that! Runs new versions of kubernetes inside a single Docker container.