6 ms·
The diagram on https://virtual-kubelet.io/docs/architecture/ https://virtual-kubelet.io/docs/architecture/ makes me wonder whether it's possible to have a k8s c
by philsnow 3y ago
The diagram on https://virtual-kubelet.io/docs/architecture/ https://virtual-kubelet.io/docs/architecture/ makes me wonder whether it's possible to have a k8s cluster where the nodes are all virtual kubelets backed by different cloud providers (and then perhaps schedule loads preferentially with selectors)
- kuhsaft 3y agoI think it’s completely possible. Though, you’ll have to manage your own control-plane. Azure AKS and EKS provide virtual-kubelet functionality in some form, but AKS is an a managed control-plane where you can’t add nodes yourself and EKS only allows nodes in the same VPC. Edit: It already is a thing. https://github.com/virtual-kubelet/tensile-kube https://github.com/virtual-kubelet/tensile-kube
- philsnow 3y agotensile-kube seems to be structured as a "k8s cluster of k8s clusters", with an upper kubemaster farming out resources to lower kubemasters (through virtual-node). I don't know if there's any particular reason to have that separation; possibly the lower kubemasters could be removed and you could just run a bunch of virtual-kubelets.
- kuhsaft 3y agoI think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.
- philsnow 3y agoI've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.
- harpratap 3y agoWouldn't the network cost be absurd in such case? Not only the pod-to-pod communication cost skyrocket, all the heartbeats, health checks, metrics, daemonsets pinging each other will probably end up costing more than the CPU and Memory