3 ms·
The node would be a virtual-kubelet. You can check out the virtual-kubelet GitHub repo for more info. Interestingly, there are already multiple providers of vi
by kuhsaft 3y ago
The node would be a virtual-kubelet. You can check out the virtual-kubelet GitHub repo for more info.
Interestingly, there are already multiple providers of virtual-kubelet. For example, Azure AKS has virtual nodes where pods are Azure Container Instances. There’s even a Nomad provider.
> So that’s what we do. When you create a cluster, we run K3s and the Virtual Kubelet on a single Fly Machine.
So probably a cluster per region. You could theoretically spin up multiple virtual-kubelets though and configure each one as a specific region.
> Because of kine, K3s can manage multiple servers, but also gracefully runs on a single server, without distributed state.
This would mean the control-plane would be on a single-server without high-availability? Although, I suppose there really isn’t any state stored since they are just proxying requests to the Fly Machine API. But still, if the machine went down your kubectl commands wouldn’t work.
- philsnow 3y agoThe diagram on https://virtual-kubelet.io/docs/architecture/ https://virtual-kubelet.io/docs/architecture/ makes me wonder whether it's possible to have a k8s cluster where the nodes are all virtual kubelets backed by different cloud providers (and then perhaps schedule loads preferentially with selectors)
- kuhsaft 3y agoI think it’s completely possible. Though, you’ll have to manage your own control-plane. Azure AKS and EKS provide virtual-kubelet functionality in some form, but AKS is an a managed control-plane where you can’t add nodes yourself and EKS only allows nodes in the same VPC. Edit: It already is a thing. https://github.com/virtual-kubelet/tensile-kube https://github.com/virtual-kubelet/tensile-kube
- philsnow 3y agotensile-kube seems to be structured as a "k8s cluster of k8s clusters", with an upper kubemaster farming out resources to lower kubemasters (through virtual-node). I don't know if there's any particular reason to have that separation; possibly the lower kubemasters could be removed and you could just run a bunch of virtual-kubelets.
- kuhsaft 3y agoI think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.
- philsnow 3y agoI've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.
- harpratap 3y agoWouldn't the network cost be absurd in such case? Not only the pod-to-pod communication cost skyrocket, all the heartbeats, health checks, metrics, daemonsets pinging each other will probably end up costing more than the CPU and Memory