3 ms·
If your k8s nodes are the same height and width as your Proxmox servers, then yes. In my case, I have clusters of ~7 nodes, 100 cores, 1.75TB RAM, but my VMs t
by linsomniac 3y ago
If your k8s nodes are the same height and width as your Proxmox servers, then yes. In my case, I have clusters of ~7 nodes, 100 cores, 1.75TB RAM, but my VMs tend to be much, much smaller. My web workers are each a handful of cores, I could double them a couple times before saturating the physical nodes core count.
- donutshop 3y agoWow that is some serious horsepower.
- withinboredom 3y agoI guess I just don't get why you'd have such small k8s nodes in a cluster, I have a cluster about half that size (in node and core count) but just run k8s directly on the metal. It means that things are quite elastic with an HPA + VPA on things that need it. I feel like we get better overall utilization of the hardware, but I'd be interested in hearing about some of the reasons for your approach. Off the top of my head, there are probably a couple of 'pros' to doing it your way; one: experimentation is probably easier. Two, I'm willing to bet networking isn't anything too fancy (getting k8s to play nice on bare metal networks is a shitty thing, especially once multiple NICs are involved -- a LOT of networking stacks, ok, all of them, just 'assume' that the first network card it sees is THE network card it should use... and don't even bother to 'sort' them so its the same order every reboot).
- Already__Taken 3y agoselfish question but I'm about to give talos/cilium a go, have you evaluated that? my smaller testing was encouraging
- withinboredom 3y agoYep. Running Cilium in raw L2 mode (no encapsulation) on the lan, and even using external IP's in L2 mode to provide load balancing (currently a beta feature, and switched from MetalLb). It has worked beautifully with very few issues.
- linsomniac 3y agoIf you're running k8s on Proxmox, it's probably because you aren't 100% k8s yet, or are a homelab and experimenting with many things.