3 ms·
I mostly use it, but I also tend to keep to stable features so that running on-prem is also relatively simple. My biggest cryparty story when it comes to on-pr
by p_l 7y ago
I mostly use it, but I also tend to keep to stable features so that running on-prem is also relatively simple.
My biggest cryparty story when it comes to on-prem kubernetes is not actually due to kubernetes, but due to Red Hat. There are words I could say about their official OpenShift deployment scripts and their authors, but I would be rightly banned for writing them.
Biggest issues I've encountered involve things on the interface between "classic distro" and running k8s on top of it, and that goes down when you move towards base OS that is more optimized for k8s (for example flatcar).
When it comes to the size of team involved, I'd say keeping "classic" stack with loadbalancers, pacemaker, custom deployment methods etc. was comparable effort - at least if we're matching feature for feature what I'd see on "base" k8s on-prem setup (and based on "what I wish I had, or that I could replace with k8s, back in 2016 for an on-prem project).
There's one thing, however, where it gets much harder to deploy k8s and I won't hide it - when you're dealing with random broken classic infrastructure with no authority to change it. K8s gets noticeably harder when you need to deal with annoying pre-existing IP layouts because you can't get IP address space allocated, when you have to sit on badly stretched L2 domains, where the local idea of routing is OSPF - or worse, static routes to gateway and a stretched L2. To the point that sometimes it's easier to setup a base "private cloud" (I like VMware for this) on the physical machines, then setup the hosts on top of that - even if you're only going to have one VM per hypervisor. The benefits of abstracting possibly broken infrastructure are too big.
- Spivak 7y ago> Badly stretched L2 domains, where the local idea of routing is OSPF Hahaha… welp. So you’re saying that stretching every “overlay” L2 domain to every hypervisor/physical host with VXLANs and OSPF isn’t maintainable. Color me surprised. I need a drink.
- p_l 7y agoHey, at least you got VXLAN! And OSPF! Though I do have reasons to dislike it from past life as network admin, and somehow there are no integrations - or at least none I found - for integrating things into OSPF routing area (like with ExaBGP). Dealing with overstretched VLANs where somehow, somehow, STP ("What is that RSTP you're telling us about?") decided to put a "trung" across random slow link >_>