3 ms·
Inherited about 12 Kube clusters with 200+ charts, which were 8 versions behind on kube version, running multiple ingress, metrics, secrets, routing and logging
by yuppie_scum 4y ago
Inherited about 12 Kube clusters with 200+ charts, which were 8 versions behind on kube version, running multiple ingress, metrics, secrets, routing and logging type schemes per cluster. since they were built really before Kube was functionally stable and there was a “you build it, you own it, you get laid off, nobody understands the app anymore” culture. No consistency in language, build ideology observability, or anything like that.
Had to cut a new Kube platform release, spin up some separate clusters with EKS vs crappy provisioner software, create standards and docs, and slowly and painstakingly work with disinterested devs to migrate to the new single standard.
The modernization and standardization work has definitely paid dividends. No more random cluster failures, costs under control, we have actual observability and certainty around compliance. It was a huge pain in the ass but I am pretty proud of the effort.
- mbrodersen 4y agoWow. How long did it take to turn the ship around?
- yuppie_scum 4y agoAbout 2 calendar years (Pandemic probably added a lot of person hours to the effort) - still not complete. This was kinda a rogue effort/moonshot by me but I think if we didn’t do it, management was gonna do something drastic like move everything to data centers or something.