I know a guy who runs a small company. He contracted a tech wiz to set up the initial infrastructure. AWS, EKS, all built with Terraform modules that wrapped ot

I know a guy who runs a small company. He contracted a tech wiz to set up the initial infrastructure. AWS, EKS, all built with Terraform modules that wrapped other terraform modules, some of which wrapped helm charts. The "proper" way to do it, right! Russian dolls, each with their own names, pinned version and weird limitations! Everything was fine for a year or two, but suddenly he noticed a warning message in the EKS console. The latest version available was 1.24. There's an upgrade button. "How hard could this be?" Well, very hard, as it happens. Time to take apart the Russian dolls! This is where pinning versions and wrapping modules becomes a gigantic pain. To see how one surface level change is made you need to go backwards, one doll at a time, hoping that the important option is exposed as a variable in the smallest doll and every doll that subsequently encases it, and that the name of the variable names don't change anywhere (because sometimes infrastructure engineers wrap a whole bunch of changes as a "set" with one big on/off switch - again, a sensible approach sometimes!). It looked like every single module and helm chart was badly out of date. If our unfortunate CEO tried to do a terraform apply from scratch, none of this code would work any more. So what do we do? How about forgetting how the cluster got there, ditching all the decaying terraform and helm crap and simply pushing the "Upgrade" button!? As you might expect, the whole thing fell apart. Time to bring in help! On inspecting the change logs there were a heap of breaking changes; from renamed apiVersions to removal of deprecated APIs to new requirements for CNI and EBS volumes to entire sections of "we don't do it that way any more". His Persistent Volumes wouldn't attach, Cert-Manager no longer worked, his ingresses disappeared because they didn't specifically name an ingressClass, one pod even had a direct dependence on docker being the container runtime (and that was no longer the case). Fixing all of this directly, ie: ignoring the decayed terraform code that was used to build it, took considerable time. You need to get new helm repos and charts, inspect successive upgrade guides for breaking changes, extract current helm values for each installed chart and update them, then remove old charts (and manually delete CRDs) and try installing new ones. You can't use the monitoring solution because it died in the fire. Let's use OpenLens to check logs and events. Very slowly put the pieces together: fix Ingress, DNS, TLS, ENS/CNI addons (and the IRSA crap to make them work) and finally upgrade your product charts so they are compatible. Doing that via Russian doll terraform and helm would have been absolute hell. Q: Does any tool in existence make this type of problem go away? EKS, GKE, OKE already kinda manage the control plane, and I'm happy to let them do it. FluxCD and ArgoCD (and others) help you manage your deployments. What sews it all up and watches everything, making your Russian dolls go away?