Here's my Kubernetes security horror story. You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never

Here's my Kubernetes security horror story. You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter). So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf. Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance... And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse. Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting. Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity. --- I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.