3 ms·
I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset. Usually when a service conks out, operations will just kill it and roll out ano
by wkat4242 3y ago
I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset.
Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then.
Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.
- _ea1k 3y agoYou mean kubernetes and not kerberos?
- wkat4242 3y agoOh yes kubernetes sorry!
- chronid 3y agoYou need better observability or tooling then. Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
- ilyt 3y agoYou achieve availability by redundancy, not by running around with shotgun murdering your cattle.
- chronid 3y agoDefinitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis. If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers. In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.