Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis. If you can't have that (and budget allows) keep unhe

Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis. If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers. In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.