Okay, if we're only considering active health checks, then I'm not sure any load balancer considers a 5xx a success by default, let alone a "production" load ba

Okay, if we're only considering active health checks, then I'm not sure any load balancer considers a 5xx a success by default, let alone a "production" load balancer. For a non-healthcheck 5xx response, it almost never is clear that this host in itself is responsible for the 5xx response. 5xx is the correct response when there is an error on the server side (ie, not an error on the client side, not a correct response), but it doesn't mean the server is a problem - it just means that the server experienced a problem in serving the request. That failure itself may be from one of many RPCs that server made to other services. As such, all web servers behind the load balancer for that request type will exhibit the 5xx response (at some rate, and depending on any state in connection sharing/reuse between the server and their upstream service), and all would subsequently be removed. Which isn't the correct response at all. As someone who has had the job title "Systems Administrator" and the job title "Software Engineer", and currently has neither but still does exactly what he's always done - solving problems by understanding systems and, among other things, by writing code - I wouldn't consider load balancing and failure domains/types/handling as the sole or even primary purview of a systems administrator - especially in the case of large installations.