3 ms·

Great "teaching by showing" example but I don't get the initial technical scenario; you have sophisticated (ie, not RR / random) load balancers that keep track

Great "teaching by showing" example but I don't get the initial technical scenario; you have sophisticated (ie, not RR / random) load balancers that keep track of queues in web servers, so they have to get some information back from them (when job was completed, from HTTP response for ex), but somehow don't react to 500 errors? seems like badly configured LBs. Still scenario can be used as teaching or interview question.

camtarn 9y ago
I've worked on a system which communicated errors by responding with a 200 and then indicating the error in the response body! Similarly, you might have e.g. a server which is responding to a 'get all widgets in category' call by quickly responding '200 OK, no widgets found', and thus ends up sucking up all the traffic.
- asmithmd1 9y ago
  Exactly, you need to have a health check that actually exercises all the layers of the system and returns real data. You don't want to use actual data, so you add a dummy account. Someone deletes the dummy account all the servers that are working perfectly start reporting they are sick.
wging 9y ago
Suppose that a machine succeeds fast, not fails fast. Say a machine's job is to run N things on its input, and the identities of these N things are picked up from configuration somewhere. One machine gets a corrupted config somehow, and has only 1 thing to run. The others are running 10 things. So one machine is (barring framework/networking/etc. overhead) 10x faster than the rest, and it thinks it's doing fine. Now even if your mind is stuck on HTTP, you have to think about the same problem but with 200 response codes. Still a non-issue? And what if the responses are opaque to the LB? TCP-level load balancing, maybe? There are reasons you might choose that. Or maybe the protocol you use is not HTTP-based and you don't have a way to teach your LB about it.

camtarn 9y ago

I've worked on a system which communicated errors by responding with a 200 and then indicating the error in the response body! Similarly, you might have e.g. a server which is responding to a 'get all widgets in category' call by quickly responding '200 OK, no widgets found', and thus ends up sucking up all the traffic.

asmithmd1 9y ago
Exactly, you need to have a health check that actually exercises all the layers of the system and returns real data. You don't want to use actual data, so you add a dummy account. Someone deletes the dummy account all the servers that are working perfectly start reporting they are sick.

wging 9y ago

Suppose that a machine succeeds fast, not fails fast. Say a machine's job is to run N things on its input, and the identities of these N things are picked up from configuration somewhere. One machine gets a corrupted config somehow, and has only 1 thing to run. The others are running 10 things. So one machine is (barring framework/networking/etc. overhead) 10x faster than the rest, and it thinks it's doing fine. Now even if your mind is stuck on HTTP, you have to think about the same problem but with 200 response codes. Still a non-issue? And what if the responses are opaque to the LB? TCP-level load balancing, maybe? There are reasons you might choose that. Or maybe the protocol you use is not HTTP-based and you don't have a way to teach your LB about it.