4 ms·

One thing you need to be careful of when load balancing like this is buggy servers. Let's say you have a bug that makes a server handle requests by immediately

by foota 2y ago

One thing you need to be careful of when load balancing like this is buggy servers. Let's say you have a bug that makes a server handle requests by immediately failing them. That machine will start to look like it's performing really well, and so you start sending more requests to it, and more fail, etc.,. This can turn even a tiny canary of a build into a complete outage. I'm also surprised the article says that is took days to equalize usage, it's unclear, but hopefully this is just because the were rolling it out slowly?

tom1337 2y ago
From the original article: > A quick note on our migration strategy It can be risky to switch load balancing strategies all at once. This is why we enable service owners to configure multiple load balancing strategies for a service in Robinhood. The LBS writes the weighted endpoints list generated by different strategies into different entries in the routing database. At Dropbox, we have a percentage-based feature gate, so we implement a mixed strategy where clients use the weighted sum of the weights generated by two load balancing strategies as the endpoint weight. For example, endpoint A might be weighted at 100 based on PID-based load balancing and 200 based on simple round-robin strategy. If we set the feature gate to 30% for PID-based load balancing, the weight of endpoint A becomes 100*0.3 + 200*0.7 = 170. This way, we can ensure that every client sees the same weight assignment for endpoints while gradually migrating to the new load balancing strategy. https://dropbox.tech/infrastructure/robinhood-in-house-load-balancing-service https://dropbox.tech/infrastructure/robinhood-in-house-load-...
- bch 2y ago
  ^-- mind atm the markup here is catching the poster by surprise. The math should read: 100*0.3 + 200*0.7 = 170. Not 1000.3 + 2000.7 (escape the *, or it bounds an italic section - *this* becomes this. \*this\* becomes *this*.)
  - tom1337 2y ago
    oops. should be fixed now
nickpsecurity 2y ago
“Let’s say you have a bug that makes a server handling requests by immediately failing them.” Perhaps such designs should build progress indicators into the servers’ responses. If not visible (eg crytpo), maybe the response size is a hint where the error messages are always a specific, small size. The load balancer would then look for progress indicators or error signatures when assigning weights.
indigodaddy 2y ago
The buggy servers would still pass health checks though?

by foota 2y ago

tom1337 2y ago

From the original article: > A quick note on our migration strategy It can be risky to switch load balancing strategies all at once. This is why we enable service owners to configure multiple load balancing strategies for a service in Robinhood. The LBS writes the weighted endpoints list generated by different strategies into different entries in the routing database. At Dropbox, we have a percentage-based feature gate, so we implement a mixed strategy where clients use the weighted sum of the weights generated by two load balancing strategies as the endpoint weight. For example, endpoint A might be weighted at 100 based on PID-based load balancing and 200 based on simple round-robin strategy. If we set the feature gate to 30% for PID-based load balancing, the weight of endpoint A becomes 100*0.3 + 200*0.7 = 170. This way, we can ensure that every client sees the same weight assignment for endpoints while gradually migrating to the new load balancing strategy. https://dropbox.tech/infrastructure/robinhood-in-house-load-balancing-service https://dropbox.tech/infrastructure/robinhood-in-house-load-...

bch 2y ago
^-- mind atm the markup here is catching the poster by surprise. The math should read: 100*0.3 + 200*0.7 = 170. Not 1000.3 + 2000.7 (escape the *, or it bounds an italic section - *this* becomes this. \*this\* becomes *this*.)
- tom1337 2y ago
  oops. should be fixed now

nickpsecurity 2y ago

“Let’s say you have a bug that makes a server handling requests by immediately failing them.” Perhaps such designs should build progress indicators into the servers’ responses. If not visible (eg crytpo), maybe the response size is a hint where the error messages are always a specific, small size. The load balancer would then look for progress indicators or error signatures when assigning weights.

indigodaddy 2y ago

The buggy servers would still pass health checks though?