4 ms·

The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) t

by fatbird 14y ago

The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.

sofal 14y ago
As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.
csense 14y ago
The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.
- jlouis 14y ago
  Indeed, and this is another point where rapgenius statistical model is wrong compared to the real world behavior of the system.

by fatbird 14y ago

sofal 14y ago

As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.

csense 14y ago

The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.

jlouis 14y ago
Indeed, and this is another point where rapgenius statistical model is wrong compared to the real world behavior of the system.