3 ms·

It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just

by csense 14y ago

It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just shard the cluster. I.e., give their bigger customers like RapGenius (who said they pay Heroku $20k/month and whose HN post spurred this debate) their own dedicated routing cluster with 2-5 nodes. Once a single customer exceeds that size, they're probably paying Heroku enough that Heroku can afford to devote an engineer to working specifically with them to implement large-scale architecture choices like running the app's DB on a cluster, etc. Pile several smaller customers on a shared routing cluster to cut costs and keep the cluster utilization high, but once the cluster gets to be a certain size (reaches a fixed number of backend dynos or metrics get bad enough), start putting customers on a new cluster. It should be fairly trivial to use DNS or router rules to dynamically move existing customers from one routing cluster to another.

fatbird 14y ago
The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.
- sofal 14y ago
  As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.
- csense 14y ago
  The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.
  - jlouis 14y ago
    Indeed, and this is another point where rapgenius statistical model is wrong compared to the real world behavior of the system.

by csense 14y ago

fatbird 14y ago

The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.

sofal 14y ago
As the artictle says, it was the gradual scaling of the router cluster that eventually led to the request routing being effectively random on the Bamboo stack.
csense 14y ago
The problem, as documented by the article, is that requests have always been served randomly, but the maximum number of requests that can be queued for any single backend node is equal to the number of frontend nodes N in the routing cluster. When N is equal to one, it's exactly what the previous discussion has labeled "intelligent routing;" when N is small, it's similar enough to intelligent routing that few will notice the difference. As N becomes large, you need a proportional (hence also large) number of requests to trigger the load balancing feature. At some point, the load balancing no longer kicks in, because for any real-world application the total workload is finite. But the performance complaints start much earlier; the "load balancing" on e.g. a 50-node balancing cluster might still kick in and stop routing requests to a hung server, but the 49 users waiting behind the hung request still suffer the latency and complain. I've adopted this picture of the situation because it agrees with both the reported behavior discussed by RapGenius and others in the previous thread, and this article's discussion of Heroku's architecture.
- jlouis 14y ago
  Indeed, and this is another point where rapgenius statistical model is wrong compared to the real world behavior of the system.