4 ms·

> But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. The thing is, their old "intelligent ro

by chc 14y ago

> But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. The thing is, their old "intelligent routing" was really just "we will only route one request at a time to a dyno." In other words, what changed is that they now allow dynos to serve multiple requests at a time. When you put it that way, it doesn't sound as horrific, does it?

arcatek 14y ago
The requests will not be served at the same time, that's the whole point. If a request is routed to a busy dyno, you will have to wait that the previous job finish before being able to start yours.
- damian2000 14y ago
  I believe the article is saying that this wait will only occur with Rails due to it being single threaded? Reactor pattern frameworks like node don't suffer the same issue.
  - teraflop 14y ago
    They will if the request is CPU-bound. In that case, throwing more concurrent requests at a server than it has cores just slows all of them down.
    - joseph_cooney 14y ago
      How many web requests are typically CPU bound though? Not many.
      - vidarh 14y ago
        Doesn't matter. At some point you hit memory limits, disk IO limits, or simply a connection limit. It doesn't matter what limit: If you have some requests that are longer running than others, random load balancing will make them start piling up once you reach some traffic threshold. You can increase the threshold by adding more backends or increasing the capacity of each backend (by optimizing, or picking beefier hardware if you're on a platform that will let you), and maybe you can increase it enough that it won't affect you. But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing. If you're lucky, the effect might be small enough to not cost you much money, and you might be able to ignore it, but it's still there.
        chc 14y ago
        > But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing. I think we have to remember that the "intelligent routing" in question here is actually marketing-speak for "one request per server." Are you saying that when your servers can only receive one request at a time, you will necessarily need fewer than if your servers can handle three requests at a time but are assigned requests randomly?
        Dylan16807 14y ago
        No it's not. "Intelligent routing" means giving each server exactly as many requests as it can handle, and no more. If your servers can handle 3 requests each, using intelligent routing to make sure they never get more than 3 will blow the latency of the random method out of the water.
    - JPKab 14y ago
      You are exactly right, but I would hope that someone wouldn't be using Node at all if CPU bound requests are even remotely commoneplace on their app. But of course, if they built their app to spawn processes when this happens, you end up with a case of a need for intelligent routing.
- antihero 14y ago
  Thing is though, it says that a Dyno is a Ubuntu virtual machine. In what sort of horrendous configuration can an ENTIRE VM serve only a SINGLE REQUEST AT A TIME?! That is utter madness, and the validity of the argument depends on whether it's the Heroku or this dude's fault that the VM is serving only a single request at a time (and it taking >1sec to handle a request).
  - vemv 14y ago
    Not Heroku's fault in this case, Rails (and any other single-threaded environment) can handle a single request at a time.
    - joesb 14y ago
      But one VM can host multiple Rails instances, each on different port. That's what Passenger or Unicorn do, acting as proxy of group of locally spawn Rails instances.
Dylan16807 14y ago
Yes it does sound as horrific. It should never route more than N requests to a dyno, where N is the number of requests it can handle simultaneously. It doesn't matter if N is 1, 2, or 10. And it should always send new requests to the emptiest dyno.
- chc 14y ago
  While I agree that routing to the emptiest dyno would be a good thing, I think it's a little melodramatic to describe giving every dyno at least three times and up to several thousand times as much power as "horrific."
  - Dylan16807 14y ago
    Power, what? The problem is when you send a dyno that has all its threads stuck on long-running computations a new request, because it won't be able to even start processing it. The power is orthogonal to the problem. The only mitigation is that if a dyno can handle a large number of threads, it probably won't get clogged. But if it can only handle 3 and gets new requests at random, you're in a bad place.

by chc 14y ago

arcatek 14y ago

The requests will not be served at the same time, that's the whole point. If a request is routed to a busy dyno, you will have to wait that the previous job finish before being able to start yours.

damian2000 14y ago
I believe the article is saying that this wait will only occur with Rails due to it being single threaded? Reactor pattern frameworks like node don't suffer the same issue.
- teraflop 14y ago
  They will if the request is CPU-bound. In that case, throwing more concurrent requests at a server than it has cores just slows all of them down.
  - joseph_cooney 14y ago
    How many web requests are typically CPU bound though? Not many.
    - vidarh 14y ago
      Doesn't matter. At some point you hit memory limits, disk IO limits, or simply a connection limit. It doesn't matter what limit: If you have some requests that are longer running than others, random load balancing will make them start piling up once you reach some traffic threshold. You can increase the threshold by adding more backends or increasing the capacity of each backend (by optimizing, or picking beefier hardware if you're on a platform that will let you), and maybe you can increase it enough that it won't affect you. But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing. If you're lucky, the effect might be small enough to not cost you much money, and you might be able to ignore it, but it's still there.
      - chc 14y ago
        > But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing. I think we have to remember that the "intelligent routing" in question here is actually marketing-speak for "one request per server." Are you saying that when your servers can only receive one request at a time, you will necessarily need fewer than if your servers can handle three requests at a time but are assigned requests randomly?
        Dylan16807 14y ago
        No it's not. "Intelligent routing" means giving each server exactly as many requests as it can handle, and no more. If your servers can handle 3 requests each, using intelligent routing to make sure they never get more than 3 will blow the latency of the random method out of the water.
  - JPKab 14y ago
    You are exactly right, but I would hope that someone wouldn't be using Node at all if CPU bound requests are even remotely commoneplace on their app. But of course, if they built their app to spawn processes when this happens, you end up with a case of a need for intelligent routing.
antihero 14y ago
Thing is though, it says that a Dyno is a Ubuntu virtual machine. In what sort of horrendous configuration can an ENTIRE VM serve only a SINGLE REQUEST AT A TIME?! That is utter madness, and the validity of the argument depends on whether it's the Heroku or this dude's fault that the VM is serving only a single request at a time (and it taking >1sec to handle a request).
- vemv 14y ago
  Not Heroku's fault in this case, Rails (and any other single-threaded environment) can handle a single request at a time.
  - joesb 14y ago
    But one VM can host multiple Rails instances, each on different port. That's what Passenger or Unicorn do, acting as proxy of group of locally spawn Rails instances.

Dylan16807 14y ago

Yes it does sound as horrific. It should never route more than N requests to a dyno, where N is the number of requests it can handle simultaneously. It doesn't matter if N is 1, 2, or 10. And it should always send new requests to the emptiest dyno.

chc 14y ago
While I agree that routing to the emptiest dyno would be a good thing, I think it's a little melodramatic to describe giving every dyno at least three times and up to several thousand times as much power as "horrific."
- Dylan16807 14y ago
  Power, what? The problem is when you send a dyno that has all its threads stuck on long-running computations a new request, because it won't be able to even start processing it. The power is orthogonal to the problem. The only mitigation is that if a dyno can handle a large number of threads, it probably won't get clogged. But if it can only handle 3 and gets new requests at random, you're in a bad place.