2 ms·

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/100 requests failed? That's not happy-making. Uncertainty is DiaI (death-in-an-infrastructure). I just created a couple of projects on Heroku and love the service, but this needs to be addressed ASAP (even if addressing it is just a blog post). Also, if you have fewest-connections available, I've never understood using round-robin or random algorithms for load-balancers...

bbatsell 14y ago
> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.
- CoffeeDregs 14y ago
  AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]
  - bbatsell 14y ago
    I've seen it frequently, and yes, this is how it behaves, e.g.: [1]: http://stackoverflow.com/a/579466/94830 http://stackoverflow.com/a/579466/94830 [2]: https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 I imagine the rationale was something along the lines of many servers/apps are written to incorrectly return 200 with a descriptive error page rather than 500 or whatever the appropriate status code would be. And at the time ab was first written, pages were a lot more static than they are now, so a different page would be more likely to indicate an incorrect response.
  - stock_toaster 14y ago
    use wrk[1] or weighttp[2] instead. [1]: https://github.com/wg/wrk https://github.com/wg/wrk [2]: redmine.lighttpd.net/projects/weighttp/wiki
donavanm 14y ago
> I've never understood using round-robin or random algorithms for load-balancers... LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it. A single host failure just became a major outage. I like WRR selection for back ends, then use a queue or fast fail when your max active conns is exceeded. Humans prefer latency to errors, so let the lb queue (to a limit) on human centric VIPs. Automated clients deal better with errors so have your lb throw a 500 directly, or RST, or direct to a pool that serves static content.
- shabble 14y ago
  You'd have some sort of error threshold/rate limit[1] at which point the server is marked dead and falls out of the backends list, surely? Or even, an alarm threshold if responses are averaging /too fast/, based on your expected load & response times. I've not done any deployment/ops beyond hte trivial/theoretical though, so I don't know how this would work in reality.
  - donavanm 14y ago
    No, lbs don't inspect established streams. The lb will periodically send requests to a known uri as a health heck instead. The problem is when the health check uri isn't indicative of availabilty. (Hint: it never is) Nope, don't do this either. Unless you like getting pages because things are working?
    - jeremyjh 14y ago
      You can configure a LB to inspect layer 7 - in HAProxy this is done with an observe directive. Error 500s would then take the member out of a pool. You are right that the health check of a static file may put it right back into the pool, but you can give it a slow rise value so that it waits a few minutes to do that. I'm not saying this is easy to get right but it is definitely possible to at least reduce the frequency of selection of a server raising 500 errors.

bbatsell 14y ago

> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.

CoffeeDregs 14y ago
AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]
- bbatsell 14y ago
  I've seen it frequently, and yes, this is how it behaves, e.g.: [1]: http://stackoverflow.com/a/579466/94830 http://stackoverflow.com/a/579466/94830 [2]: https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 I imagine the rationale was something along the lines of many servers/apps are written to incorrectly return 200 with a descriptive error page rather than 500 or whatever the appropriate status code would be. And at the time ab was first written, pages were a lot more static than they are now, so a different page would be more likely to indicate an incorrect response.
- stock_toaster 14y ago
  use wrk[1] or weighttp[2] instead. [1]: https://github.com/wg/wrk https://github.com/wg/wrk [2]: redmine.lighttpd.net/projects/weighttp/wiki

donavanm 14y ago

> I've never understood using round-robin or random algorithms for load-balancers... LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it. A single host failure just became a major outage. I like WRR selection for back ends, then use a queue or fast fail when your max active conns is exceeded. Humans prefer latency to errors, so let the lb queue (to a limit) on human centric VIPs. Automated clients deal better with errors so have your lb throw a 500 directly, or RST, or direct to a pool that serves static content.

shabble 14y ago
You'd have some sort of error threshold/rate limit[1] at which point the server is marked dead and falls out of the backends list, surely? Or even, an alarm threshold if responses are averaging /too fast/, based on your expected load & response times. I've not done any deployment/ops beyond hte trivial/theoretical though, so I don't know how this would work in reality.
- donavanm 14y ago
  No, lbs don't inspect established streams. The lb will periodically send requests to a known uri as a health heck instead. The problem is when the health check uri isn't indicative of availabilty. (Hint: it never is) Nope, don't do this either. Unless you like getting pages because things are working?
  - jeremyjh 14y ago
    You can configure a LB to inspect layer 7 - in HAProxy this is done with an observe directive. Error 500s would then take the member out of a pool. You are right that the health check of a static file may put it right back into the pool, but you can give it a slow rise value so that it waits a few minutes to do that. I'm not saying this is easy to get right but it is definitely possible to at least reduce the frequency of selection of a server raising 500 errors.