5 ms·

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069 98% 29282 99% 30029 100% 30029 (longest request) Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits? Why in the world would a company spend $20,000 a month for service this awful?

eli 14y ago
Well, at X level of concurrency, wouldn't most set ups with load balancers start to spit numbers like that?
- trotsky 14y ago
  no, at x level of concurrency most set ups wont spend 16 seconds or more on 10% of their requests.
  - eli 14y ago
    You mean because they'll crash before then? Otherwise I don't follow. Surely there's always a limit to how many simultaneous requests can be processed at once.
    - trotsky 14y ago
      Hmm, you're right of course. Somewhere our terms got crossed, I wouldn't call (total requests/servers) concurrency, I'd call that request density.
deleted 14y ago
[deleted]
csense 14y ago
High cost/risk associated with switching providers, and frog-in-heating-water syndrome.
CoffeeDregs 14y ago
Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/100 requests failed? That's not happy-making. Uncertainty is DiaI (death-in-an-infrastructure). I just created a couple of projects on Heroku and love the service, but this needs to be addressed ASAP (even if addressing it is just a blog post). Also, if you have fewest-connections available, I've never understood using round-robin or random algorithms for load-balancers...
- bbatsell 14y ago
  > But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.
  - CoffeeDregs 14y ago
    AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]
    - bbatsell 14y ago
      I've seen it frequently, and yes, this is how it behaves, e.g.: [1]: http://stackoverflow.com/a/579466/94830 http://stackoverflow.com/a/579466/94830 [2]: https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 I imagine the rationale was something along the lines of many servers/apps are written to incorrectly return 200 with a descriptive error page rather than 500 or whatever the appropriate status code would be. And at the time ab was first written, pages were a lot more static than they are now, so a different page would be more likely to indicate an incorrect response.
JPKab 14y ago
I've been scanning through the comments, and I have yet to see anything written by a Heroku engineer to defend the company. I'm hoping its in here and I missed it. I have a feeling that this all might be absolutely true, and they have lawyers/PR trying to think of a damage control plan. I suspect that the reason they'be been pushed to do this is financial, and it makes me think that Nodejitsu's model of simply not providing ANY free plans other than one month trials is a good one. I realize it's apples and oranges, since NJ is focused on async and this wouldn't even be a problem for a Node app, but from a business perspective I feel like this would alleviate pressure. How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what?
- joeblau 14y ago
  Nodejitsu actually does have a free plan if your project is Open Source. http://blog.nodejitsu.com/opensource-dot-jitsu http://blog.nodejitsu.com/opensource-dot-jitsu
- badgar 14y ago
  > How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what? Of course the random routing mesh isn't necessitated by anything, this problem is already solved by bigger companies.

eli 14y ago

Well, at X level of concurrency, wouldn't most set ups with load balancers start to spit numbers like that?

trotsky 14y ago
no, at x level of concurrency most set ups wont spend 16 seconds or more on 10% of their requests.
- eli 14y ago
  You mean because they'll crash before then? Otherwise I don't follow. Surely there's always a limit to how many simultaneous requests can be processed at once.
  - trotsky 14y ago
    Hmm, you're right of course. Somewhere our terms got crossed, I wouldn't call (total requests/servers) concurrency, I'd call that request density.

deleted 14y ago

[deleted]

csense 14y ago

High cost/risk associated with switching providers, and frog-in-heating-water syndrome.

CoffeeDregs 14y ago

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/100 requests failed? That's not happy-making. Uncertainty is DiaI (death-in-an-infrastructure). I just created a couple of projects on Heroku and love the service, but this needs to be addressed ASAP (even if addressing it is just a blog post). Also, if you have fewest-connections available, I've never understood using round-robin or random algorithms for load-balancers...

bbatsell 14y ago
> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.
- CoffeeDregs 14y ago
  AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]
  - bbatsell 14y ago
    I've seen it frequently, and yes, this is how it behaves, e.g.: [1]: http://stackoverflow.com/a/579466/94830 http://stackoverflow.com/a/579466/94830 [2]: https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 https://issues.apache.org/bugzilla/show_bug.cgi?id=42040 I imagine the rationale was something along the lines of many servers/apps are written to incorrectly return 200 with a descriptive error page rather than 500 or whatever the appropriate status code would be. And at the time ab was first written, pages were a lot more static than they are now, so a different page would be more likely to indicate an incorrect response.

JPKab 14y ago

I've been scanning through the comments, and I have yet to see anything written by a Heroku engineer to defend the company. I'm hoping its in here and I missed it. I have a feeling that this all might be absolutely true, and they have lawyers/PR trying to think of a damage control plan. I suspect that the reason they'be been pushed to do this is financial, and it makes me think that Nodejitsu's model of simply not providing ANY free plans other than one month trials is a good one. I realize it's apples and oranges, since NJ is focused on async and this wouldn't even be a problem for a Node app, but from a business perspective I feel like this would alleviate pressure. How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what?

joeblau 14y ago
Nodejitsu actually does have a free plan if your project is Open Source. http://blog.nodejitsu.com/opensource-dot-jitsu http://blog.nodejitsu.com/opensource-dot-jitsu
badgar 14y ago
> How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what? Of course the random routing mesh isn't necessitated by anything, this problem is already solved by bigger companies.