4 ms·

The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring l

The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How not to measure latency" [1] for more info. Also be sure you are not falling into the "Coordinated Omission" trap where you end up measureing the latency wrong. [1] http://www.infoq.com/presentations/latency-pitfalls http://www.infoq.com/presentations/latency-pitfalls

bhauer 13y ago
Hello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide that in some form. And perhaps the author of Wrk could add that in time. However, for the time being, I consider the matter somewhat academic. Not to be dismissive--I value your opinion--but I don't believe that would measurably impact my assessment of each framework's performance. Though, it would be fascinating to be able to validate my suspicion that Onion, being written in C, does not suffer even the tiny garbage collection pauses of the Java frameworks. [1] https://github.com/wg/wrk https://github.com/wg/wrk
- diroussel 13y ago
  Ok, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31 https://github.com/wg/wrk/issues/31 Perhaps you could upvote or something? Thanks for all the great work in these benchmarks. A useful resource.
tlarkworthy 13y ago
"average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.
- mynegation 13y ago
  Probably the statement should be that comparing mean and variance are only relevant if both metrics follow the same distribution. In the absence of distribution information (and it is usually absent in empirical tests like that) quantiles would help to do a better job at comparing performance.
  - diroussel 13y ago
    Indeed. Often latency measurements are clumpy. You might have one clump of fast responses when no GC occurs, another when some GC occurs, and a smaller clump where a stop-the-world full GC has occurred. In such a case average is not meaningful.

bhauer 13y ago

Hello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide that in some form. And perhaps the author of Wrk could add that in time. However, for the time being, I consider the matter somewhat academic. Not to be dismissive--I value your opinion--but I don't believe that would measurably impact my assessment of each framework's performance. Though, it would be fascinating to be able to validate my suspicion that Onion, being written in C, does not suffer even the tiny garbage collection pauses of the Java frameworks. [1] https://github.com/wg/wrk https://github.com/wg/wrk

diroussel 13y ago
Ok, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31 https://github.com/wg/wrk/issues/31 Perhaps you could upvote or something? Thanks for all the great work in these benchmarks. A useful resource.

tlarkworthy 13y ago

"average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.

mynegation 13y ago
Probably the statement should be that comparing mean and variance are only relevant if both metrics follow the same distribution. In the absence of distribution information (and it is usually absent in empirical tests like that) quantiles would help to do a better job at comparing performance.
- diroussel 13y ago
  Indeed. Often latency measurements are clumpy. You might have one clump of fast responses when no GC occurs, another when some GC occurs, and a smaller clump where a stop-the-world full GC has occurred. In such a case average is not meaningful.