4 ms·

Means and (especially) medians mean little to nothing when you're doing them over entirely different benchmarks. Why bother? A 3D plot would be a lot clearer.

by leif 16y ago

acangiano 16y ago
> Means and (especially) medians mean little to nothing It's the other way around. Given the skewed nature of the data, the median is a better measure of the central tendency of the data set. But note that I'm not plotting the median only. I use a box plot which gives you a much better statistical picture than using a simple median (or mean). Nevertheless, summarizing different benchmarks has its limitations and box plots give you a rough, general idea of what's going on.
- bodhi 16y ago
  Did you normalise the results before plotting them and calculating means and medians? Or is it a summary of the raw data? I could probably find out myself, but since you are posting here, I thought I'd ask.
- leif 16y ago
  I disagree. If you write ten benchmarks, and they finish in the same order on all platforms, the median really doesn't give you any information apart from how the middle one did. At least the mean will be changed if the slowest benchmark for some reason is a lot slower on one platform. In this case, there was some variation, but for the most part it looks like the median is just one of 4 or 5 middling benchmarks in all cases, so you're really just removing data, if anything. The box plot is a good choice here. What I would like to see is a graph plotting time against {VM} x {benchmark size} to see how each platform scales within a benchmark. With the data you've got, apart from confusing me a little by including the mean and median, you did a good job presenting it. I'd love to see why some platforms do so much well than others in some cases (except JRuby which everyone knows is a memory hog), but this seems way out of scope for this article. :)

Means and (especially) medians mean little to nothing when you're doing them over entirely different benchmarks. Why bother? A 3D plot would be a lot clearer.

by leif 16y ago

Means and (especially) medians mean little to nothing when you're doing them over entirely different benchmarks. Why bother? A 3D plot would be a lot clearer.

acangiano 16y ago

> Means and (especially) medians mean little to nothing It's the other way around. Given the skewed nature of the data, the median is a better measure of the central tendency of the data set. But note that I'm not plotting the median only. I use a box plot which gives you a much better statistical picture than using a simple median (or mean). Nevertheless, summarizing different benchmarks has its limitations and box plots give you a rough, general idea of what's going on.

bodhi 16y ago
Did you normalise the results before plotting them and calculating means and medians? Or is it a summary of the raw data? I could probably find out myself, but since you are posting here, I thought I'd ask.
leif 16y ago
I disagree. If you write ten benchmarks, and they finish in the same order on all platforms, the median really doesn't give you any information apart from how the middle one did. At least the mean will be changed if the slowest benchmark for some reason is a lot slower on one platform. In this case, there was some variation, but for the most part it looks like the median is just one of 4 or 5 middling benchmarks in all cases, so you're really just removing data, if anything. The box plot is a good choice here. What I would like to see is a graph plotting time against {VM} x {benchmark size} to see how each platform scales within a benchmark. With the data you've got, apart from confusing me a little by including the mean and median, you did a good job presenting it. I'd love to see why some platforms do so much well than others in some cases (except JRuby which everyone knows is a memory hog), but this seems way out of scope for this article. :)