The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, bu

The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, but C gets 3x slower, then your overall workload gets slower, even though the "median performance improvement" is 3x. Obviously the effect is magnified with only 3 data points, but the number of points used in the examples on that page is not so much greater as to eliminate this effect. Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C). What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be). If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.