4 ms·

This study says close to nothing about "Fortran" and "C", or about any of the other languages involved. Sorry, but I disagree here. It's one of the best resour

by gcp 15y ago

This study says close to nothing about "Fortran" and "C", or about any of the other languages involved. Sorry, but I disagree here. It's one of the best resources out there for getting numbers on the relative performance of languages, despite the pitfalls. The implementations of popular languages are well tuned (sometimes extremely so), the used compilers and settings are quite sane, and the benchmark programs are fairly well distributed in functionality. The median is a reasonable choice because it's less sensitive to outliers and was probably chosen for that reason. "Spectacularly poor metric" seems to be an opinion, not a fact. I admit it would have been better to compare Intel C with Intel Fortran. Why wasn't this done? Perhaps some of the submitted programs don't compile correctly with Intel C, or perhaps Intel C doesn't actually perform well on them. Who knows. But to say that this is a "terribly done study" because of that is doing it injustice.

chalst 15y ago
Median is a bad choice, because it completely ignores just how bad the worst performance holes are for a compiler. Discarding outliers is bad methodology in this case.
- igouy 15y ago
  The outliers are not discarded. The outliers are shown both in the chart and in the table (Fortran Intel 25.26). http://shootout.alioth.debian.org/u64q/which-programming-languages-are-fastest.php?calc=chart&ifc=on&gcc=on http://shootout.alioth.debian.org/u64q/which-programming-lan...
  - chalst 15y ago
    Median can be described as what you get when you discard all but the centremost value or two as outliers.
    - gcp 15y ago
      True, but what is centremost still depends on the data you just threw away. So this is pretty lame and meaningless.
      - chalst 15y ago
        What the outliers are is always distribution dependent.
    - igouy 15y ago
      You don't seem to have even looked at the web page. For each programming language implementation, that web page shows 7 descriptive values - not just the median! As I've already said - The outliers are not discarded. The outliers are shown both in the chart and in the table.
      - chalst 15y ago
        "7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it. The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.
        igouy 15y ago
        > that is why I criticise it You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median. That's empty criticism.
        chalst 15y ago
        What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether. I think the distance-weighted estimator would serve the purpose of the ranking better. http://en.wikipedia.org/wiki/Distance-weighted_estimator http://en.wikipedia.org/wiki/Distance-weighted_estimator This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result. I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey. You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?
stephencanon 15y ago
The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, but C gets 3x slower, then your overall workload gets slower, even though the "median performance improvement" is 3x. Obviously the effect is magnified with only 3 data points, but the number of points used in the examples on that page is not so much greater as to eliminate this effect. Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C). What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be). If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.
- xixi77 15y ago
  Mean would be a better choice if a typical real-world application were split approximately equally between each of these tasks. Which is of course patently untrue, particularly so because writing number-crunching code in Fortran does not prevent you from writing your hash-table code in something else (note how the best C binary-trees code is actually calling an external library). Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.
- igouy 15y ago
  > What's really right is to show the complete distribution of speedups and slowdowns The page the OP linked DOES show all the distribution - that's the beauty of box plots.

by gcp 15y ago

chalst 15y ago

Median is a bad choice, because it completely ignores just how bad the worst performance holes are for a compiler. Discarding outliers is bad methodology in this case.

igouy 15y ago
The outliers are not discarded. The outliers are shown both in the chart and in the table (Fortran Intel 25.26). http://shootout.alioth.debian.org/u64q/which-programming-languages-are-fastest.php?calc=chart&ifc=on&gcc=on http://shootout.alioth.debian.org/u64q/which-programming-lan...
- chalst 15y ago
  Median can be described as what you get when you discard all but the centremost value or two as outliers.
  - gcp 15y ago
    True, but what is centremost still depends on the data you just threw away. So this is pretty lame and meaningless.
    - chalst 15y ago
      What the outliers are is always distribution dependent.
  - igouy 15y ago
    You don't seem to have even looked at the web page. For each programming language implementation, that web page shows 7 descriptive values - not just the median! As I've already said - The outliers are not discarded. The outliers are shown both in the chart and in the table.
    - chalst 15y ago
      "7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it. The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.
      - igouy 15y ago
        > that is why I criticise it You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median. That's empty criticism.
        chalst 15y ago
        What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether. I think the distance-weighted estimator would serve the purpose of the ranking better. http://en.wikipedia.org/wiki/Distance-weighted_estimator http://en.wikipedia.org/wiki/Distance-weighted_estimator This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result. I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey. You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?

stephencanon 15y ago

The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, but C gets 3x slower, then your overall workload gets slower, even though the "median performance improvement" is 3x. Obviously the effect is magnified with only 3 data points, but the number of points used in the examples on that page is not so much greater as to eliminate this effect. Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C). What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be). If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.

xixi77 15y ago
Mean would be a better choice if a typical real-world application were split approximately equally between each of these tasks. Which is of course patently untrue, particularly so because writing number-crunching code in Fortran does not prevent you from writing your hash-table code in something else (note how the best C binary-trees code is actually calling an external library). Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.
igouy 15y ago
> What's really right is to show the complete distribution of speedups and slowdowns The page the OP linked DOES show all the distribution - that's the beauty of box plots.