4 ms·

I am curious why every time there is a "language" beats "other language" in some test posted on here, there is an inevitable slew of comments along the lines of

I am curious why every time there is a "language" beats "other language" in some test posted on here, there is an inevitable slew of comments along the lines of "losing language was poorly written code". Is the probability really so high as to be 100% that all optimization tests were against a poorly written competitor? That's amazing if that's the case.

taeric 14y ago
I would wager a large contributor to this is that the idioms of some of these specialized fields do not generalize to other use cases. So, while some researchers have gotten it such that a relatively straight forward implementation in a new language with new tools is better than the old way, the folks working the old way have not exactly stagnated. This seems especially likely when you consider that for many of the really fast implementations, it is not that they are using no abstractions. The abstractions they are using are very close to the machine, because the machine is the abstraction they are using. (That make sense?)
moomin 14y ago
It seems to me that most X beats Y posts are written by people with a great deal of knowledge of X, but without a matching knowledge of Y. This, unfortunately, leaves every post wide open to this sort of attack.
jules 14y ago
I suspect this comment is at least partially directed at me. Did you read the paper? Excluding some O(1) operations, the goal of the second benchmark is this: double s = 0; for(int i=0; i<n; i++) s += pow(a[i]-b[i],2); return s; But they did not use this C code. The C code they used made 3 calls to the BLAS library. The end result is that the C code is doing something equivalent to this: double x=0, y=0, z=0; for(int i=0; i<n; i++) x += a[i]*a[i]; for(int i=0; i<n; i++) y += a[i]*b[i]; for(int i=0; i<n; i++) z += b[i]*b[i]; return x - 2*y + z; While this does return the same result (assuming infinite precision arithmetic) it is obviously not the way anybody would do it, since it's doing 3 times the work. Even worse, because their code is calling into the BLAS library for each loop, the compiler is explicitly prevented from optimizing the three loops and memory accesses by combining them into one. Note that the Haskell code is doing the efficient single loop. So yes, it is poorly written code.
- ky3 14y ago
  There are 2 pieces of C benchmarked. The other one computes `a[i]-b[i]` into a temporary array and then makes a single call to BLAS. The speed is roughly half that of Haskell, which dispenses with the space cost of the temporary array.
  - jules 14y ago
    Yes, you're right, the other C code is even less efficient.
    - ky3 14y ago
      Good catch! It's the 3x BLAS that's 1/2 of Haskell's speed. The 1x BLAS that uses 2x RAM gets shafted, presumably by cache lossage.
raylu 14y ago
Well, yes. Nobody is going to write a post titled "C beats Ruby" with reasonable implementations of both. If you make a post in the other direction, it's often the case that you did something horribly wrong.
Paradigma11 14y ago
Selection bias. The problem is you only hear about surprising results. Nobody writes about a benchmark where c++ beats python. Now the reason for such a surprising result may very often be poorly written code in the contender language.
army 14y ago
Because benchmarks are really hard to do fairly. There's a great deal of subjectivity in what is a "fair comparison" regardless, since there's always a bit more tuning you can do. That said, a lot of benchmarks are just completely bogus. It's surprisingly common to see someone claim that they're beating the "state of the art" when they're comparing against a weak baseline. One problem is that people don't understand the systems they're benchmarking sufficiently well to obtain good results. In this case, I have no doubt the results are much better than they've obtained in Haskell before - I think they've done an impressive job and I don't doubt the value of the work. But if the question is whether rewriting your C code in Haskell would make it go faster... well... I wouldn't bet the house on it.

taeric 14y ago

I would wager a large contributor to this is that the idioms of some of these specialized fields do not generalize to other use cases. So, while some researchers have gotten it such that a relatively straight forward implementation in a new language with new tools is better than the old way, the folks working the old way have not exactly stagnated. This seems especially likely when you consider that for many of the really fast implementations, it is not that they are using no abstractions. The abstractions they are using are very close to the machine, because the machine is the abstraction they are using. (That make sense?)

moomin 14y ago

It seems to me that most X beats Y posts are written by people with a great deal of knowledge of X, but without a matching knowledge of Y. This, unfortunately, leaves every post wide open to this sort of attack.

jules 14y ago

I suspect this comment is at least partially directed at me. Did you read the paper? Excluding some O(1) operations, the goal of the second benchmark is this: double s = 0; for(int i=0; i<n; i++) s += pow(a[i]-b[i],2); return s; But they did not use this C code. The C code they used made 3 calls to the BLAS library. The end result is that the C code is doing something equivalent to this: double x=0, y=0, z=0; for(int i=0; i<n; i++) x += a[i]*a[i]; for(int i=0; i<n; i++) y += a[i]*b[i]; for(int i=0; i<n; i++) z += b[i]*b[i]; return x - 2*y + z; While this does return the same result (assuming infinite precision arithmetic) it is obviously not the way anybody would do it, since it's doing 3 times the work. Even worse, because their code is calling into the BLAS library for each loop, the compiler is explicitly prevented from optimizing the three loops and memory accesses by combining them into one. Note that the Haskell code is doing the efficient single loop. So yes, it is poorly written code.

ky3 14y ago
There are 2 pieces of C benchmarked. The other one computes `a[i]-b[i]` into a temporary array and then makes a single call to BLAS. The speed is roughly half that of Haskell, which dispenses with the space cost of the temporary array.
- jules 14y ago
  Yes, you're right, the other C code is even less efficient.
  - ky3 14y ago
    Good catch! It's the 3x BLAS that's 1/2 of Haskell's speed. The 1x BLAS that uses 2x RAM gets shafted, presumably by cache lossage.

raylu 14y ago

Well, yes. Nobody is going to write a post titled "C beats Ruby" with reasonable implementations of both. If you make a post in the other direction, it's often the case that you did something horribly wrong.

Paradigma11 14y ago

Selection bias. The problem is you only hear about surprising results. Nobody writes about a benchmark where c++ beats python. Now the reason for such a surprising result may very often be poorly written code in the contender language.

army 14y ago

Because benchmarks are really hard to do fairly. There's a great deal of subjectivity in what is a "fair comparison" regardless, since there's always a bit more tuning you can do. That said, a lot of benchmarks are just completely bogus. It's surprisingly common to see someone claim that they're beating the "state of the art" when they're comparing against a weak baseline. One problem is that people don't understand the systems they're benchmarking sufficiently well to obtain good results. In this case, I have no doubt the results are much better than they've obtained in Haskell before - I think they've done an impressive job and I don't doubt the value of the work. But if the question is whether rewriting your C code in Haskell would make it go faster... well... I wouldn't bet the house on it.