3 ms·

From what I understand, the point of these benchmarks was to write relatively straightforward idiomatic code in all the laguages. The Julia devs were well aware

From what I understand, the point of these benchmarks was to write relatively straightforward idiomatic code in all the laguages. The Julia devs were well aware that their knowledge of julia would make benchmark hacking straightforward, so they purposefully did not take many optimization opportunities available to them. They did the same with the Fotran and C benchmarks. These things are all pretty subjective though when it comes to 'what is a fair benchmarking methodology'? Another valid approach would be to have experts in each language try to write as optimal an implementation as possible, but this would quickly become a bit of an arms race.

TheRealKing 6y ago
But even "relatively straightforward idiomatic code" is highly opinionated and developer dependent. For example, the array operation `a(:) = b(:) + c(:)` is very well known to be slower (and much uglier) than `a = b + c` in Fortran. Yet, many developers write in the former style and I see instances of such style usage in the Fortran code of Julia benchmark. Also, Fortran is a natively parallel concurrent vectorized language. The for-loops are automatically parallelized with `do concurrent` construct. The fact that the other languages, and perhaps Julia, lack such native capabilities, does not mean such features of Fortran should be excluded in a benchmark. Otherwise, what is really the point of a benchmark?
- ChrisRackauckas 6y ago
  The point of the benchmark is to benchmark serial loops and automatic SIMD of the compilers, so no, parallelizing it doesn't make sense. In fact, Julia has native multithreaded and distributed parallelism (which can utilize MPI for the interconnects). It would be a separate test that would test native distributed parallelism, and that's not a very interesting test though since any language with these same constructs (so Julia, C++, Fortran) will all end up bandwidth limited at the same speed. This is something to give as a homework problem to students though since it's easy to demonstrate and teaches how to use the parallel constructs.

TheRealKing 6y ago

But even "relatively straightforward idiomatic code" is highly opinionated and developer dependent. For example, the array operation `a(:) = b(:) + c(:)` is very well known to be slower (and much uglier) than `a = b + c` in Fortran. Yet, many developers write in the former style and I see instances of such style usage in the Fortran code of Julia benchmark. Also, Fortran is a natively parallel concurrent vectorized language. The for-loops are automatically parallelized with `do concurrent` construct. The fact that the other languages, and perhaps Julia, lack such native capabilities, does not mean such features of Fortran should be excluded in a benchmark. Otherwise, what is really the point of a benchmark?

ChrisRackauckas 6y ago
The point of the benchmark is to benchmark serial loops and automatic SIMD of the compilers, so no, parallelizing it doesn't make sense. In fact, Julia has native multithreaded and distributed parallelism (which can utilize MPI for the interconnects). It would be a separate test that would test native distributed parallelism, and that's not a very interesting test though since any language with these same constructs (so Julia, C++, Fortran) will all end up bandwidth limited at the same speed. This is something to give as a homework problem to students though since it's easy to demonstrate and teaches how to use the parallel constructs.