4 ms·

Other languages in those pidigits benchmarks (Perl, Lua, PHP, Racket, Lisp SBCL and probably more) also use the GMP maths library. So there is nothing stopping

Other languages in those pidigits benchmarks (Perl, Lua, PHP, Racket, Lisp SBCL and probably more) also use the GMP maths library. So there is nothing stopping Ruby benchmarks also using it as well (just need to request relevant Ruby GMP library are installed on the Alioth benchmark machines). Anyway you didn't seem too fussed about what the Alioth benchmarks showed in the past - https://news.ycombinator.com/item?id=634738 https://news.ycombinator.com/item?id=634738 :)

petercooper 14y ago
Because someone can't change their opinion in 4 years.. I hadn't looked at the source code back then. Only now when this particular variant of the benchmark shows some rather unrealistic results. Another example (binary trees, I believe) has Ruby using a single process whereas the Python example is using multiple processes to spread the load, so even though Ruby wins on CPU time, Python wins with a slightly lower wall-clock time. And making the Ruby pidigits benchmark use GMP too? That defeats the point of benchmarking one language against another if you can just call out to C libraries for everything (hey, there's an idea.. Ruby could be parallel with C if we just did that). It should be pure language against pure language otherwise it's totally meaningless (and, well, it is, as it stands).
- draegtun 14y ago
  re: single process/multicores The single core benchmarks doesn't change the picture much either - http://benchmarksgame.alioth.debian.org/u32/ruby.php http://benchmarksgame.alioth.debian.org/u32/ruby.php re: GMP / C libraries Generally I agree with you on this however agreeing on a level playing field may not be easy! I assume that some languages, stdlibs or VM's come with GMP (or similar ilk) which perhaps gave them an unfair advantage? And what about languages that come with coroutines? In real world I use Coro/AnyEvent with Perl but these are separate modules and not part of Perl core libraries. So languages like Lua/Python have an (unfair?) advantage here? > It should be pure language against pure language Ideally then it should be same algorithm against algorithm? I recently rewrote the fasta.pl implementation on Alioth using an exact port of the fasta.rb code. Before change Ruby fasta was two times faster than the Perl version. Afterwards Ruby was 30% slower than Perl - https://news.ycombinator.com/item?id=5141179 https://news.ycombinator.com/item?id=5141179
- igouy 14y ago
  What's preventing Ruby 2.0 from "using multiple processes to spread the load"? >>if you can just call out to C libraries for everything<< You can't just call out for everything. There have been language implementations that used GMP under-the-covers, so - to level the playing field - everyone can call out for GMP and PCRE.
  - petercooper 14y ago
    Nothing, it seems. But a benchmark is pointless if you aren't comparing like for like, yet people are clearly taking the results as some sort of serious indicator. As it is, it's like sending two different cars around two different tracks and comparing how long they took.
    - igouy 14y ago
      >>But a benchmark is pointless if you aren't comparing like for like<< According to you it defeats the point when we do compare like to like -- "if you can just call out to C libraries". >>yet people are clearly taking the results as some sort of serious indicator<< Offer them something better. >>sending two different cars around two different tracks<< No. It's like saying here's the track - Ruby guy bring out your best cars, Python guy bring out your best cars - and sending them around the same track. The possible mismatch between programs written for quad-core and those written for single-core is marked. You can mention the difference between CPU time and wall-clock time because both are intentionally shown, along with "CPU Load" so you can see how much difference quad-core has made. And as draegtun informed you, there's another set of measurements that removes that difference by forcing all the programs onto a single core. The problem with the Ruby binary-trees programs is that they run out of memory. http://benchmarksgame.alioth.debian.org/u32q/program.php?test=binarytrees&lang=yarv&id=2 http://benchmarksgame.alioth.debian.org/u32q/program.php?tes...

petercooper 14y ago

Because someone can't change their opinion in 4 years.. I hadn't looked at the source code back then. Only now when this particular variant of the benchmark shows some rather unrealistic results. Another example (binary trees, I believe) has Ruby using a single process whereas the Python example is using multiple processes to spread the load, so even though Ruby wins on CPU time, Python wins with a slightly lower wall-clock time. And making the Ruby pidigits benchmark use GMP too? That defeats the point of benchmarking one language against another if you can just call out to C libraries for everything (hey, there's an idea.. Ruby could be parallel with C if we just did that). It should be pure language against pure language otherwise it's totally meaningless (and, well, it is, as it stands).

draegtun 14y ago
re: single process/multicores The single core benchmarks doesn't change the picture much either - http://benchmarksgame.alioth.debian.org/u32/ruby.php http://benchmarksgame.alioth.debian.org/u32/ruby.php re: GMP / C libraries Generally I agree with you on this however agreeing on a level playing field may not be easy! I assume that some languages, stdlibs or VM's come with GMP (or similar ilk) which perhaps gave them an unfair advantage? And what about languages that come with coroutines? In real world I use Coro/AnyEvent with Perl but these are separate modules and not part of Perl core libraries. So languages like Lua/Python have an (unfair?) advantage here? > It should be pure language against pure language Ideally then it should be same algorithm against algorithm? I recently rewrote the fasta.pl implementation on Alioth using an exact port of the fasta.rb code. Before change Ruby fasta was two times faster than the Perl version. Afterwards Ruby was 30% slower than Perl - https://news.ycombinator.com/item?id=5141179 https://news.ycombinator.com/item?id=5141179
igouy 14y ago
What's preventing Ruby 2.0 from "using multiple processes to spread the load"? >>if you can just call out to C libraries for everything<< You can't just call out for everything. There have been language implementations that used GMP under-the-covers, so - to level the playing field - everyone can call out for GMP and PCRE.
- petercooper 14y ago
  Nothing, it seems. But a benchmark is pointless if you aren't comparing like for like, yet people are clearly taking the results as some sort of serious indicator. As it is, it's like sending two different cars around two different tracks and comparing how long they took.
  - igouy 14y ago
    >>But a benchmark is pointless if you aren't comparing like for like<< According to you it defeats the point when we do compare like to like -- "if you can just call out to C libraries". >>yet people are clearly taking the results as some sort of serious indicator<< Offer them something better. >>sending two different cars around two different tracks<< No. It's like saying here's the track - Ruby guy bring out your best cars, Python guy bring out your best cars - and sending them around the same track. The possible mismatch between programs written for quad-core and those written for single-core is marked. You can mention the difference between CPU time and wall-clock time because both are intentionally shown, along with "CPU Load" so you can see how much difference quad-core has made. And as draegtun informed you, there's another set of measurements that removes that difference by forcing all the programs onto a single core. The problem with the Ruby binary-trees programs is that they run out of memory. http://benchmarksgame.alioth.debian.org/u32q/program.php?test=binarytrees&lang=yarv&id=2 http://benchmarksgame.alioth.debian.org/u32q/program.php?tes...