3 ms·

Have people actually looked at the code? You can’t treat the benchmarks game code as a simple black box. Most implementations look like C regardless of the lang

by bufo 6y ago

Have people actually looked at the code? You can’t treat the benchmarks game code as a simple black box. Most implementations look like C regardless of the language, eg. https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-ghc-3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... or https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-ghc-2.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... There is very little to infer from code which is so far from being idiomatic, aside from the fact that you might as well use C if you ever find yourself in the benchmark situation.

faccacta 6y ago
Some other examples of what you are talking about: In the regular expression benchmarks, some entries implement their own idiomatic regex parsers or link to the PCRE2 library, rather than use the regex library that comes with the language implementation: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-python3-2.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/..., https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-8.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... This is, arguably, totally fine, because these are still valid programs that run (and run quickly). BUT, it makes the benchmark programs poor choices to compare the verbosity of languages. So statements like "For a language famed for its terseness, Haskell it turns out, isn’t as terse as expected" can't be supported when comparing benchmark programs that were written to maximize speed, rather than written to minimize developer time. Fortunately, the Benchmark Game does publish all of its programs, including the ones that don't "win" the speed race, and it's possible to find nice, concise, idiomatic Haskell programs in there.
- igouy 6y ago
  > BUT, it makes the benchmark programs poor choices to compare the verbosity of languages. Because? Because the way some link to an external library is more verbose than the way they link to an included library?
  - faccacta 6y ago
    Yes, that is what I mean. By the way, on the benchmark game website, is it still possible to sort benchmark results not by speed, but by gzip'd source code size?
    - igouy 6y ago
      On the task pages, click the column header: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
igouy 6y ago
> You can’t treat the benchmarks game code as a simple black box. Well you can and obviously people do, but the advice given is "Always look at the source code." > Most implementations look like… So you counted ! How many of the 33 Haskell programs "look like C" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/ghc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Or did you see a couple that "look like C" and cherry-pick ? :-)
- bufo 6y ago
  The actual fastest Haskell programs are here: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/haskell.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... These are the ones which are used for benchmark results. The only fairly idiomatic ones (no pointers, no malloc, no unsafe) I could find are pidigits, binary-trees, and k-nucleotide. That's 3 out of 10.
  - igouy 6y ago
    So you did not mean "Most implementations look like C …" you meant the fastest implementations look like C ? You might find this interesting — "One can, with sufficient effort, essentially write C code in Haskell using various unsafe primitives. We would argue that this is not true to the spirit and goals of Haskell, and we have attempted in this paper to remain within the space of reasonably idiomatic Haskell. However, we have made abundant use of strictness annotations, explicit strictness, and unboxed vectors. We have, more controversially perhaps, used unsafe array subscripting in places. Are our choices reasonable?" pdf "Measuring the Haskell Gap" http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf http://www.leafpetersen.com/leaf/publications/ifl2013/haskel...
    - bufo 6y ago
      I think you're misunderstanding the point of my comment. Keep in mind the point of the original article: "For a language famed for its terseness, Haskell it turns out, isn’t as terse as expected - it’s average size of source code is larger than the average Go source code size." I'm responding to this. Languages with many escape hatches like Haskell or Swift will result in less idiomatic code (and potentially longer programs) when optimized for the benchmarks games. Languages with less escape hatches like Go and Ocaml less so. I don't have anything against any of these languages. Feel free to write C-like code in Haskell if you want.

by bufo 6y ago

faccacta 6y ago

Some other examples of what you are talking about: In the regular expression benchmarks, some entries implement their own idiomatic regex parsers or link to the PCRE2 library, rather than use the regex library that comes with the language implementation: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-python3-2.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/..., https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/regexredux-csharpcore-8.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... This is, arguably, totally fine, because these are still valid programs that run (and run quickly). BUT, it makes the benchmark programs poor choices to compare the verbosity of languages. So statements like "For a language famed for its terseness, Haskell it turns out, isn’t as terse as expected" can't be supported when comparing benchmark programs that were written to maximize speed, rather than written to minimize developer time. Fortunately, the Benchmark Game does publish all of its programs, including the ones that don't "win" the speed race, and it's possible to find nice, concise, idiomatic Haskell programs in there.

igouy 6y ago
> BUT, it makes the benchmark programs poor choices to compare the verbosity of languages. Because? Because the way some link to an external library is more verbose than the way they link to an included library?
- faccacta 6y ago
  Yes, that is what I mean. By the way, on the benchmark game website, is it still possible to sort benchmark results not by speed, but by gzip'd source code size?
  - igouy 6y ago
    On the task pages, click the column header: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux-gz.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

igouy 6y ago

> You can’t treat the benchmarks game code as a simple black box. Well you can and obviously people do, but the advice given is "Always look at the source code." > Most implementations look like… So you counted ! How many of the 33 Haskell programs "look like C" ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/ghc.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Or did you see a couple that "look like C" and cherry-pick ? :-)

bufo 6y ago
The actual fastest Haskell programs are here: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/haskell.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... These are the ones which are used for benchmark results. The only fairly idiomatic ones (no pointers, no malloc, no unsafe) I could find are pidigits, binary-trees, and k-nucleotide. That's 3 out of 10.
- igouy 6y ago
  So you did not mean "Most implementations look like C …" you meant the fastest implementations look like C ? You might find this interesting — "One can, with sufficient effort, essentially write C code in Haskell using various unsafe primitives. We would argue that this is not true to the spirit and goals of Haskell, and we have attempted in this paper to remain within the space of reasonably idiomatic Haskell. However, we have made abundant use of strictness annotations, explicit strictness, and unboxed vectors. We have, more controversially perhaps, used unsafe array subscripting in places. Are our choices reasonable?" pdf "Measuring the Haskell Gap" http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf http://www.leafpetersen.com/leaf/publications/ifl2013/haskel...
  - bufo 6y ago
    I think you're misunderstanding the point of my comment. Keep in mind the point of the original article: "For a language famed for its terseness, Haskell it turns out, isn’t as terse as expected - it’s average size of source code is larger than the average Go source code size." I'm responding to this. Languages with many escape hatches like Haskell or Swift will result in less idiomatic code (and potentially longer programs) when optimized for the benchmarks games. Languages with less escape hatches like Go and Ocaml less so. I don't have anything against any of these languages. Feel free to write C-like code in Haskell if you want.