4 ms·

You _hate_ the Benchmarks Game because … ?

by igouy 12y ago

steveklabnik 12y ago
Because it's arbitrary, most of the benchmarks aren't useful or interesting, and it encourages a mentality that I find to be a net negative. Many of the tests boil down to "Does your language have GMP bindings" or "Can you write non-idiomatic code to theoretically make this as fast as something else?" In other words, I don't find it to be useful, except in the absolute broadest sense.
- igouy 12y ago
  "arbitrary" is just name-calling. What about benchmarks game is arbitrary in your opinion? Exactly one task might be described as boil down to "Does your language have GMP bindings" and that shows: 1) differences with the same language implementation and same GMP binding 2) differences with the same language implementation with/without GMP (for example, Rust #2 and Rust #1) 3) differences with different language implementations' use of GMP. None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to! If the Rust programs shown on the benchmarks game are "non-idiomatic" it's because so-far that's been the choice of the Rust community. You know what you mean by "in the absolute broadest sense" but we don't.
  - steveklabnik 12y ago
    > What about benchmarks game is arbitrary in your opinion? For example, look at the history of D being in the game vs. not. Generally speaking, all of http://benchmarksgame.alioth.debian.org/play.html#languagex http://benchmarksgame.alioth.debian.org/play.html#languagex is incredibly arbitrary. I don't think that the maintainers of the game have some sort of moral obligation to support things they don't want to, but it is an arbitrary line which influences what kind of benchmark it is. As another, the domain of the examples, which are largely related to numeric computing and/or memory usage. Which is fine, but isn't going to give me information about other use cases. > None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to! Yes, they do. Because one man is not an island: people use the raw numbers of the Benchmark Game to claim "X is faster than Y", even if the implementation of X and Y are totally different than what their implementation would be. > that's been the choice of the Rust community. I'm actually not thinking of Rust here. I'm thinking more of languages like Haskell, who have implementations that look significantly more like C than like Haskell. But regardless of if the implementations are idiomatic or not, it means that The Benchmark Game is useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another. Because this isn't the code that I'd write: it's code that takes advantage of every little last corner to get as small a number as possible.
    - igouy 12y ago
      >>… being in the game vs. not … but isn't going to give me information about other use cases.<< 1) That might be a credible criticism if the benchmarks game claimed to be some kind of exhaustive comparison. In fact (if it wasn't already completely obvious that the benchmarks game has nothing to say about things it does not show) over & over again the web pages state -- "These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples." 2) You seem to have assumed that because you personally don't know the reasons, there were no reasons. >>…totally different than what their implementation would be.<< What does the benchmarks game have to say about those "totally different" programs that are not shown? Nothing. >>…more like C than like Haskell.<< http://benchmarksgame.alioth.debian.org/u64q/program.php?test=meteor&lang=ghc&id=4#sourcecode http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... Idiomatic Haskell? Which would that be? http://www.willamette.edu/~fruehr/haskell/evolution.html http://www.willamette.edu/~fruehr/haskell/evolution.html Do you think there's an idiomatic style for programs written as though performance matters or does idiomatic just mean programs written as though performance doesn't matter? >>…useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another.<< Does the benchmarks game website claim to be useful for that purpose? It sounds like magical thinking to me -- "… but the question is still asked - Will my program be faster if I write it in language X? - and there's still a wish for a simpler answer than - It depends how you write it!" http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
      - dbaupp 12y ago
        You can state those things all you like, in as large a font as you like, but it doesn't stop people from comparing languages based on the benchmarks game. It's magical thinking to say otherwise: just because something doesn't claim to be X (or claims to not be X) doesn't stop popular opinion from thinking X. Maybe popular opinion is wrong, but it exists and it's what we have to contend with. e: random recent example of someone doing this sort of comparison: https://news.ycombinator.com/item?id=8743698 https://news.ycombinator.com/item?id=8743698

You _hate_ the Benchmarks Game because … ?

by igouy 12y ago

You _hate_ the Benchmarks Game because … ?

Because it's arbitrary, most of the benchmarks aren't useful or interesting, and it encourages a mentality that I find to be a net negative. Many of the tests boil down to "Does your language have GMP bindings" or "Can you write non-idiomatic code to theoretically make this as fast as something else?" In other words, I don't find it to be useful, except in the absolute broadest sense.

igouy 12y ago
"arbitrary" is just name-calling. What about benchmarks game is arbitrary in your opinion? Exactly one task might be described as boil down to "Does your language have GMP bindings" and that shows: 1) differences with the same language implementation and same GMP binding 2) differences with the same language implementation with/without GMP (for example, Rust #2 and Rust #1) 3) differences with different language implementations' use of GMP. None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to! If the Rust programs shown on the benchmarks game are "non-idiomatic" it's because so-far that's been the choice of the Rust community. You know what you mean by "in the absolute broadest sense" but we don't.
- steveklabnik 12y ago
  > What about benchmarks game is arbitrary in your opinion? For example, look at the history of D being in the game vs. not. Generally speaking, all of http://benchmarksgame.alioth.debian.org/play.html#languagex http://benchmarksgame.alioth.debian.org/play.html#languagex is incredibly arbitrary. I don't think that the maintainers of the game have some sort of moral obligation to support things they don't want to, but it is an arbitrary line which influences what kind of benchmark it is. As another, the domain of the examples, which are largely related to numeric computing and/or memory usage. Which is fine, but isn't going to give me information about other use cases. > None of the tasks boil down to "Can you write non-idiomatic code to theoretically make this as fast as something else?" -- if you don't want them to! Yes, they do. Because one man is not an island: people use the raw numbers of the Benchmark Game to claim "X is faster than Y", even if the implementation of X and Y are totally different than what their implementation would be. > that's been the choice of the Rust community. I'm actually not thinking of Rust here. I'm thinking more of languages like Haskell, who have implementations that look significantly more like C than like Haskell. But regardless of if the implementations are idiomatic or not, it means that The Benchmark Game is useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another. Because this isn't the code that I'd write: it's code that takes advantage of every little last corner to get as small a number as possible.
  - igouy 12y ago
    >>… being in the game vs. not … but isn't going to give me information about other use cases.<< 1) That might be a credible criticism if the benchmarks game claimed to be some kind of exhaustive comparison. In fact (if it wasn't already completely obvious that the benchmarks game has nothing to say about things it does not show) over & over again the web pages state -- "These are not the only compilers and interpreters. These are not the only programs that could be written. These are not the only tasks that could be solved. These are just 10 tiny examples." 2) You seem to have assumed that because you personally don't know the reasons, there were no reasons. >>…totally different than what their implementation would be.<< What does the benchmarks game have to say about those "totally different" programs that are not shown? Nothing. >>…more like C than like Haskell.<< http://benchmarksgame.alioth.debian.org/u64q/program.php?test=meteor&lang=ghc&id=4#sourcecode http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... Idiomatic Haskell? Which would that be? http://www.willamette.edu/~fruehr/haskell/evolution.html http://www.willamette.edu/~fruehr/haskell/evolution.html Do you think there's an idiomatic style for programs written as though performance matters or does idiomatic just mean programs written as though performance doesn't matter? >>…useless for evaluating if a program I write in a language is going to generally be faster or not than if I did it in another.<< Does the benchmarks game website claim to be useful for that purpose? It sounds like magical thinking to me -- "… but the question is still asked - Will my program be faster if I write it in language X? - and there's still a wish for a simpler answer than - It depends how you write it!" http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
    - dbaupp 12y ago
      You can state those things all you like, in as large a font as you like, but it doesn't stop people from comparing languages based on the benchmarks game. It's magical thinking to say otherwise: just because something doesn't claim to be X (or claims to not be X) doesn't stop popular opinion from thinking X. Maybe popular opinion is wrong, but it exists and it's what we have to contend with. e: random recent example of someone doing this sort of comparison: https://news.ycombinator.com/item?id=8743698 https://news.ycombinator.com/item?id=8743698