7 ms·

Are you saying that the benchmarks game programs you wrote "cheat" ?

by igouy 11y ago

kibwen 11y ago
He's saying that all the fastest benchmarks game programs in all languages cheat, relative to the slowest ones. :P This isn't an attack on your capacity as the site's maintainer, this is an entirely expected outcome due to the high-profile nature of the site. He's saying that random tests on the internet are "better" because they attract less attention and thus have less exposure. It's a problem with microbenchmarks in general.
- igouy 11y ago
  >>He's saying that...<< Please let Veedrac speak for himself. Veedrac's saying a bunch of stuff -- saying it doesn't make trash-talk into The Truth(TM). >>fastest benchmarks game programs cheat, relative to the slowest ones<< So programs that are compiled cheat, relative to programs that are interpreted? Seems like "cheat" needs to be in scare quotes.
  - kibwen 11y ago
    Your defensiveness is unwarranted and your interpretation assumes bad faith. > So programs that are compiled cheat, relative to programs > that are interpreted? Programming languages whose dominant implementations are interpreted generally (though don't necessarily need to) provide less direct control over hardware, memory layout, and algorithms, and in practice will defer to C via FFI when such control is necessary. Because the benchmarks game generally prohibits calling out to C, it means that many "tricks" employed by low-level languages are unavailable to the higher-level languages. Whether or not this is a desirable property is up to the reader's interpretation. > Seems like "cheat" needs to be in scare quotes. Indeed, because it's essentially impossible to enforce anything like "idiomatic" code in any of the benchmarks game languages, and so whether a program is "cheating" is up to the reader's interpretation. That said, the fact that it is up to the reader's interpretation means that it is valid for some readers to conclude that some programs are "cheating", while other programmers may differ. Personally, I think the point is moot, and that microbenchmarks aren't worth getting so worked up over. :P
    - igouy 11y ago
      >>... is valid for some readers to conclude that some programs are "cheating", while other programmers may differ.<< Which takes us back to the unanswered question -- Veedrac, are you saying that the benchmarks game programs you wrote "cheat" ?
      - kibwen 11y ago
        I think you've overlooked this part of his comment: "I say this as someone who spent a fair while optimizing the Rust benchmarks to use all of the hacks the other languages use" Do you think the C, C++, Haskell, etc. implementations "cheat"? :P
        igouy 11y ago
        "cheat"? You seem to think programs not written to your notion of "idiomatic" code "cheat". I think that's just name-calling.
Veedrac 11y ago
Consider the fasta benchmark. It would be completely disingenuous to suggest that Haskell is on top for any reason other than it doing a different thing to the other benchmarks. Haskell is just not that fast. So, in a sense, it cheats. Of course, I copied these techniques in my Rust implementation. So, if Haskell cheated, I cheated. The same can be said for almost all other benchmarks - but not normally quite to this extent. But it's not cheating in a literal sense, since (AFAIK) the Haskell program has been accepted on the basis that it follows the rules. So it doesn't cheat in that it follows the rules, but it cheats in that it should be breaking the rules. I can be more specific if needed.
- codygman 11y ago
  > I can be more specific if needed. Please do! Can you give some specific examples, preferably in order of those with the most effect on performance.
  - Veedrac 11y ago
    Since I'm lazy, I'll point out that most of the stuff is indirectly mentioned in this write-up: https://llogiq.github.io/2015/10/03/fast.html https://llogiq.github.io/2015/10/03/fast.html This reminds me I haven't written up stuff for chameneos-redux, which is probably the most fun and over-engineered of them all. The fasta/Haskell thing I was mentioning are the parts at "The Haskell code does a quite clever alternative" and "the Haskell code just caches them all in an array". The pre-compacting I mention in k_nucleotide was already done in the Scala implementation. How parallelism is added also varies. The Scala code does each of the test cases in parallel whilst the Rust parallelizes each internally. (My new Rust code parallelizes even better.) Several implementation make their own specialized hash table, and there isn't one particular hash table to implement. This is very important since hash table lookups take a substantial fraction of the time. Note that I pick on Haskell and Scala here only because they have the best hacks, not because they're the only ones doing hacks. (I'd imagine listing all of the differences would take much longer than I have time for.) thread-ring has lots of different implementations. Some are just scheduled coroutines restricted to a single process, then you have stuff like [1] which uses mutexes, real threads and sets thread affinity. I'm not totally sure what [2] is but I think it's something else entirely. I'm not being too specific here because honestly this benchmark confuses me. [1] http://benchmarksgame.alioth.debian.org/u64q/program.php?test=threadring&lang=gcc&id=3 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... [2] http://benchmarksgame.alioth.debian.org/u64/program.php?test=threadring&lang=gpp&id=5 http://benchmarksgame.alioth.debian.org/u64/program.php?test...

Are you saying that the benchmarks game programs you wrote "cheat" ?

by igouy 11y ago

Are you saying that the benchmarks game programs you wrote "cheat" ?

He's saying that all the fastest benchmarks game programs in all languages cheat, relative to the slowest ones. :P This isn't an attack on your capacity as the site's maintainer, this is an entirely expected outcome due to the high-profile nature of the site. He's saying that random tests on the internet are "better" because they attract less attention and thus have less exposure. It's a problem with microbenchmarks in general.

igouy 11y ago
>>He's saying that...<< Please let Veedrac speak for himself. Veedrac's saying a bunch of stuff -- saying it doesn't make trash-talk into The Truth(TM). >>fastest benchmarks game programs cheat, relative to the slowest ones<< So programs that are compiled cheat, relative to programs that are interpreted? Seems like "cheat" needs to be in scare quotes.
- kibwen 11y ago
  Your defensiveness is unwarranted and your interpretation assumes bad faith. > So programs that are compiled cheat, relative to programs > that are interpreted? Programming languages whose dominant implementations are interpreted generally (though don't necessarily need to) provide less direct control over hardware, memory layout, and algorithms, and in practice will defer to C via FFI when such control is necessary. Because the benchmarks game generally prohibits calling out to C, it means that many "tricks" employed by low-level languages are unavailable to the higher-level languages. Whether or not this is a desirable property is up to the reader's interpretation. > Seems like "cheat" needs to be in scare quotes. Indeed, because it's essentially impossible to enforce anything like "idiomatic" code in any of the benchmarks game languages, and so whether a program is "cheating" is up to the reader's interpretation. That said, the fact that it is up to the reader's interpretation means that it is valid for some readers to conclude that some programs are "cheating", while other programmers may differ. Personally, I think the point is moot, and that microbenchmarks aren't worth getting so worked up over. :P
  - igouy 11y ago
    >>... is valid for some readers to conclude that some programs are "cheating", while other programmers may differ.<< Which takes us back to the unanswered question -- Veedrac, are you saying that the benchmarks game programs you wrote "cheat" ?
    - kibwen 11y ago
      I think you've overlooked this part of his comment: "I say this as someone who spent a fair while optimizing the Rust benchmarks to use all of the hacks the other languages use" Do you think the C, C++, Haskell, etc. implementations "cheat"? :P
      - igouy 11y ago
        "cheat"? You seem to think programs not written to your notion of "idiomatic" code "cheat". I think that's just name-calling.

Veedrac 11y ago

Consider the fasta benchmark. It would be completely disingenuous to suggest that Haskell is on top for any reason other than it doing a different thing to the other benchmarks. Haskell is just not that fast. So, in a sense, it cheats. Of course, I copied these techniques in my Rust implementation. So, if Haskell cheated, I cheated. The same can be said for almost all other benchmarks - but not normally quite to this extent. But it's not cheating in a literal sense, since (AFAIK) the Haskell program has been accepted on the basis that it follows the rules. So it doesn't cheat in that it follows the rules, but it cheats in that it should be breaking the rules. I can be more specific if needed.

codygman 11y ago
> I can be more specific if needed. Please do! Can you give some specific examples, preferably in order of those with the most effect on performance.
- Veedrac 11y ago
  Since I'm lazy, I'll point out that most of the stuff is indirectly mentioned in this write-up: https://llogiq.github.io/2015/10/03/fast.html https://llogiq.github.io/2015/10/03/fast.html This reminds me I haven't written up stuff for chameneos-redux, which is probably the most fun and over-engineered of them all. The fasta/Haskell thing I was mentioning are the parts at "The Haskell code does a quite clever alternative" and "the Haskell code just caches them all in an array". The pre-compacting I mention in k_nucleotide was already done in the Scala implementation. How parallelism is added also varies. The Scala code does each of the test cases in parallel whilst the Rust parallelizes each internally. (My new Rust code parallelizes even better.) Several implementation make their own specialized hash table, and there isn't one particular hash table to implement. This is very important since hash table lookups take a substantial fraction of the time. Note that I pick on Haskell and Scala here only because they have the best hacks, not because they're the only ones doing hacks. (I'd imagine listing all of the differences would take much longer than I have time for.) thread-ring has lots of different implementations. Some are just scheduled coroutines restricted to a single process, then you have stuff like [1] which uses mutexes, real threads and sets thread affinity. I'm not totally sure what [2] is but I think it's something else entirely. I'm not being too specific here because honestly this benchmark confuses me. [1] http://benchmarksgame.alioth.debian.org/u64q/program.php?test=threadring&lang=gcc&id=3 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes... [2] http://benchmarksgame.alioth.debian.org/u64/program.php?test=threadring&lang=gpp&id=5 http://benchmarksgame.alioth.debian.org/u64/program.php?test...