7 ms·

I wouldn't trust any of the benchmarks in the Benchmarks Game. I say this as someone who spent a fair while optimizing the Rust benchmarks to use all of the hac

by Veedrac 11y ago

I wouldn't trust any of the benchmarks in the Benchmarks Game. I say this as someone who spent a fair while optimizing the Rust benchmarks to use all of the hacks the other languages use in some[1]. _Very_ few of the benchmarks are actually fair, and those that are normally don't matter. You're generally better off with random internet tests than the Benchmarks Game, since they at least don't have the kind of cheating-but-not-cheating that ruins the Benchmarks Game. That said, it's totally possible to do assembly-level tuning in Rust and, sans-SIMD, there's no real difficulty with getting Rust to go fast on these workflows. [1] https://alioth.debian.org/tracker/?func=browse&group_id=100815&atid=413122; https://alioth.debian.org/tracker/?func=browse&group_id=1008... See ones with "Joshua Landau" in title. All (but thread_ring, which is literally the silliest benchmark I've ever seen) beat C++ (sometimes by a lot). The disclaimer would be that none of my submissions have been submitted (or rejected) yet.

igouy 11y ago
>> cheating-but-not-cheating << Make your mind-up! Is it cheating or not cheating?
- jakevn 11y ago
  It's not considered cheating for the Language Benchmarks Game, seeing as it's just a game, but it would probably be considered cheating for a benchmark attempting to compare performance for "realistic" programs.
  - igouy 11y ago
    >>it's just a game<< http://benchmarksgame.alioth.debian.org/sometimes-people-just-make-up-stuff.html#name-game http://benchmarksgame.alioth.debian.org/sometimes-people-jus... >>"realistic" programs<< For example? "The performance of a benchmark, even if it is derived from a real program, may not help to predict the performance of similar programs that have different hot spots." http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#not-prophesy http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
    - wpietri 11y ago
      This sort of querulous point-sniping mainly persuades me that the criticisms are entirely valid. So if you're looking to make people think that the Benchmarks Game has some real-world utility, you might want to try a different approach.
      - deleted 11y ago
        [deleted]
      - igouy 11y ago
        To be persuaded on such grounds is unfortunate, when relevant information is before you.
        wpietri 11y ago
        I'll give you points for consistency. But enough relevant information is already before me. With tens of millions of websites, I have no obligation to look carefully at each one to decide it's not for me.
        igouy 11y ago
        You have no obligation to persuade yourself "that the criticisms are entirely valid" without seeing what's being discussed.
        wpietri 11y ago
        Yes, exactly. In the same way that if I see bad Yelp reviews and the business owner responds poorly, I think there's no need to go by for a sandwich just to be sure.
        igouy 11y ago
        Is it pointing to the FAQ that you consider "responds poorly", or is it pointing to Veedrac's refusal to be unambiguous?
igouy 11y ago
Are you saying that the benchmarks game programs you wrote "cheat" ?
- kibwen 11y ago
  He's saying that all the fastest benchmarks game programs in all languages cheat, relative to the slowest ones. :P This isn't an attack on your capacity as the site's maintainer, this is an entirely expected outcome due to the high-profile nature of the site. He's saying that random tests on the internet are "better" because they attract less attention and thus have less exposure. It's a problem with microbenchmarks in general.
  - igouy 11y ago
    >>He's saying that...<< Please let Veedrac speak for himself. Veedrac's saying a bunch of stuff -- saying it doesn't make trash-talk into The Truth(TM). >>fastest benchmarks game programs cheat, relative to the slowest ones<< So programs that are compiled cheat, relative to programs that are interpreted? Seems like "cheat" needs to be in scare quotes.
    - kibwen 11y ago
      Your defensiveness is unwarranted and your interpretation assumes bad faith. > So programs that are compiled cheat, relative to programs > that are interpreted? Programming languages whose dominant implementations are interpreted generally (though don't necessarily need to) provide less direct control over hardware, memory layout, and algorithms, and in practice will defer to C via FFI when such control is necessary. Because the benchmarks game generally prohibits calling out to C, it means that many "tricks" employed by low-level languages are unavailable to the higher-level languages. Whether or not this is a desirable property is up to the reader's interpretation. > Seems like "cheat" needs to be in scare quotes. Indeed, because it's essentially impossible to enforce anything like "idiomatic" code in any of the benchmarks game languages, and so whether a program is "cheating" is up to the reader's interpretation. That said, the fact that it is up to the reader's interpretation means that it is valid for some readers to conclude that some programs are "cheating", while other programmers may differ. Personally, I think the point is moot, and that microbenchmarks aren't worth getting so worked up over. :P

by Veedrac 11y ago

igouy 11y ago

>> cheating-but-not-cheating << Make your mind-up! Is it cheating or not cheating?

jakevn 11y ago
It's not considered cheating for the Language Benchmarks Game, seeing as it's just a game, but it would probably be considered cheating for a benchmark attempting to compare performance for "realistic" programs.
- igouy 11y ago
  >>it's just a game<< http://benchmarksgame.alioth.debian.org/sometimes-people-just-make-up-stuff.html#name-game http://benchmarksgame.alioth.debian.org/sometimes-people-jus... >>"realistic" programs<< For example? "The performance of a benchmark, even if it is derived from a real program, may not help to predict the performance of similar programs that have different hot spots." http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#not-prophesy http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
  - wpietri 11y ago
    This sort of querulous point-sniping mainly persuades me that the criticisms are entirely valid. So if you're looking to make people think that the Benchmarks Game has some real-world utility, you might want to try a different approach.
    - deleted 11y ago
      [deleted]
    - igouy 11y ago
      To be persuaded on such grounds is unfortunate, when relevant information is before you.
      - wpietri 11y ago
        I'll give you points for consistency. But enough relevant information is already before me. With tens of millions of websites, I have no obligation to look carefully at each one to decide it's not for me.
        igouy 11y ago
        You have no obligation to persuade yourself "that the criticisms are entirely valid" without seeing what's being discussed.
        wpietri 11y ago
        Yes, exactly. In the same way that if I see bad Yelp reviews and the business owner responds poorly, I think there's no need to go by for a sandwich just to be sure.
        igouy 11y ago
        Is it pointing to the FAQ that you consider "responds poorly", or is it pointing to Veedrac's refusal to be unambiguous?

igouy 11y ago

Are you saying that the benchmarks game programs you wrote "cheat" ?

kibwen 11y ago
He's saying that all the fastest benchmarks game programs in all languages cheat, relative to the slowest ones. :P This isn't an attack on your capacity as the site's maintainer, this is an entirely expected outcome due to the high-profile nature of the site. He's saying that random tests on the internet are "better" because they attract less attention and thus have less exposure. It's a problem with microbenchmarks in general.
- igouy 11y ago
  >>He's saying that...<< Please let Veedrac speak for himself. Veedrac's saying a bunch of stuff -- saying it doesn't make trash-talk into The Truth(TM). >>fastest benchmarks game programs cheat, relative to the slowest ones<< So programs that are compiled cheat, relative to programs that are interpreted? Seems like "cheat" needs to be in scare quotes.
  - kibwen 11y ago
    Your defensiveness is unwarranted and your interpretation assumes bad faith. > So programs that are compiled cheat, relative to programs > that are interpreted? Programming languages whose dominant implementations are interpreted generally (though don't necessarily need to) provide less direct control over hardware, memory layout, and algorithms, and in practice will defer to C via FFI when such control is necessary. Because the benchmarks game generally prohibits calling out to C, it means that many "tricks" employed by low-level languages are unavailable to the higher-level languages. Whether or not this is a desirable property is up to the reader's interpretation. > Seems like "cheat" needs to be in scare quotes. Indeed, because it's essentially impossible to enforce anything like "idiomatic" code in any of the benchmarks game languages, and so whether a program is "cheating" is up to the reader's interpretation. That said, the fact that it is up to the reader's interpretation means that it is valid for some readers to conclude that some programs are "cheating", while other programmers may differ. Personally, I think the point is moot, and that microbenchmarks aren't worth getting so worked up over. :P