5 ms·
Fascinating things I learned from this article: 1. The benchmarks game is mostly a game of who can solve the problem fastest with the constraint of "all the co
by geofft 11y ago
Fascinating things I learned from this article:
1. The benchmarks game is mostly a game of who can solve the problem fastest with the constraint of "all the code has to be in this language." It's not about how anyone would realistically write the code; if ludicrous optimizations were in-scope for any real-world project, so would be calling out to a different language. It's worse than the usual problem of benchmarks being unrepresentative of real code; the implementations are also unrepresentative.
2. Apparently performant Haskell involves unchecked raw memory access? What's the story there?
- jacobolus 11y agoAll of these cross-language benchmarks inevitably end up like this. They tend to tell you as much or more about the cleverness/knowledge/available free time of the people writing the specific implementation in each language as they tell you about the languages’ inherent differences. I don’t think it’s a solvable problem though, as there’s no perfect definition of “how anyone would realistically write the code”. Realistically, people are going to write the first naïve implementation they can think of until it becomes a performance problem and someone has time/resources to throw at speeding things up. In the case that a piece of code is a bottleneck and performance matters enough to devote engineering effort, folks will start doing all kinds of crazy “unrealistic” optimizations to real-world code.
- fauigerzigerk 11y agoI think the benchmarks would tell us much more if they compared implementations of text book algorithms (and data structures) instead of comparing solutions to text book problems. I realize that this would not necessarily make the code any more realistic or idiomatic. On the contrary, it could make the code extremely contrived for some algorithms in some languages. Some combinations might even be unimplementable. But I don't care about that, because, as you say, what is and isn't idiomatic is very subjective and it gets thrown out anyway once we decide that some piece of code does need to run fast. What makes these benchmarks pretty useless is that you don't know whether the choice of algorithm or the choice of programming language dominates a particular result.
- titanomachy 11y agoWell, you can learn something if you read the code at least.
- igouy 11y ago>>implementations of text book algorithms<< You can do that comparison and publish the results. >> choice of algorithm or the choice of programming language<< Do the n-body programs implement different algorithms? http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=nbody http://benchmarksgame.alioth.debian.org/u64q/performance.php...
- fauigerzigerk 11y agoSorry, my mistake. I thought that they did not mandate one particular algorithm, but apparently I was wrong. They say: "We are trying to show the performance of various programming language implementations - so we ask that contributed programs not only give the correct result, but also use the same algorithm to calculate that result." Thanks for the correction.
- igouy 11y agoWell, "we ask" is not the same as "we receive" :-) Also, because of the differences between such a wide range of programming languages, there has to be some slop and flexibility in the way the programs are allowed to be written -- otherwise it really would be C written in Haskell. http://benchmarksgame.alioth.debian.org/dont-jump-to-conclusions.html#comparison http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...
- deleted 11y ago[deleted]
- kbenson 11y agoWell, yes. Another way to look at the benchmark game is "if I'm not using a fast low-level language, theoretically what speed gain might I expect if I wrote a portion of my project in X language and called out to it (ignoring data marshaling)?" It would be interesting if entries were categorized into idiomatic and non-idiomatic representations of that language, but that would also require people agree on what is idiomatic for that language, which is easier said than done in a lot of cases.
- jamii 11y agoIf you just look at the 'best' numbers for each language and treat it as a fight, it's pretty useless. But if you are evaluating a new language it can be very educational to look at all the solutions for that language and see the range of tradeoffs between speed, space and readability. The short and simple programs show you what kind of performance you can expect from straightforward, idiomatic code and the crazy optimised versions show you how far you can push the performance bracket and what you have to sacrifice to get there (performance, safety etc).
- igouy 11y agoIf we just look at the 'best' numbers for each language and treat it as a fight -- we've chosen to do something pretty useless ;-)
- deleted 11y ago[deleted]
- igouy 11y ago>>ludicrous optimizations<< If only you'd look at the code. There can be several quite different programs for the same language implementation: http://benchmarksgame.alioth.debian.org/u64q/program.php?test=spectralnorm&lang=gpp&id=1 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
- qznc 11y ago> the constraint of "all the code has to be in this language." No that is not a usual constraint, many implementations use third party libraries. E.g., regexdna implementations usually rely on some regex library and pidigits just measures libgmp.
- steveklabnik 11y agoThe "third party" bit is hard to define as well. For example, Rust doesn't include regular expressions in the standard library, because we have made an explicit choice to keep the standard library small. But the regex package is maintained by the Rust team, so it's not _exactly_ third party. But some languages with regexes in the language just have bindings to PCRE or something similar... so it's "first party" in the sense that it's built-in, but "third party" in the sense that the code wasn't really written and isn't maintained by them.
- mcguire 11y agoIt is a valid constraint, though. Many long-standing complaints from the Java side of things were along the lines of "We can't use libgmp."
- igouy 11y agohttp://benchmarksgame.alioth.debian.org/u64q/program.php?test=pidigits&lang=java&id=2#sourcecode http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...