3 ms·

1) Of all the things the data can be used for, one of the least useful ones -- to the point of being nearly meaningless -- is comparing the performance of diffe

by igouy 8y ago

1) Of all the things the data can be used for, one of the least useful ones -- to the point of being nearly meaningless -- is comparing the performance of different languages to each other, the thing that the game's presentation emphasizes most. Is that what the presentation emphasizes most? "Will your toy benchmark program be faster if you write it in a different programming language? It depends how you write it!" "Which programs are faster?" "Which programs are fast?" "Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages." 2) Or look even at "Java" vs "Substrate VM" pidigits. Currently 2 Java pidigits programs are shown. The program with the fastest measurements on OpenJDK uses GMP and something different needs to be done to make that program work on Substrate VM. 3) …but crashes, and what appears to be some junk result is reported. What is reported? Measurements at the largest workload both programs worked. 4) examples … you… seem to say, see, warmup doesn't matter See, for programs with runtimes of 0.24s, 1/10th of a second is a lot! See, for programs with runtimes of 4s and 20s, 1/10ths of a second… ! 5) Now I'm looking at Haskell's fannkuch-redux And… ? 6) … if the comparison was for the same language (and compiler) then this would be interesting, but what does this tell you when comparing different languages? That seems to be a subjective statement about your interests. Someone else might think that neither "tell you" much until you investigate Why?

pron 8y ago
> Is that what the presentation emphasizes most? Yes, because that's what the site shows most prominently: comparisons between languages. The fact that the explanations to the benchmarks basically say that these comparisons mean nothing doesn't make the comparisons mean much more than nothing, especially as there's no explanation to what the comparisons do (or are supposed to) mean. So in addition to having no hypothesis, no analysis and no exposition, there's a notice saying, don't use these comparisons for so and so. Then what are those comparisons for? Again, I don't think the game's comparisons are worse than many other benchmark comparisons out there, but that doesn't make it good. What's unfortunate is that unlike those other benchmarks, they have a lot of data that could be used for some really cool stuff (not to compare languages). It's possible that the language competition is meant to encourage more people to submit and show how their favorite language wins, but there's no information about the market aspects. > See, for programs with runtimes of 4s and 20s, 1/10ths of a second… ! I run a toy benchmark that's the same size as the game's, and HotSpot's C2 gives me a 30-50% boost after ~10s. It really varies by program (not by problem). This tends to be worse for programs that use a lot of parallelism and aren't configured to run compilation in blocking mode, because then the application threads compete with the compilation threads, but again, there isn't really a good rule of thumb here. In any event, it is simply wrong that C2 would never yield significant performance boosts after a few tenths of a second, even on small programs (it may be true for some specific benchmark entries and maybe even all current ones, it may vary by JDK version, and it definitely varies on whether HotSpot uses Graal or C2). Although, given all the other serious problems that render the comparisons mostly meaningless (different algorithms), my guess would be that this effect is smaller by comparison. Still, there's a lot of cool stuff the game could do with the current programs (and historical ones), but comparing languages is not really one of them.
- igouy 8y ago
  > … what the site shows most prominently… Program after Program after Program. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/java.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-java-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Of course, it's really not possible to show any computer program which is not written in a programming language. > … basically say that these comparisons mean nothing … Obviously not: please don't put your words in others' mouths. > … a lot of data that could be used for some really cool stuff… Take the data and do that really cool stuff. https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame > … but there's no information about the market aspects. What do you mean by "market aspects" ? > …it may be true for some specific benchmark entries and maybe even all current ones… So it would be wrong to try to dismiss those measurements, based on nothing more than extrapolation about how some other toy benchmark program might perform. > … cool stuff the game could do … You're allowed your own personal opinion about what's cool and what isn't (and no doubt others will have their own personal opinion).
  - pron 8y ago
    > Obviously not It's certainly not obvious to me what they comparisons mean (is it obvious to you?) and the authors of the site don't even state, let alone support, a hypothesis. They just say what the comparisons don't mean. So unless it's a work of art, I think observers can be forgiven for thinking it doesn't mean anything. > What do you mean by "market aspects"? I mean a hypothesis about "free market competition" driving people to write really fast programs. > You're allowed your own personal opinion about what's cool and what isn't True, but I meant something that could be used in a more rigorous/meaningful way (e.g. comparisons where only one variable changes). Meaning is perhaps also subjective, but I don't even know the subjective opinion of the meaning behind the game, because the site doesn't say. > https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame Cool. Anyway, now that I've taken a slightly closer look, I realize I spoke too soon when I said that the SBCL vs. Java benchmark is "better than nothing." If, say, Java uses parallelism while SBCL doesn't, then if Java is 4x faster that may not be a "better than nothing" indication.

by igouy 8y ago

pron 8y ago

> Is that what the presentation emphasizes most? Yes, because that's what the site shows most prominently: comparisons between languages. The fact that the explanations to the benchmarks basically say that these comparisons mean nothing doesn't make the comparisons mean much more than nothing, especially as there's no explanation to what the comparisons do (or are supposed to) mean. So in addition to having no hypothesis, no analysis and no exposition, there's a notice saying, don't use these comparisons for so and so. Then what are those comparisons for? Again, I don't think the game's comparisons are worse than many other benchmark comparisons out there, but that doesn't make it good. What's unfortunate is that unlike those other benchmarks, they have a lot of data that could be used for some really cool stuff (not to compare languages). It's possible that the language competition is meant to encourage more people to submit and show how their favorite language wins, but there's no information about the market aspects. > See, for programs with runtimes of 4s and 20s, 1/10ths of a second… ! I run a toy benchmark that's the same size as the game's, and HotSpot's C2 gives me a 30-50% boost after ~10s. It really varies by program (not by problem). This tends to be worse for programs that use a lot of parallelism and aren't configured to run compilation in blocking mode, because then the application threads compete with the compilation threads, but again, there isn't really a good rule of thumb here. In any event, it is simply wrong that C2 would never yield significant performance boosts after a few tenths of a second, even on small programs (it may be true for some specific benchmark entries and maybe even all current ones, it may vary by JDK version, and it definitely varies on whether HotSpot uses Graal or C2). Although, given all the other serious problems that render the comparisons mostly meaningless (different algorithms), my guess would be that this effect is smaller by comparison. Still, there's a lot of cool stuff the game could do with the current programs (and historical ones), but comparing languages is not really one of them.

igouy 8y ago
> … what the site shows most prominently… Program after Program after Program. https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/java.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-java-5.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... Of course, it's really not possible to show any computer program which is not written in a programming language. > … basically say that these comparisons mean nothing … Obviously not: please don't put your words in others' mouths. > … a lot of data that could be used for some really cool stuff… Take the data and do that really cool stuff. https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame > … but there's no information about the market aspects. What do you mean by "market aspects" ? > …it may be true for some specific benchmark entries and maybe even all current ones… So it would be wrong to try to dismiss those measurements, based on nothing more than extrapolation about how some other toy benchmark program might perform. > … cool stuff the game could do … You're allowed your own personal opinion about what's cool and what isn't (and no doubt others will have their own personal opinion).
- pron 8y ago
  > Obviously not It's certainly not obvious to me what they comparisons mean (is it obvious to you?) and the authors of the site don't even state, let alone support, a hypothesis. They just say what the comparisons don't mean. So unless it's a work of art, I think observers can be forgiven for thinking it doesn't mean anything. > What do you mean by "market aspects"? I mean a hypothesis about "free market competition" driving people to write really fast programs. > You're allowed your own personal opinion about what's cool and what isn't True, but I meant something that could be used in a more rigorous/meaningful way (e.g. comparisons where only one variable changes). Meaning is perhaps also subjective, but I don't even know the subjective opinion of the meaning behind the game, because the site doesn't say. > https://salsa.debian.org/benchmarksgame-team/benchmarksgame https://salsa.debian.org/benchmarksgame-team/benchmarksgame Cool. Anyway, now that I've taken a slightly closer look, I realize I spoke too soon when I said that the SBCL vs. Java benchmark is "better than nothing." If, say, Java uses parallelism while SBCL doesn't, then if Java is 4x faster that may not be a "better than nothing" indication.