5 ms·
It's ridiculous and sad that they would include Rust, after banning LuaJIT a few years ago, citing "too much maintenance effort" (even though LuaJIT benchmarks
by tomp 13y ago
It's ridiculous and sad that they would include Rust, after banning LuaJIT a few years ago, citing "too much maintenance effort" (even though LuaJIT benchmarks were significantly more complete that Rust's)
- masklinn 13y agoThe shootout has been utterly ridiculous for quite a long time. I remember when the maintainer(s?) threw a fit when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy) even though multiple programs per language had not been an issue before. Then they ended up banning multiple implementations, but their implementation pick is completely arbitrary.
- brandonbloom 13y agoSeems like an opportunity for an enterprising hacker to create a more open and collaborative "benchmarking social network" of sorts.
- GregBuchholz 13y agoYou might think so, but keep in mind that it is a thankless volunteer job that's been on-going for something like 10 years. I'm pretty sure that Isaac would encourage your efforts to make that site. http://benchmarksgame.alioth.debian.org/play.php#faqs http://benchmarksgame.alioth.debian.org/play.php#faqs
- brandonbloom 13y agoSurely in today's world with elastic compute and robust virtual containers, it should be possible to replace thankless volunteer work with automated, self-service processes. I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
- awj 13y agoTaking the measurements isn't really the thankless volunteer work involved here. The work is in ensuring new programs actually follow the intent of the benchmark and work across supported architectures. This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
- brandonbloom 13y agoFair enough. Although, I view crowdsourcing as an implementation detail of an automated process.
- mutagen 13y agoTechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking. They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark". I'm tempted to tackle this even though I don't currently have the time. [1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- pekk 13y agoExcept they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed. The point of saying "send us a pull request" is just PR and it seems to be working.
- RyanZAG 13y agoAFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc. I believe they have plans to add additional tests which will allow for caching and that kind of thing. I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
- hansjorg 13y agoThe Benchmarks Game maintainer explains why in the comments to this blog post: http://alexgaynor.net/2011/apr/03/my-experience-computer-language-shootout/ http://alexgaynor.net/2011/apr/03/my-experience-computer-lan... Sounds entirely reasonable to me.
- igouy 13y ago>>when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy)<< NOT TRUE. Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted. >>banning multiple implementations<< NOT TRUE. Measurements for both JRuby and CRuby are shown. Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements. The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.
- frik 13y agoI always wondered why LuaJIT 2, HHVM (HipHop PHP) and PyPy are missing. That's sad. Someone should fork it... would be great. Another good benchmark that compares them all: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=update http://www.techempower.com/benchmarks/#section=data-r8&hw=i7... (found it on HN some month ago)
- alayne 13y agoigouy pretty much gets nothing but grief maintaining the shootout. He's told people for years that they should publish their own numbers for different language implementations. Never happens, just complaints.
- frik 13y agoLuaJIT, PyPy and HHVM are common and well know. If it is true that they devs contacted him and he refused to add it, that's sad. I know maintaining a project like that is a lot of work. But adding superior implementations of already featured languages should be a piece of cake. Source code of Python, PHP and Lua is already in the repo, it's just a matter of installing the VM's on the computer and modify the scripts. Beside, if the maintainer reads this, please keep up the good work, and maybe add LuaJIT, PyPy and HHVM. Thanks.
- awj 13y ago> it's just a matter of installing the VM's on the computer and modify the scripts. That can be a pretty huge "it's just". In practice many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs. Since the alternate implementation, benchmark programs, and benchmark tools are all available, I'd recommend you put your theory to the test. It'll either prove eye opening or make for a nice contribution back to the game!
- frik 13y ago> many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs HHVM is compatible with PHP 5.3, LuaJIT with Lua 5.1 and PyPy with Python 2.7.3. All benchmark code is written the languages code and sometimes relies on an language extension library (which is written in C or C++). But three above JIT VMs come with their own compatible language extension libraries or support the extension API. Look at the code, it should be easy: http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?test=pidigits&lang=php&id=1&data=u32q http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?t... and https://github.com/kragen/shootout/tree/master/bench https://github.com/kragen/shootout/tree/master/bench
- pjmlp 13y agoThey also banned D.
- igouy 13y agoThey also did not ban D.
- ginko 13y agoI always wondered why hand-optimized Assembly wasn't in the list of competing languages. That would be good as a baseline. Otherwise people might get the impression that C is the "speed of light" as far as program speed is concerned.
- frik 13y agoYes optimized ASM would be a great baseline. Fortran is already faster in several benchmarks than C.
- igouy 13y agoafaict we all feel the same way about this, we all feel that we should sit on our hands and wait for someone else to do the chores we don't wish to do. Of course, measurements of the "proggit popular" language implementations like PyPy and LuaJIT will attract attention and be the basis of yet another successful website (unlike more Fortran or Ada or Pascal or Lisp). http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex