5 ms·

I think we're in dire need of a new benchmarks game. The author: removed PyPy removed the clear overview table that showed all languages, keeping only the bar

by spion 14y ago

I think we're in dire need of a new benchmarks game. The author: removed PyPy removed the clear overview table that showed all languages, keeping only the barely visible gray bar charts with vertical text (why not horizontal bar charts, they can be scrolled and read without head tilting) refuses to add LuaJIT and other languages responds to most change inquiries with "publish your own version" which probably means that he doesn't really care about the project anymore As a result, the spirit of the benchmarks game was lost a long time ago as its made less fun with each iteration.

igouy 14y ago
Not only that, but "the author" has for many years provided the source code for the scripts to make the measurements -- so anyone who actually wants to can easily start a new benchmarks game. Go do! http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex
- gnuvince 14y ago
  Why ask people to start their own benchmarks (presumably with Blackjack and hookers) instead of allowing people to contribute to the Benchmark Games?
  - igouy 14y ago
    Because they don't wish to contribute to the benchmarks game, they wish to see something different -- and that's fine with me.
- spion 14y ago
  Its far from "easily". Its not like you could just fork the code and run the benchmarks game - you would not only have to setup the environment for all languages, but also understand the not-so-well documented idiosyncrasies and deficiencies of the implementation (hardcoded paths? seriously?) It doesn't seem like the code was originally written to be understood and modified by others and its not fully documented. As such your claim that its simple to fork it isn't really valid. (However I'm willing to admit the possibility that the configuration complexity is not easily avoidable) I agree that its far from easy to write a benchmarks game, setup a test environment and invest oh-so-many CPU cycles into running the tests periodically. But I wouldn't go around telling people that forking the original is easy either - that would be very misleading. Much will have to be done to improve the current engine to make it easier to fork. I'm seriously considering doing that (or alternatively making a new benchmarks game from scratch with the intent of making it easy to configure and fork)
  - igouy 14y ago
    > Its not like you could just fork the code and run the benchmarks game... Some people download the scripts and make measurements straight-away, as always YMMV. 1) REQUIREMENTS Willingness to read the README Willingness to write ini file name=value properties Willingness to (sometimes) write make commands Install Python 2.5+ (these are Python scripts)
    - spion 14y ago
      > Some people download the scripts and make measurements straight-away, as always YMMV. Without setting up the language environments? I doubt it. Some highlights: 3) LINUX QUICK START [ WIN32 QUICK START ] 3.1) unzip in ~ directory [ unzip in c:\ ] Requiring a path? Very poor. *** TO BE CONTINUED *** Oh so its not fully documented, but its easy to use straight away... I'll take the bite. Lets try and add luajit and compare it to lua. > 12.1) In the [tools] section of the ini file, define a name that will substitute for the location of the new language implementation. Where is that INI file? The README doesn't say. Oh its my.linux.ini (its not mentioned anywhere in the document). > For example [tools] PYTHON3 = /usr/local/src/Python-3.2/bin/python3.2 Fair enough, lets add my current luajit LUAJIT = /usr/local/bin/luajit > 12.2) Choose a file extension to identify programs > and measurements made with the new language > implementation, for example - python3. Okay, picking .luajit > 12.3) In the [commandlines] section of the ini file, define a command line that will be used to run program source code files that have the new file extension you chose. > For example, for file extension python3 [commandlines] python3 = $PYTHON3 %X %A No comment on what %X or %A may mean - they could mean anything, really. Of course %X is the source (or built) file and %A are the arguments. Easy enough. luajit = $LUAJIT %X %A > 12.4.1) EITHER alias existing source code files that have a different file extension with the new file extension, in the [alias] section of the ini file. > For example, re-use all source code files with file extension python but make measurements identified with file extension python3 [alias] python = python3 So let me get this straight, to define the "python3" alias, I need to add the "python" key to the alias section and set it to "pyton3". Definitely not intuitive, I would rewrite this to be reversed. But okay, I'll define the luajit alias [alias] lua = luajit > 12.5) Run the bencher.py script. Running. bencher release 0.9 gtop import failed - measure cpu & elapsed time Wed 21:22:51 .....OK .....nbody.python [2] Wed 21:22:58 .OK .....regexdna.python [1] What, only python? Oh, I need to copy the programs from the "bench" dir. Where in the readme does it say that? Nowhere. Okay, did that, re run the script, and it runs all nbody and regexdna benchmarks, and none of the others. Why? Going back to my.linux.ini. Could it be "onlydirs" perhaps? So I should add all the items in onlydirs, lets try that. Okay, now its working - at least for the extensions that have a command defined. Its time to wait... But wait, it complained about gtop? Why? Oh, it uses an old way of importing GTop, the new way is to use the introspected bindings from gi.repository import GTop lets patch uselinux.py Wed 21:55:42 .EMPTY pidigits.luajit of course, I had to remove everything in tmp/* before re-running the test, otherwise it thinks there is "nothing to be done". Okay, that didn't quite work. Lets forget about gtop for now and look at the results we got without it > 6) LOOKING AT THE MEASUREMENTS > ~/bencher/summary/all_measurements.csv > Open all_measurements.csv with a text editor. No summary dir here. Well shoot, what does the README say about a missing summary dir? Nothing. Oh, its in ~/bencher/tmp/all_measurements.csv So what do the times say? binarytrees,luajit,3,0,633,0.004,0,0,%,0.010 binarytrees,lua,2,0,525,0.000,0,-2,%,0.005 binarytrees,lua,3,0,633,0.000,0,-2,%,0.007 binarytrees,luajit,2,0,525,0.000,0,0,%,0.003 Lets compare that with the summary data: binary-trees,Lua,2,12,446,0.796,4516,0,0% 0% 1% 100%,0.801 So uhh, I have to configure the parameters for every benchmark. Why don't you share your configuration? Oh, I see, there are various paths defined there appropriate for your environment. Wait what? Why don't you separate the test configuration from the environment configuration? First conclusion: The bencher needs more work to make it easier to fork: * improve readme by actually mentioning which ini file users should look at, telling them about the tmp dir, explaining the value syntax for the commandlines section etc. (important) * invert entries in the [alias] section, because the way its set up now it makes no sense. (not that important) * separate test and environment configuration in different files and share the test configuration files used on the website to help people reproduce the same results. (very important) * fix the script to use the introspection bindings for gtop Second conclusion: The claim that its "easy" to set up has been nullified.

by spion 14y ago

igouy 14y ago

Not only that, but "the author" has for many years provided the source code for the scripts to make the measurements -- so anyone who actually wants to can easily start a new benchmarks game. Go do! http://benchmarksgame.alioth.debian.org/play.php#languagex http://benchmarksgame.alioth.debian.org/play.php#languagex

gnuvince 14y ago
Why ask people to start their own benchmarks (presumably with Blackjack and hookers) instead of allowing people to contribute to the Benchmark Games?
- igouy 14y ago
  Because they don't wish to contribute to the benchmarks game, they wish to see something different -- and that's fine with me.
spion 14y ago
Its far from "easily". Its not like you could just fork the code and run the benchmarks game - you would not only have to setup the environment for all languages, but also understand the not-so-well documented idiosyncrasies and deficiencies of the implementation (hardcoded paths? seriously?) It doesn't seem like the code was originally written to be understood and modified by others and its not fully documented. As such your claim that its simple to fork it isn't really valid. (However I'm willing to admit the possibility that the configuration complexity is not easily avoidable) I agree that its far from easy to write a benchmarks game, setup a test environment and invest oh-so-many CPU cycles into running the tests periodically. But I wouldn't go around telling people that forking the original is easy either - that would be very misleading. Much will have to be done to improve the current engine to make it easier to fork. I'm seriously considering doing that (or alternatively making a new benchmarks game from scratch with the intent of making it easy to configure and fork)
- igouy 14y ago
  > Its not like you could just fork the code and run the benchmarks game... Some people download the scripts and make measurements straight-away, as always YMMV. 1) REQUIREMENTS Willingness to read the README Willingness to write ini file name=value properties Willingness to (sometimes) write make commands Install Python 2.5+ (these are Python scripts)
  - spion 14y ago
    > Some people download the scripts and make measurements straight-away, as always YMMV. Without setting up the language environments? I doubt it. Some highlights: 3) LINUX QUICK START [ WIN32 QUICK START ] 3.1) unzip in ~ directory [ unzip in c:\ ] Requiring a path? Very poor. *** TO BE CONTINUED *** Oh so its not fully documented, but its easy to use straight away... I'll take the bite. Lets try and add luajit and compare it to lua. > 12.1) In the [tools] section of the ini file, define a name that will substitute for the location of the new language implementation. Where is that INI file? The README doesn't say. Oh its my.linux.ini (its not mentioned anywhere in the document). > For example [tools] PYTHON3 = /usr/local/src/Python-3.2/bin/python3.2 Fair enough, lets add my current luajit LUAJIT = /usr/local/bin/luajit > 12.2) Choose a file extension to identify programs > and measurements made with the new language > implementation, for example - python3. Okay, picking .luajit > 12.3) In the [commandlines] section of the ini file, define a command line that will be used to run program source code files that have the new file extension you chose. > For example, for file extension python3 [commandlines] python3 = $PYTHON3 %X %A No comment on what %X or %A may mean - they could mean anything, really. Of course %X is the source (or built) file and %A are the arguments. Easy enough. luajit = $LUAJIT %X %A > 12.4.1) EITHER alias existing source code files that have a different file extension with the new file extension, in the [alias] section of the ini file. > For example, re-use all source code files with file extension python but make measurements identified with file extension python3 [alias] python = python3 So let me get this straight, to define the "python3" alias, I need to add the "python" key to the alias section and set it to "pyton3". Definitely not intuitive, I would rewrite this to be reversed. But okay, I'll define the luajit alias [alias] lua = luajit > 12.5) Run the bencher.py script. Running. bencher release 0.9 gtop import failed - measure cpu & elapsed time Wed 21:22:51 .....OK .....nbody.python [2] Wed 21:22:58 .OK .....regexdna.python [1] What, only python? Oh, I need to copy the programs from the "bench" dir. Where in the readme does it say that? Nowhere. Okay, did that, re run the script, and it runs all nbody and regexdna benchmarks, and none of the others. Why? Going back to my.linux.ini. Could it be "onlydirs" perhaps? So I should add all the items in onlydirs, lets try that. Okay, now its working - at least for the extensions that have a command defined. Its time to wait... But wait, it complained about gtop? Why? Oh, it uses an old way of importing GTop, the new way is to use the introspected bindings from gi.repository import GTop lets patch uselinux.py Wed 21:55:42 .EMPTY pidigits.luajit of course, I had to remove everything in tmp/* before re-running the test, otherwise it thinks there is "nothing to be done". Okay, that didn't quite work. Lets forget about gtop for now and look at the results we got without it > 6) LOOKING AT THE MEASUREMENTS > ~/bencher/summary/all_measurements.csv > Open all_measurements.csv with a text editor. No summary dir here. Well shoot, what does the README say about a missing summary dir? Nothing. Oh, its in ~/bencher/tmp/all_measurements.csv So what do the times say? binarytrees,luajit,3,0,633,0.004,0,0,%,0.010 binarytrees,lua,2,0,525,0.000,0,-2,%,0.005 binarytrees,lua,3,0,633,0.000,0,-2,%,0.007 binarytrees,luajit,2,0,525,0.000,0,0,%,0.003 Lets compare that with the summary data: binary-trees,Lua,2,12,446,0.796,4516,0,0% 0% 1% 100%,0.801 So uhh, I have to configure the parameters for every benchmark. Why don't you share your configuration? Oh, I see, there are various paths defined there appropriate for your environment. Wait what? Why don't you separate the test configuration from the environment configuration? First conclusion: The bencher needs more work to make it easier to fork: * improve readme by actually mentioning which ini file users should look at, telling them about the tmp dir, explaining the value syntax for the commandlines section etc. (important) * invert entries in the [alias] section, because the way its set up now it makes no sense. (not that important) * separate test and environment configuration in different files and share the test configuration files used on the website to help people reproduce the same results. (very important) * fix the script to use the introspection bindings for gtop Second conclusion: The claim that its "easy" to set up has been nullified.