5 ms·
No, you didn't seem to read my suggestions at all. >>What, only python? Oh, I need to copy the programs from the "bench" dir. Where in the readme does it say t
by spion 14y ago
No, you didn't seem to read my suggestions at all.
>>What, only python? Oh, I need to copy the programs from the "bench" dir. Where in the readme does it say that? Nowhere.<<
> You don't seem to have read sections #3 through #11.
If you grep the file, there is no mention of the bench dir anywhere in sections 3-11. Please do tell where you found it. The readme does say how to add new programs, but it doesn't say where the original programs of the shootout are in the distribution.
>> of course, I had to remove everything in tmp/* before re-running the test, otherwise it thinks there is "nothing to be done".<<
> No you didn't. You don't seem to have read sections #3 through #11.
No, you didn't read what I was doing. I had to remove everything in tmp/* because I was trying to enable gtop by editing the source code of the program. The old results were made without gtop and only contained cpu and elapsed time.
Finally the "actual comments" on what variables are available for the commandline strings aren't written anywhere. Only "%A" is explained in 9.3 and "%I" isn't even mentioned.
So it looks like all my suggestions remain valid:
From the first suggetion, removing the "tmp" dir assuming that the summary dir bug is fixed in the future:
1) improve readme by mentioning the location of my.linux.ini and my.win32.ini and adding the complete syntax for the commandlines section etc. (important)
2) invert entries in the [alias] section, because the way its set up now it makes no sense. (not important)
3) separate test and environment configuration in different files and share the test configuration files used on the website to help people reproduce the same results. (very important)
4) fix the script to use the introspection bindings for gtop
Note that you failed to comment on (3) which will by far improve the bencher the most. Infact, that is exactly where I stopped trying to reproduce your results - I stopped at the point where I had to extract all the values for the testrange from your CSV files, as the ones used on the website aren't distributed anywhere.
But of course, you're free to continue ignoring useful comments from other people. Which brings me to the original point, we're in dire need of a new benchmarks game.
- igouy 14y ago>>The readme does say how to add new programs, but it doesn't say where the original programs of the shootout are in the distribution.<< You seem to have found the benchmarks game "original programs" in the project tarball. People measure their own programs with bencher, not the benchmarks game programs -- it's not dependent on the benchmarks game programs. >>I had to remove everything in tmp/* because...<< #5.3 #5.4 >>what variables are available for the commandline strings aren't written anywhere<< my.linux.ini [commandlines] >>testrange from your CSV files, as the ones used on the website aren't distributed anywhere<< nanobench/makefiles/u64q.ini nanobench/makefiles/u64.ini nanobench/makefiles/u32q.ini nanobench/makefiles/u32.ini >>you're free to continue ignoring useful comments<< You're comments are useful because they show what someone might become confused about.
- spion 14y ago> You seem to have found the benchmarks game "original programs" in the project tarball. > People measure their own programs with bencher, not the benchmarks game programs -- it's not dependent on the benchmarks game programs. It is not, but if you sincerely want to make it easy for people to reproduce all or parts of the benchmark, you should at least document where they can find all the settings of the original. >>I had to remove everything in tmp/* because...<< #5.3 #5.4 Fair enough, I missed those two points in the README. A weaker point still remains though: its not very user friendly or obvious, like say a --force flag (or something in the spirit of make's --always-make). These things may not seem important, but I think they are when distributing software with the intent of someone else running it. >>testrange from your CSV files, as the ones used on the website aren't distributed anywhere<< nanobench/makefiles/u64q.ini nanobench/makefiles/u64.ini nanobench/makefiles/u32q.ini nanobench/makefiles/u32.ini Ah in nanobench. Well that was definitely not easy or obvious, was it? My fault for not using find on the directory. And how about that suggestion of splitting the configuration file to two separate files: program configuration and environment configuration? It would make it easier to run the benchmarks on the same set of programs with the same parameters, except in a differently configured environment. (the set of language implementations and their build and run commands would be a part of the environment configuration) > You're comments are useful because they show what someone might become confused about. That was exactly my point - to prove that your "easily" claim is a stretch. The README does not cover everything, especially not the parts needed for someone to bench the programs found on the website. Which is understandable since its a README for the bencher program. Perhaps a separate README for the entire archive (documenting where the original configurations and programs are) will fix most of these confusion issues. More descriptive directory names would also help (e.g. "game-programs" and "game-configurations") And if I seem so negative, its because I used to love the benchmarks game and felt that every change you've made lately significantly subtracted from the game's fun factor of it while not significantly adding anything. For example, that recent removal of the (not very useful, but still extremely fun!) combined language comparison table. Why remove things? Its sufficient to simply warn the user. If they ignore the warning and take those numbers seriously, that is their own fault. Why should they be ruining the fun for everyone else? Oh well...