3 ms·

Seems like an opportunity for an enterprising hacker to create a more open and collaborative "benchmarking social network" of sorts.

GregBuchholz 13y ago
You might think so, but keep in mind that it is a thankless volunteer job that's been on-going for something like 10 years. I'm pretty sure that Isaac would encourage your efforts to make that site. http://benchmarksgame.alioth.debian.org/play.php#faqs http://benchmarksgame.alioth.debian.org/play.php#faqs
- brandonbloom 13y ago
  Surely in today's world with elastic compute and robust virtual containers, it should be possible to replace thankless volunteer work with automated, self-service processes. I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
  - awj 13y ago
    Taking the measurements isn't really the thankless volunteer work involved here. The work is in ensuring new programs actually follow the intent of the benchmark and work across supported architectures. This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
    - brandonbloom 13y ago
      Fair enough. Although, I view crowdsourcing as an implementation detail of an automated process.
mutagen 13y ago
TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking. They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark". I'm tempted to tackle this even though I don't currently have the time. [1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- pekk 13y ago
  Except they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed. The point of saying "send us a pull request" is just PR and it seems to be working.
  - RyanZAG 13y ago
    AFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc. I believe they have plans to add additional tests which will allow for caching and that kind of thing. I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.

GregBuchholz 13y ago

You might think so, but keep in mind that it is a thankless volunteer job that's been on-going for something like 10 years. I'm pretty sure that Isaac would encourage your efforts to make that site. http://benchmarksgame.alioth.debian.org/play.php#faqs http://benchmarksgame.alioth.debian.org/play.php#faqs

brandonbloom 13y ago
Surely in today's world with elastic compute and robust virtual containers, it should be possible to replace thankless volunteer work with automated, self-service processes. I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
- awj 13y ago
  Taking the measurements isn't really the thankless volunteer work involved here. The work is in ensuring new programs actually follow the intent of the benchmark and work across supported architectures. This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
  - brandonbloom 13y ago
    Fair enough. Although, I view crowdsourcing as an implementation detail of an automated process.

mutagen 13y ago

TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking. They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark". I'm tempted to tackle this even though I don't currently have the time. [1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks

pekk 13y ago
Except they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed. The point of saying "send us a pull request" is just PR and it seems to be working.
- RyanZAG 13y ago
  AFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc. I believe they have plans to add additional tests which will allow for caching and that kind of thing. I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.