5 ms·
TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general lang
by mutagen 13y ago
TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/
[2] https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- pekk 13y agoExcept they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed. The point of saying "send us a pull request" is just PR and it seems to be working.
- RyanZAG 13y agoAFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc. I believe they have plans to add additional tests which will allow for caching and that kind of thing. I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.