21 ms·
This is why you can never accept benchmarks provided solely by the software creators. Same for accepting studies about a company's product when the company's co
by meowface 7y ago
This is why you can never accept benchmarks provided solely by the software creators. Same for accepting studies about a company's product when the company's commissioned and funded the studies.
It'd be cool if there were neutral third-parties, kind of like Jepsen, that any project could defer rigorous benchmarking to, perhaps in exchange for a flat fee (everyone pays the same fee, no matter how big or small they are).
- munmaek 7y agoAnd then they learn how to game the benchmarks. You just can’t win.
- skrebbel 7y agoNo, because the trick is that you're paying a knowledgeable person to run the benchmark. That person would presumably actively iterate on the benchmarks and try to detect / avoid cheating.
- kmbriedis 7y agoPeople would probably find out what hardware they use for benchmarks and optimize for that, leading to performance decrease for many othes
- deleted 7y ago[deleted]
- mhh__ 7y agoI am writing a small benchmarking site (aimed at asymptotic performance rather than singular tasks) which should hopefully have a fairly generic API/Specification, so I'll look into the more "competitive" side of it (Project vs Project as opposed "Uh oh, Commit f4r0adfja has made the build slower and [test 3] slower for large n". The logic itself is easy but I'm having to step into the dark arts of DevOps and web-cancer - should be online by the summer.