4 ms·
There's flawed benchmarks and there are completely wrong benchmarks. I've seen so many benchmarks where the timing or loop overhead dominated to the point that
by aidenn0 5y ago
There's flawed benchmarks and there are completely wrong benchmarks.
I've seen so many benchmarks where the timing or loop overhead dominated to the point that they weren't measuring anything but the cost of taking timing values
I've seen benchmarks where they run a single time without warming up the disk cache, so whichever benchmark they happen to run first shows as 2 orders of magnitude slower.
I've seen benchmarks where the standard deviation was much larger than the difference between the times reported, but they didn't bother to check for this.
I've seen benchmarks where they don't even use the same algorithm, and the difference in the algorithm chosen dominates any other concerns.
- still_grokking 5y agoExactly! And even those very wrong results give quite interesting insides. Such stuff is worthless as benchmarks, sure. But all the other things you can learn form such stuff is sometimes even quite fascinating. So discussing completely wrong benchmarks has imho often some value.
- aidenn0 5y agoIt's entirely possible that the net utility of such benchmarks being published is negative; enough people come away with the initial wrong conclusion without critically investigating it. It's almost certainly true that the net value to the company whose product looks bad in the benchmarks is negative, which is why these clauses exist.
- kaba0 5y agoOf course you can learn from these. But do they describe anything related to the product at hand?