4 ms·
> I've been saying for decades that benchmarks are almost always meaningless I find this statement egregious: benchmarks are probably the most important thing
by flumpcakes 5y ago
> I've been saying for decades that benchmarks are almost always meaningless
I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.
I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
- kentonv 5y agoOh absolutely, benchmarks are great for measuring your own performance and improving it. I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
- _57jb 5y agoI absolutely agree with this sentiment. Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks. The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
- seoaeu 5y agoIt is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting