2 ms·
> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully. Very i
by carapace 8y ago
> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully.
Very important and often overlooked point.
But I wonder, why not forbid public dissemination of inaccurate, non-reproducible benchmarks?
> Spreading wrong performance information can hurt a business.
Wouldn't that be libel? (IANAL)
- SpicyLemonZest 8y agoThe problem is when the benchmark is accurate, reproducible, and based on a completely nonsensical scenario. It's not libel to say "Jira is 50% slower than Bugzilla" when your benchmark serially creates 25000 tickets, but it's not a fair claim either. That's not to say that anti-benchmarking licenses are the right solution, of course. I can just sympathize with why Atlassian wants one.
- mesozoic 8y agoExpertise so rare it seems Atlassian themselves don't have it.
- SamReidHughes 8y agoSuppose you state that a product performed poorly on measurement Y of benchmark X. The problem is not reproducibility or accuracy, it's that either Y or X are always really stupid in some way or another.