4 ms·

Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that ha

Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that have the budget for the hardware are even more rare. The last thing the world needs is for the web to be polluted by so many incompetent "benchmarks" performed by the clueless that you can't find the authoritative information that you need to do your job. Take a wild guess about how much memory and how many disk spindles it takes to get a TPC benchmark on a modern server to be CPU limited. Take a guess as to how many weeks of engineering time it takes to perform and validate the benchmark after getting a pile of hardware delivered to your lab. The average blogger writing 3 posts a week ain't gonna do it.

zimbatm 8y ago
There is always a well sounding rationale attached to censorship. It also prevents meaningful benchmarks to be published.
- walterbell 8y ago
  And it's a self-fulfilling prophecy if we don't allow inexperienced benchmarkers to learn how to run meaningful tests.
- kjeetgill 8y ago
  I'm on the fence about the benchmarks but I think this is a generic broad brush to sweep away the parent's comment given the real concerns they've presented and you've chose to not address. Do you really thing any agreement that involves silence is censorship NDAs included? Sure, it's easy to simply say: sucks to be you! Especially to "community bullies" like Oracle. Even just between semi-competing open source projects benchmarks can be so bad they border on libelous.
  - harry8 8y ago
    Oracle has just enough budget to hire someone knowledgeable to engage in the discussion, pointing out the issues and their fixes, possibly even linking the benchmarks that were (in their view) properly performed. One good engineer. Compare with the legal fees required to draft such an appalling, scared and knowingly guilty clause as "you can't talk about how our product performs despite having paid for it." This really needs to be treated with total derision, and utter contempt. It's an approach that has earned that response for it's sheer laziness before even beginning to measure it against any ethical standards. Oracle are just an awful company in every dimension. This, is just like most everything else they do in that it reinforces the reputation they earned for themselves.
jenscow 8y ago
Well, most times I'd like to know how a badly configured X compares with badly configured Y, on <random hardware> - because that's usually how things are run.
adrianratnapala 8y ago
But this doesn't mean having the vendor control publication is going to get you any benchmarks that are better. We can predict pretty easily that the benchmarks they publish will be skewed in their favour. On the other hand, we can't predict what solutions society comes up with for dealing with incompetent benchmarks. For example, you say professional bloggers would do lousy jobs, but this is exaclty the kind of things that neckbeard hobbyist bloggers would take upon themselves and do excellent jobs at. That or other solutions could emerge, but the whole point of censorship is to prevent that.
zaat 8y ago
Except they started the practice in response to an academic researcher publishing, while simultaneously trying to get the researcher fired. https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/
flukus 8y ago
> Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that have the budget for the hardware are even more rare. The last thing the world needs is for the web to be polluted by so many incompetent "benchmarks" performed by the clueless that you can't find the authoritative information that you need to do your job. If these apparently incompetent people running the benchmarks are also the incompetent people creating the schema's and queries out in the wild then the benchmarks are still worthwhile. I want to know what performance I can expect, not what I can expect when the environment is highly tuned by rare experts that will never touch any of the systems I have to deal with.

zimbatm 8y ago

There is always a well sounding rationale attached to censorship. It also prevents meaningful benchmarks to be published.

walterbell 8y ago
And it's a self-fulfilling prophecy if we don't allow inexperienced benchmarkers to learn how to run meaningful tests.
kjeetgill 8y ago
I'm on the fence about the benchmarks but I think this is a generic broad brush to sweep away the parent's comment given the real concerns they've presented and you've chose to not address. Do you really thing any agreement that involves silence is censorship NDAs included? Sure, it's easy to simply say: sucks to be you! Especially to "community bullies" like Oracle. Even just between semi-competing open source projects benchmarks can be so bad they border on libelous.
- harry8 8y ago
  Oracle has just enough budget to hire someone knowledgeable to engage in the discussion, pointing out the issues and their fixes, possibly even linking the benchmarks that were (in their view) properly performed. One good engineer. Compare with the legal fees required to draft such an appalling, scared and knowingly guilty clause as "you can't talk about how our product performs despite having paid for it." This really needs to be treated with total derision, and utter contempt. It's an approach that has earned that response for it's sheer laziness before even beginning to measure it against any ethical standards. Oracle are just an awful company in every dimension. This, is just like most everything else they do in that it reinforces the reputation they earned for themselves.

jenscow 8y ago

Well, most times I'd like to know how a badly configured X compares with badly configured Y, on <random hardware> - because that's usually how things are run.

adrianratnapala 8y ago

But this doesn't mean having the vendor control publication is going to get you any benchmarks that are better. We can predict pretty easily that the benchmarks they publish will be skewed in their favour. On the other hand, we can't predict what solutions society comes up with for dealing with incompetent benchmarks. For example, you say professional bloggers would do lousy jobs, but this is exaclty the kind of things that neckbeard hobbyist bloggers would take upon themselves and do excellent jobs at. That or other solutions could emerge, but the whole point of censorship is to prevent that.

zaat 8y ago

Except they started the practice in response to an academic researcher publishing, while simultaneously trying to get the researcher fired. https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/

flukus 8y ago

> Here is the thing, at least in the case of database software, the people that are competent to set up a benchmark correctly are fairly rare. The people that have the budget for the hardware are even more rare. The last thing the world needs is for the web to be polluted by so many incompetent "benchmarks" performed by the clueless that you can't find the authoritative information that you need to do your job. If these apparently incompetent people running the benchmarks are also the incompetent people creating the schema's and queries out in the wild then the benchmarks are still worthwhile. I want to know what performance I can expect, not what I can expect when the environment is highly tuned by rare experts that will never touch any of the systems I have to deal with.