34 ms·
Not forbidden. You can email a specific address to share results/setup and get permission to publish if you want. Other commercial databases also do that, becau
by mike_hearn 3mo ago
Not forbidden. You can email a specific address to share results/setup and get permission to publish if you want. Other commercial databases also do that, because there's so many ways to misconfigure a database to make it slow and competitors are strongly incentivized to do so.
The question is what you'd want to benchmark. For example, imagine testing Postgres with a write load that goes well beyond what a single machine can do. It would collapse and query latencies would go through the roof. A horizontally scaled DB would keep going and process all those queries. Would you accept this as evidence that Postgres is slow or would you say it's not valid to benchmark at traffic loads Postgres physically cannot handle, given it never claimed to scale horizontally? Stuff like this is where benchmarking gets complicated.
I used to work at a different company that sold a kind of database system. It was much faster than our nearest competitor, so we were surprised when that competitor claimed to an important customer they were just as fast as us. Their benchmark counted transactions that failed and rolled back due to overload (optimistic concurrency) as "successful".
- inigyou 3mo agoI suppose Oracle will only allow publication if your benchmark results look good. Can you tell us about a time you did a benchmark that made Oracle look bad, and they still let you publish it?
- marcosdumay 3mo ago> Other commercial databases also do that That's why you won't see many serious comments bragging about the performance of SQL Server either. Even though is has way fewer performance land-mines that you must design your entire architecture around than Oracle. Anyway, you are correct that RAC is a very impressive piece of software. If Postgres had something like it, it would be a beast. It's not enough to save Oracle, though.
- mike_hearn 3mo agoWhat sort of landmines are you referring to? This is quite interesting. Still it's not just SQL Server and Oracle. PlanetScale - the author of this blog post - also had a DeWitt clause until literally just days ago. They replaced it with a slightly different clause but I wouldn't be willing to benchmark under their new rules. https://planetscale.com/blog/transparency-in-benchmarking https://planetscale.com/blog/transparency-in-benchmarking It says benchmarking is allowed only if "the Benchmark is conducted in good faith and uses a fair and transparent methodology". What does good faith mean, or fair and transparent? You still effectively need their permission because otherwise they might decide you weren't benchmarking in "good faith" retroactively. It's no surprise. The stuff that makes database benchmarking useless is the variety of features. You can always design a benchmark that makes a specific vendor look good or bad. Like, Oracle can add missing indexes on the fly. If someone designed a benchmark in which the core table was huge but missing an index, PostgreSQL would die and Oracle would recover then fly. But how many would accept that as a fair test?
- marcosdumay 3mo ago> What sort of landmines are you referring to? For example the Oracle's incapacity of reordering tests of different kinds (as in constants, functions, rows...) when optimizing a query; or the quadratic run time of triggers that change data in large updates.