3 ms·
https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law You stated it almost directly.
by nathanwh 4y ago
https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law
You stated it almost directly.
- noxvilleza 4y agoOh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.
- bombcar 4y agoIt's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.
- gfody 4y agothere are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/ https://www.singlestore.com/blog/tpc-benchmarking-results/
- qoega 4y agoIt is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.
- gfody 4y agoas long as vendors are presenting the results of their own products, I'd expect them to de-normalize and tune the queries to leverage every advantage their engine can possibly bring to bear on the benchmark's problem set - so long as they're transparent about what they did. Clickhouse apparently does quite well on TPC-DS according to: https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ but I'd love to see a more official result that the CH engineers would stand behind.