4 ms·

Is there an existing named adage for something like "if one creates a benchmark in order to rank general performance of some products, some of those products wi

Is there an existing named adage for something like "if one creates a benchmark in order to rank general performance of some products, some of those products will ultimately sacrifice general performance in order to optimize for that benchmark"?

nathanwh 4y ago
https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law You stated it almost directly.
- noxvilleza 4y ago
  Oh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.
  - bombcar 4y ago
    It's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.
    - gfody 4y ago
      there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/ https://www.singlestore.com/blog/tpc-benchmarking-results/
      - qoega 4y ago
        It is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.
        gfody 4y ago
        as long as vendors are presenting the results of their own products, I'd expect them to de-normalize and tune the queries to leverage every advantage their engine can possibly bring to bear on the benchmark's problem set - so long as they're transparent about what they did. Clickhouse apparently does quite well on TPC-DS according to: https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ but I'd love to see a more official result that the CH engineers would stand behind.
deleted 4y ago
[deleted]

nathanwh 4y ago

https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law You stated it almost directly.

noxvilleza 4y ago
Oh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.
- bombcar 4y ago
  It's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.
  - gfody 4y ago
    there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/ https://www.singlestore.com/blog/tpc-benchmarking-results/
    - qoega 4y ago
      It is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.
      - gfody 4y ago
        as long as vendors are presenting the results of their own products, I'd expect them to de-normalize and tune the queries to leverage every advantage their engine can possibly bring to bear on the benchmark's problem set - so long as they're transparent about what they did. Clickhouse apparently does quite well on TPC-DS according to: https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ but I'd love to see a more official result that the CH engineers would stand behind.

deleted 4y ago

[deleted]