3 ms·

This is a nice list to read through. The concept quoted, Goodhart’s law, came up in a project review today, though I didn’t know it was named. It feels nearly u

This is a nice list to read through. The concept quoted, Goodhart’s law, came up in a project review today, though I didn’t know it was named. It feels nearly unavoidable, but at least recognition helps.

jedbrown 7y ago
This paper attempts to classify and evaluate mitigation techniques for flawed metrics. https://mpra.ub.uni-muenchen.de/90649/1/MPRA_paper_90649.pdf https://mpra.ub.uni-muenchen.de/90649/1/MPRA_paper_90649.pdf
- cpeterso 7y ago
  Thanks! I had hoped the original article might provide approaches for balancing the positives and negatives of metrics, but didn't find any. The "Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws" paper you linked is exactly what I was looking for. :)
csours 7y ago
Keep your eye on the prize, and the prize isn't measured by metrics. Any prize in the real world is something complex and complicated, hard to describe and prone to squishiness. Metrics are things that are "easy" to measure and compare across systems. Setting goals in terms of metrics is appealing and even necessary because it makes things measurable and comparable in the real world, but you need to have some cross-check where you can come back and verify that you are moving closer to your real world goal. --- A real world example: Consumer products are often distinguished by appearance, and surface quality is an important facet of appearance. You purchase a surface quality tester and improve the measurements produced by the surface tester to the point where you beat a competitor's surface measurements. Then you have a customer clinic and have real potential customers evaluate the two products again. Customers prefer the competitor product because you were not actually measuring attractiveness or appearance of quality, you were measuring something that the surface tester measures.
- vanniv 7y ago
  If the impact of your work can be losslessly reduced to something that can be directly and cheaply measured, you're probably solving an easy problem or a theoretical one.

jedbrown 7y ago

This paper attempts to classify and evaluate mitigation techniques for flawed metrics. https://mpra.ub.uni-muenchen.de/90649/1/MPRA_paper_90649.pdf https://mpra.ub.uni-muenchen.de/90649/1/MPRA_paper_90649.pdf

cpeterso 7y ago
Thanks! I had hoped the original article might provide approaches for balancing the positives and negatives of metrics, but didn't find any. The "Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws" paper you linked is exactly what I was looking for. :)

csours 7y ago

Keep your eye on the prize, and the prize isn't measured by metrics. Any prize in the real world is something complex and complicated, hard to describe and prone to squishiness. Metrics are things that are "easy" to measure and compare across systems. Setting goals in terms of metrics is appealing and even necessary because it makes things measurable and comparable in the real world, but you need to have some cross-check where you can come back and verify that you are moving closer to your real world goal. --- A real world example: Consumer products are often distinguished by appearance, and surface quality is an important facet of appearance. You purchase a surface quality tester and improve the measurements produced by the surface tester to the point where you beat a competitor's surface measurements. Then you have a customer clinic and have real potential customers evaluate the two products again. Customers prefer the competitor product because you were not actually measuring attractiveness or appearance of quality, you were measuring something that the surface tester measures.

vanniv 7y ago
If the impact of your work can be losslessly reduced to something that can be directly and cheaply measured, you're probably solving an easy problem or a theoretical one.