Exams and sports competitions are a particularly nasty example of Campbell's Law (or Goodheart's Law) – collapsing the correlation between metric and the true g

Exams and sports competitions are a particularly nasty example of Campbell's Law (or Goodheart's Law) – collapsing the correlation between metric and the true goal. Exams, for example, are supposed to measure how much students progressed. "True goal" - learn some knowledge or master some skill. "Metric" is the quiz/tests/exam result. If you try to design an exam for any new skill or knowledge, you'll come up with a pretty good metric. But soon, people will start ignoring the true goal and focus on the metric. Instead of "measuring" how well they learned the skill, your metric measures "how well they overfitted for the exam". In a way, the mere fact that people need to "prepare for exams" is a testament to how failed the system is. Sport is similar. The initial idea of the judging system and competitions is to "measure" the performance of the athlete. Rewards and rules are introduced to make this metric clear and unambiguous. Fast-forward a couple of decades, and athletes and coaches don't care about overall performance – they care about winning the medal, so they focus only on the subset of a skill that gives points. This problem is especially horrifying in the sports like figure skating after introducing the IJS system in 2006. Correlation between "true goal" (skating skills) and "metrics" (points given for specific elements) is collapsed – people chase points, not the skill. It's not a simple problem. There are few neat papers on how to approach solving Goodheart's problems ([1] [2]). But on a practical level, the number one tool seems to be to decrease the importance of the metric. Some sports, for example, have "one major competition" once in 4 years (Olympic Games). The price of winning it is insane. Who cares about the true goal if it's almost life-and-death importance? You chase the points. But other sports have a more healthy approach – tennis, for example, doesn't care much about the Olympics. They have Slams and different surfaces competitions throughout the year. You lost a game - not the end of the world. There will be many more. The higher prize is to win all four Slams in your career, so the price of the metric at a single competition is not that huge. And this collapsing metric is a reason why people cheat. You can't cheat on a true goal, only on a proxy metric. [1] Categorizing Variants of Goodhart's Law - https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585 [2] Building Less Flawed Metrics: Dodging Goodhart and Campbell's Laws - https://www.researchgate.net/publication/334478956_Building_Less_Flawed_Metrics_Dodging_Goodhart_and_Campbell%27s_Laws https://www.researchgate.net/publication/334478956_Building_...