14 ms·

There is a deeper and largely overlooked problem with sports judging, especially in the sports with artistic component (like gymnastics or figure skating) – Goo

by divan 3y ago

There is a deeper and largely overlooked problem with sports judging, especially in the sports with artistic component (like gymnastics or figure skating) – Goodhart's effects. Goodhart's effect stands for "when the metric becomes a target it stops being a good metric". As soon as you make measurement part of the reward system, it changes the whole system. Take figure skating as an example. Since 2006, ISU has introduced a similar judging system - IJS – that uses points for elements and an increasingly complex code of rules on how to assign those points. Each element has a so-called "base value" and judges determine GOE value in the range from -5 to +5 for each element. The computer averages that and takes trimmed mean to produce the final points for the element. Those base values and points are logically assigned by the complexity of elements. More complex elements cost more. Great skaters can perform more complex elements. Sounds logical, right? What has happened, though, is that using these points as a target has changed the way how skaters train. Why spend time learning "easy" elements that give little points? All time must be spent now learning "complex" elements that give more points. In a way, before, IJS skaters were chasing overall greatness; now, they chase points. "Skater's greatness" – whatever that means – is hard to define, let alone measure. The ability to perform complex jumps was a consequence of it, not a cause. Using points as a metric changes the whole system of training, and a new generation of skaters can perform insane quads (often because of pre-puberty body sizes, though), but no one would call them great skaters. I'm not even starting with how these points affected artistry. "Program components" (PCS) that should measure artistry are practically a joke now. You basically can multiply jump scores by some constant and get PCS. [1] All skating programs look the same now and are very predictable because of points maximization. Back to AI. The rules for figure skating become more and more complex each year in order to duct-tape the constantly emerging issues. Many judges admit that complexity is so high that the human brain can't even possibly apply them in real time and has to resort to cognitive shortcuts. There are some early flirtations with using AI for judging figure skating [2], but nothing serious yet. As outlined in the "Categorized Variants of Goodhart's Law" [3], one of the major reasons of this effect in sports is not just the fact that elements != greatness, but the fact that there is no shared definition of what is a True Goal of this sport. What exactly judging system try to measure? What would constitute the best skater? Where there is no explicitly stated True Goal, any proxy variable used as metrics (like "ability to perform complex elements") will be far from good and will be distorting the system even further. Eventually, this boils down to the structure and decision-making in the international federation that governs this sport. And these things cannot be solved by AI. In a way, using AI is a yak-shaving for ever-increasing rules of an inherently wrong judging system. [1] http://libjournals.unca.edu/OJS/index.php/mas/article/view/23 http://libjournals.unca.edu/OJS/index.php/mas/article/view/2... [2] https://www.youtube.com/watch?v=pq_mv1eyZyA https://www.youtube.com/watch?v=pq_mv1eyZyA [3] https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585

listenallyall 3y ago
> Why spend time learning "easy" elements that give little points? Is this really a result of the current points system? Even in old-school, 6.0 human judging, wouldn't it be preferable to reward a much more difficult routine with higher scores? Is there any reason not to reward, say, a triple-triple combo better scores than a similar routine which tops out at a double-double?
- divan 3y ago
  Well, in the 6.0 system, however subjective and imperfect, the judges assessed the overall skater's level. One judge might have preferred complex jumps, while others might be more inclined toward edge control and skating skills. And there were no clearly defined metric to measure, which makes it more subjective, but also prone to Goodhart's effects. In a way it was similar to how classical music contests are judged. No one could claim that they can measure the performance, so they rely on the subjective and imperfect judgments of the humans who are considered experts in music. One judge might be consistently high-scoring, another might be consistently low-scoring, and they might look at the different aspects (technicality, musicality, etc.). They just try to capture their ranking (voting) distribution. In a way, the old system was more like voting. Even if the judge can't formally explain why s/he thinks skater A has better skating skills than B, the system would capture their preference. (And of course, this comes with all sort of biases – sequential bias, national bias, etc). But an important distinction is that in this system there is no proxy variable to optimize for. In the IJS it's very clear what to optimize for. So everyone is doing it. Also, one of the major changes, of course, was eliminating compulsory figures from competitions. Modern skaters don't even know they existed as a thing. Nowadays, everyone knows that you win a figure skating competition not by having Patrick Chan level skating, but by rotating your body in the air quickly – that is what gives points. So in modern clubs all over the world people start learning jumps in the first half of a year of practice. Which, ironically, leads to the lower probability of them actually mastering the jumps, because you still need insanely good edge control for take-offs and landings.
  - listenallyall 3y ago
    > there were no clearly defined metric to measure This is fine for an artistic competition, like classical music as you mention. But figure skating is a sport, it's in the Olympics, not having stated goals seems unfair to the competitors (athletes), and I can't think of any other sport which doesn't have very specific rules describing the criteria for winning.

by divan 3y ago

listenallyall 3y ago

> Why spend time learning "easy" elements that give little points? Is this really a result of the current points system? Even in old-school, 6.0 human judging, wouldn't it be preferable to reward a much more difficult routine with higher scores? Is there any reason not to reward, say, a triple-triple combo better scores than a similar routine which tops out at a double-double?

divan 3y ago
Well, in the 6.0 system, however subjective and imperfect, the judges assessed the overall skater's level. One judge might have preferred complex jumps, while others might be more inclined toward edge control and skating skills. And there were no clearly defined metric to measure, which makes it more subjective, but also prone to Goodhart's effects. In a way it was similar to how classical music contests are judged. No one could claim that they can measure the performance, so they rely on the subjective and imperfect judgments of the humans who are considered experts in music. One judge might be consistently high-scoring, another might be consistently low-scoring, and they might look at the different aspects (technicality, musicality, etc.). They just try to capture their ranking (voting) distribution. In a way, the old system was more like voting. Even if the judge can't formally explain why s/he thinks skater A has better skating skills than B, the system would capture their preference. (And of course, this comes with all sort of biases – sequential bias, national bias, etc). But an important distinction is that in this system there is no proxy variable to optimize for. In the IJS it's very clear what to optimize for. So everyone is doing it. Also, one of the major changes, of course, was eliminating compulsory figures from competitions. Modern skaters don't even know they existed as a thing. Nowadays, everyone knows that you win a figure skating competition not by having Patrick Chan level skating, but by rotating your body in the air quickly – that is what gives points. So in modern clubs all over the world people start learning jumps in the first half of a year of practice. Which, ironically, leads to the lower probability of them actually mastering the jumps, because you still need insanely good edge control for take-offs and landings.
- listenallyall 3y ago
  > there were no clearly defined metric to measure This is fine for an artistic competition, like classical music as you mention. But figure skating is a sport, it's in the Olympics, not having stated goals seems unfair to the competitors (athletes), and I can't think of any other sport which doesn't have very specific rules describing the criteria for winning.