30 ms·

How AI is changing gymnastics judging

111 points by mjwhansen 3y ago

rasz 3y ago
>It became official in 2017, when the FIG formally announced its collaboration with Fujitsu Does FIG have powers of prosecution like UK Postal Office? Asking for a friend.
divan 3y ago
There is a deeper and largely overlooked problem with sports judging, especially in the sports with artistic component (like gymnastics or figure skating) – Goodhart's effects. Goodhart's effect stands for "when the metric becomes a target it stops being a good metric". As soon as you make measurement part of the reward system, it changes the whole system. Take figure skating as an example. Since 2006, ISU has introduced a similar judging system - IJS – that uses points for elements and an increasingly complex code of rules on how to assign those points. Each element has a so-called "base value" and judges determine GOE value in the range from -5 to +5 for each element. The computer averages that and takes trimmed mean to produce the final points for the element. Those base values and points are logically assigned by the complexity of elements. More complex elements cost more. Great skaters can perform more complex elements. Sounds logical, right? What has happened, though, is that using these points as a target has changed the way how skaters train. Why spend time learning "easy" elements that give little points? All time must be spent now learning "complex" elements that give more points. In a way, before, IJS skaters were chasing overall greatness; now, they chase points. "Skater's greatness" – whatever that means – is hard to define, let alone measure. The ability to perform complex jumps was a consequence of it, not a cause. Using points as a metric changes the whole system of training, and a new generation of skaters can perform insane quads (often because of pre-puberty body sizes, though), but no one would call them great skaters. I'm not even starting with how these points affected artistry. "Program components" (PCS) that should measure artistry are practically a joke now. You basically can multiply jump scores by some constant and get PCS. [1] All skating programs look the same now and are very predictable because of points maximization. Back to AI. The rules for figure skating become more and more complex each year in order to duct-tape the constantly emerging issues. Many judges admit that complexity is so high that the human brain can't even possibly apply them in real time and has to resort to cognitive shortcuts. There are some early flirtations with using AI for judging figure skating [2], but nothing serious yet. As outlined in the "Categorized Variants of Goodhart's Law" [3], one of the major reasons of this effect in sports is not just the fact that elements != greatness, but the fact that there is no shared definition of what is a True Goal of this sport. What exactly judging system try to measure? What would constitute the best skater? Where there is no explicitly stated True Goal, any proxy variable used as metrics (like "ability to perform complex elements") will be far from good and will be distorting the system even further. Eventually, this boils down to the structure and decision-making in the international federation that governs this sport. And these things cannot be solved by AI. In a way, using AI is a yak-shaving for ever-increasing rules of an inherently wrong judging system. [1] http://libjournals.unca.edu/OJS/index.php/mas/article/view/23 http://libjournals.unca.edu/OJS/index.php/mas/article/view/2... [2] https://www.youtube.com/watch?v=pq_mv1eyZyA https://www.youtube.com/watch?v=pq_mv1eyZyA [3] https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585
- listenallyall 3y ago
  > Why spend time learning "easy" elements that give little points? Is this really a result of the current points system? Even in old-school, 6.0 human judging, wouldn't it be preferable to reward a much more difficult routine with higher scores? Is there any reason not to reward, say, a triple-triple combo better scores than a similar routine which tops out at a double-double?
  - divan 3y ago
    Well, in the 6.0 system, however subjective and imperfect, the judges assessed the overall skater's level. One judge might have preferred complex jumps, while others might be more inclined toward edge control and skating skills. And there were no clearly defined metric to measure, which makes it more subjective, but also prone to Goodhart's effects. In a way it was similar to how classical music contests are judged. No one could claim that they can measure the performance, so they rely on the subjective and imperfect judgments of the humans who are considered experts in music. One judge might be consistently high-scoring, another might be consistently low-scoring, and they might look at the different aspects (technicality, musicality, etc.). They just try to capture their ranking (voting) distribution. In a way, the old system was more like voting. Even if the judge can't formally explain why s/he thinks skater A has better skating skills than B, the system would capture their preference. (And of course, this comes with all sort of biases – sequential bias, national bias, etc). But an important distinction is that in this system there is no proxy variable to optimize for. In the IJS it's very clear what to optimize for. So everyone is doing it. Also, one of the major changes, of course, was eliminating compulsory figures from competitions. Modern skaters don't even know they existed as a thing. Nowadays, everyone knows that you win a figure skating competition not by having Patrick Chan level skating, but by rotating your body in the air quickly – that is what gives points. So in modern clubs all over the world people start learning jumps in the first half of a year of practice. Which, ironically, leads to the lower probability of them actually mastering the jumps, because you still need insanely good edge control for take-offs and landings.
    - listenallyall 3y ago
iandanforth 3y ago
[flagged]
- deleted 3y ago
  [deleted]
feverzsj 3y ago
[flagged]
- deleted 3y ago
  [deleted]
ungreased0675 3y ago
“Gymnastics is a subjective sport, like diving or dressage, where factors like nationality, body type, the location of a judge’s chair, and vague concepts like “artistry” and “performance” affect scores.” Even though it’s a subjectively judged event, the scores somehow have three decimal places!
- deleted 3y ago
  [deleted]
- mjwhansen 3y ago
  There's a 200+ page Code of Points[1] that determines how much each skill is worth as well as each error. At the elite level, there's a panel of 9 judges for each event, divided into judges who tally the execution score and the difficulty score, and the reference panel that effectively audits the execution score.[2] The difficulty and execution scores are averaged. This is how you can get three-decimal scores due to averaging. Here is an example of a judge's score sheet.[2] (Nevertheless, there are still issues with bias, where athletes or teams who benefit from a particular judge's biases might have their errors more gently scored. For example, on the bars, gymnasts are required to hit a handstand position within a 10-degree range of vertical. But a judge who is impacted by a bias, or a certain viewing position, many not notice that a particular handstand was at 15 degrees, and not deduct accordingly. This is where the AI judging may be particularly helpful.) [1] https://www.gymnastics.sport/publicdir/rules/files/en_2022-2024%20WAG%20COP.pdf https://www.gymnastics.sport/publicdir/rules/files/en_2022-2... [2] https://www.youtube.com/watch?v=IYjhFnaoF90&t=408s https://www.youtube.com/watch?v=IYjhFnaoF90&t=408s [3] https://www.nbcolympics.com/news/gymnastics-101-scoring https://www.nbcolympics.com/news/gymnastics-101-scoring
  - Ferret7446 3y ago
    This whole affair seems rather silly IMO. For all the hullabaloo about AI killing art, I feel like this kind of fine grained scoring does more damage to art than AI ever will.
    - marcosdumay 3y ago
jpnc 3y ago
>JSS is not currently used to make determinations about artistry, and it still isn’t considered ready to use on certain complex routine components, like beam connections and dance elements on floor; the tech just isn’t there yet So at what point does a sensor of some kind that does some computation become 'AI'? Is this just marketing buzzword bandwagoning?
- deleted 3y ago
  [deleted]
- mjwhansen 3y ago
  A more precise term might be "computer vision," which is indeed a field of AI, but it is a less-known term even among a reasonably technical laypeople (such as might be reading a business publication like the MIT Tech Review).
- ben_w 3y ago
  > So at what point does a sensor of some kind that does some computation become 'AI'? By going backwards in time — I'd guess sometime between 2010 and 1995. https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
  - Kim_Bruning 3y ago
    I sometimes joke that at some point humans cease to be considered intelligent too.
world2vec 3y ago
>(...) says FIG’s Steve Butcher, who previously served as sport director for the organization and is now the president of the element-recognition working group for Fujitsu, the company that developed the AI." Oh boy, considering the Horizon debacle going on in the UK, I can expect a similar story in Olympic gymnastics in the near future...
- deleted 3y ago
  [deleted]
vander_elst 3y ago
Anyone aware of there are proposals to use computer vision for judging the technical part and the judges for the artistic part? On paper that should help fulfilling both schools of thought
Rushi_b 3y ago
[flagged]
t_mann 3y ago
It's not just the je ne sais quoi performance aspect that I'd be concerned about, it's something I'd call Wii-Sports-ification: the first time you try out movement-controlled games like they made the Wii console popular, you'll likely put a lot of effort into the movements as if you were wielding a real tennis racket or a golf club. After a while, you realize that the motion sensor is much more coarse, and you do much more efficient, bastardized versions of the original movements. While I acknowledge that this system is supposed to be much more sensitive, I'd expect a similar effect to be happening once athletes get their hands on those machines in training: they'll figure out how to best trigger the detection of each move, and how much flexing of the toes, bending in the back and overall effort is just enough for the machine, and it's going to change the look and feel of the sport.
- steveBK123 3y ago
  Exactly, its basically a variation of Goodhart's Law - “when a measure becomes a target, it ceases to be a good measure.”
  - Nashooo 3y ago
    Not really. It depends on the quality of the measure and any underlying assumptions. If the target is knowing who is the fastest then a simple stopwatch (measure) is 100% aligned with the target. So the measure becoming the target is completely fine. The problem comes when one (or multiple) measure(s) are an approximation of some ideal measure for the target. For example the famous example [1] of the measuring rat tails to determine killed rats (target). The faulty assumption here was that the only way to hand in a rat's tail is by killing them. [1] https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre
    - je42 3y ago
      Unfortunately not true. It is not a single metric. we have extra rules on top: No doping, separation of genders,...
    - IanCal 3y ago
aurareturn 3y ago
Where AI sports judging needs to happen fast is refereeing at the NBA and NFL level. Referees ruin games. Fans are turned off and always skeptical. If an AI can ref a game better, it'd be instant success for the leagues - not to mention getting rid of the unions who protect poorly performing referees. MLB is weird because everything needed to ref a baseball game does not need any advanced AI. MLB simply refuses to let robots ref games.
- smackay 3y ago
  Be careful what you ask for. FIFA's Video assistant referee was supposed to banish controversial decisions in soccer matches forever. Instead, they simply replaced one form of controversial decisions with another. https://en.wikipedia.org/wiki/Video_assistant_referee https://en.wikipedia.org/wiki/Video_assistant_referee
  - dagw 3y ago
    The NBA and NFL already have their own versions of VAR, with all the complaints and problems that the football version has.
    - deleted 3y ago
      [deleted]
  - dist-epoch 3y ago
    If people make mistakes watching the event for tens of seconds from multiple cameras with VAR, how could you trust the same decision made without it in a split second.
    - walthamstow 3y ago
      Top soccer refs get >96% of decisions right. That's enough for me. It's sport, it's not criminal justice or healthcare. I want the split second decision. It used to be that the ball would go into the net, you'd glance over at the linesman, see his flag down by his side, the ref indicates for a goal, the fans lose their minds. It would all happen in a second or two, now we have 4 minutes of video pedantry following every goal. This idea that soccer is a 'flowing' game with minimal stoppages is very important to fans.
femto 3y ago
[flagged]
nhinck3 3y ago
I'd be more than a little worried that AI might start picking up on the wrong signals (like uniform or skin colour) to assign points.
penjelly 3y ago
this is fine with me, but it shouldnt replace the existing panel, instead augment the panelists with it. It could analyze the rotations and timings much more closely and replay/study it at a more technical level.
TimSchumann 3y ago
https://archive.md/sJrYj https://archive.md/sJrYj
jerf 3y ago
This is a specific issue of a general problem of AI/computer judging, where humans think they're judging on specific criteria, and write it into the computer, without ever taking a moment to ask whether that's actually what they want. I use speeding in vehicles a lot. The point of a 55MPH speed limit isn't really that cars should be going exactly 55MPH at all times. It's an approximation of a complex situation that could handle only one number. But it's not actually a law of physics. Rigidly handing out tickets for people doing 56MPH is missing the point of speed limit. (As are many of the "You're doing X over" flashing signs, generally put in places that are precisely where the speed limit is too low and what people are doing is actually fine.) Now that we finally have the technology to determine whether a gymnast was or was not one millimeter over some line, the question should be raised, is that what we actually want to judge on? Because just because we humans thought we were judging on that, just because we thought that's what we wanted, is it really? It is this examination step that has been missing in every rush to insert computers into some judging system. We thoughtlessly reify accidents of history and what used to be easy to judge. At least in the case of gymnastics we're not going to send anyone to jail because of such thoughtlessness, but the sport could be destroyed. In the end, it still needs views, however abstracted the sports may seem to be from that, and after the novelty wears off, seeing humans out there competing for the most robotic and soulless performances runs the risk of gradually, but steadily, destroying all support for the sport. Now, personally, I'm not sure this wouldn't be a net gain for humanity in this particular place. But it's certainly not what the gymastics community as a whole wants and they really need to slow down and apply some thought to what they really want before they let someone reify that into code. They won't. Nobody else is either, for things far more important than this. But they should.
- fabioz 3y ago
  Particularly, I'd say that it would be awesome and could save sports which are graded based on such rules as the current state of affairs is pretty appalling (I hate seeing competitions which are graded based on subjective things and judge biases, for me it's the opposite -- such biases make seeing these sports maddening for me up to the point that I prefer not watching at all as I'm 100% sure it's unfair from the start).
- DonsDiscountGas 3y ago
maxglute 3y ago
Computer vision determining if a ball touches the line, or which body part crosses the line first seems fine. Judging artistic merits not so much. If this like 3d capturing the entire routine and analytically evaluating difficulting based on some biomechanical formula them maybe. Otherwise I'd rather biased judging be limited to people / geopolitics than whichever vendor writes the code. Or artistic elements need to weigh down / eliminated.
kbos87 3y ago
I feel like the obvious effect of this is that gymnastics becomes all about testing against and optimizing for an algorithm. That sounds sterile and lacking any of the romance (best word I can think of) that goes into physical competition. Then I remind myself that optimizing for the opinions of judges is the status quo. Judges that are susceptible to wild bias in all sorts of undetectable ways. This will change the sport for sure, but I can't say that I think it will be all bad.
fnordpiglet 3y ago
In discussing AI they lament bias. In discussing human judges they laud the fact nationality and body shape and the judges seat position plays a role. So, when a machine is biased it’s a problem but when nationalism, racism, etc is rooted in a human judge it’s a feature? Technology really shouldn’t be written about by journalism majors.
Kalanos 3y ago
great! now the gymnasts won't have to sleep w the judges, and the judges won't be subject to political pressure
ahaferburg 3y ago
How accurate is this? There's a video embedded in the article. https://www.youtube.com/watch?v=CinAYBZYANg https://www.youtube.com/watch?v=CinAYBZYANg To me it looks very shaky, e. g. at 0:20-0:25. It seems like they're not using an underlying physical model that takes momentum into account. Also it looks like the system is interpreting the movement of the ponytail as head movement.
kouru225 3y ago
I honestly feel like what AI is really doing is just clarifying what we’re actually doing. This whole article reminds me of my reaction when I first saw chatgpt write an essay. I immediately recognized that this essay would’ve been accepted by my teacher when I was a kid, and then remembered just how useless I thought essays were back when I had to write them. It feels like, with AI, I had proof that these essays were just “busy work” rather than anything else. As a student, I was just taking the prompt that my teacher gave me and expanding it into an essay, and there was no information in the essay that wasn’t already in the prompt itself. Why is it useful to take something that could be said in a single sentence and say it in 5 pages? I wasn’t encouraged to think about the book I was writing the essay about because my thoughts were seen as secondary to doing the technical work and simply copying the style and format of the essay, but now, with AI, that part had been automated and all that education was for nothing. It seems like Gymnastics really is, at it’s heart, a very technical act. Your goal is to achieve perfect normal angles for some reason. But as robots begin to dispel the haze around technical work, we stop being so interested in technicality. You can see it as an attack if you’re really invested in that technical stuff, but really you gotta ask yourself why you were so interested in it in the first place.
- mort96 3y ago
  Do you not believe it to be valuable to be able to write 5 pages of coherent text? Is that not something that schools should teach users how to do?

111 points by mjwhansen 3y ago

rasz 3y ago

>It became official in 2017, when the FIG formally announced its collaboration with Fujitsu Does FIG have powers of prosecution like UK Postal Office? Asking for a friend.

divan 3y ago

There is a deeper and largely overlooked problem with sports judging, especially in the sports with artistic component (like gymnastics or figure skating) – Goodhart's effects. Goodhart's effect stands for "when the metric becomes a target it stops being a good metric". As soon as you make measurement part of the reward system, it changes the whole system. Take figure skating as an example. Since 2006, ISU has introduced a similar judging system - IJS – that uses points for elements and an increasingly complex code of rules on how to assign those points. Each element has a so-called "base value" and judges determine GOE value in the range from -5 to +5 for each element. The computer averages that and takes trimmed mean to produce the final points for the element. Those base values and points are logically assigned by the complexity of elements. More complex elements cost more. Great skaters can perform more complex elements. Sounds logical, right? What has happened, though, is that using these points as a target has changed the way how skaters train. Why spend time learning "easy" elements that give little points? All time must be spent now learning "complex" elements that give more points. In a way, before, IJS skaters were chasing overall greatness; now, they chase points. "Skater's greatness" – whatever that means – is hard to define, let alone measure. The ability to perform complex jumps was a consequence of it, not a cause. Using points as a metric changes the whole system of training, and a new generation of skaters can perform insane quads (often because of pre-puberty body sizes, though), but no one would call them great skaters. I'm not even starting with how these points affected artistry. "Program components" (PCS) that should measure artistry are practically a joke now. You basically can multiply jump scores by some constant and get PCS. [1] All skating programs look the same now and are very predictable because of points maximization. Back to AI. The rules for figure skating become more and more complex each year in order to duct-tape the constantly emerging issues. Many judges admit that complexity is so high that the human brain can't even possibly apply them in real time and has to resort to cognitive shortcuts. There are some early flirtations with using AI for judging figure skating [2], but nothing serious yet. As outlined in the "Categorized Variants of Goodhart's Law" [3], one of the major reasons of this effect in sports is not just the fact that elements != greatness, but the fact that there is no shared definition of what is a True Goal of this sport. What exactly judging system try to measure? What would constitute the best skater? Where there is no explicitly stated True Goal, any proxy variable used as metrics (like "ability to perform complex elements") will be far from good and will be distorting the system even further. Eventually, this boils down to the structure and decision-making in the international federation that governs this sport. And these things cannot be solved by AI. In a way, using AI is a yak-shaving for ever-increasing rules of an inherently wrong judging system. [1] http://libjournals.unca.edu/OJS/index.php/mas/article/view/23 http://libjournals.unca.edu/OJS/index.php/mas/article/view/2... [2] https://www.youtube.com/watch?v=pq_mv1eyZyA https://www.youtube.com/watch?v=pq_mv1eyZyA [3] https://arxiv.org/abs/1803.04585 https://arxiv.org/abs/1803.04585

listenallyall 3y ago
> Why spend time learning "easy" elements that give little points? Is this really a result of the current points system? Even in old-school, 6.0 human judging, wouldn't it be preferable to reward a much more difficult routine with higher scores? Is there any reason not to reward, say, a triple-triple combo better scores than a similar routine which tops out at a double-double?
- divan 3y ago
  Well, in the 6.0 system, however subjective and imperfect, the judges assessed the overall skater's level. One judge might have preferred complex jumps, while others might be more inclined toward edge control and skating skills. And there were no clearly defined metric to measure, which makes it more subjective, but also prone to Goodhart's effects. In a way it was similar to how classical music contests are judged. No one could claim that they can measure the performance, so they rely on the subjective and imperfect judgments of the humans who are considered experts in music. One judge might be consistently high-scoring, another might be consistently low-scoring, and they might look at the different aspects (technicality, musicality, etc.). They just try to capture their ranking (voting) distribution. In a way, the old system was more like voting. Even if the judge can't formally explain why s/he thinks skater A has better skating skills than B, the system would capture their preference. (And of course, this comes with all sort of biases – sequential bias, national bias, etc). But an important distinction is that in this system there is no proxy variable to optimize for. In the IJS it's very clear what to optimize for. So everyone is doing it. Also, one of the major changes, of course, was eliminating compulsory figures from competitions. Modern skaters don't even know they existed as a thing. Nowadays, everyone knows that you win a figure skating competition not by having Patrick Chan level skating, but by rotating your body in the air quickly – that is what gives points. So in modern clubs all over the world people start learning jumps in the first half of a year of practice. Which, ironically, leads to the lower probability of them actually mastering the jumps, because you still need insanely good edge control for take-offs and landings.
  - listenallyall 3y ago

iandanforth 3y ago

[flagged]

deleted 3y ago
[deleted]

feverzsj 3y ago

[flagged]

deleted 3y ago
[deleted]

ungreased0675 3y ago

“Gymnastics is a subjective sport, like diving or dressage, where factors like nationality, body type, the location of a judge’s chair, and vague concepts like “artistry” and “performance” affect scores.” Even though it’s a subjectively judged event, the scores somehow have three decimal places!

deleted 3y ago
[deleted]
mjwhansen 3y ago
There's a 200+ page Code of Points[1] that determines how much each skill is worth as well as each error. At the elite level, there's a panel of 9 judges for each event, divided into judges who tally the execution score and the difficulty score, and the reference panel that effectively audits the execution score.[2] The difficulty and execution scores are averaged. This is how you can get three-decimal scores due to averaging. Here is an example of a judge's score sheet.[2] (Nevertheless, there are still issues with bias, where athletes or teams who benefit from a particular judge's biases might have their errors more gently scored. For example, on the bars, gymnasts are required to hit a handstand position within a 10-degree range of vertical. But a judge who is impacted by a bias, or a certain viewing position, many not notice that a particular handstand was at 15 degrees, and not deduct accordingly. This is where the AI judging may be particularly helpful.) [1] https://www.gymnastics.sport/publicdir/rules/files/en_2022-2024%20WAG%20COP.pdf https://www.gymnastics.sport/publicdir/rules/files/en_2022-2... [2] https://www.youtube.com/watch?v=IYjhFnaoF90&t=408s https://www.youtube.com/watch?v=IYjhFnaoF90&t=408s [3] https://www.nbcolympics.com/news/gymnastics-101-scoring https://www.nbcolympics.com/news/gymnastics-101-scoring
- Ferret7446 3y ago
  This whole affair seems rather silly IMO. For all the hullabaloo about AI killing art, I feel like this kind of fine grained scoring does more damage to art than AI ever will.
  - marcosdumay 3y ago

jpnc 3y ago

>JSS is not currently used to make determinations about artistry, and it still isn’t considered ready to use on certain complex routine components, like beam connections and dance elements on floor; the tech just isn’t there yet So at what point does a sensor of some kind that does some computation become 'AI'? Is this just marketing buzzword bandwagoning?

deleted 3y ago
[deleted]
mjwhansen 3y ago
A more precise term might be "computer vision," which is indeed a field of AI, but it is a less-known term even among a reasonably technical laypeople (such as might be reading a business publication like the MIT Tech Review).
ben_w 3y ago
> So at what point does a sensor of some kind that does some computation become 'AI'? By going backwards in time — I'd guess sometime between 2010 and 1995. https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- Kim_Bruning 3y ago
  I sometimes joke that at some point humans cease to be considered intelligent too.

world2vec 3y ago

>(...) says FIG’s Steve Butcher, who previously served as sport director for the organization and is now the president of the element-recognition working group for Fujitsu, the company that developed the AI." Oh boy, considering the Horizon debacle going on in the UK, I can expect a similar story in Olympic gymnastics in the near future...

deleted 3y ago
[deleted]

vander_elst 3y ago

Anyone aware of there are proposals to use computer vision for judging the technical part and the judges for the artistic part? On paper that should help fulfilling both schools of thought

Rushi_b 3y ago

[flagged]

t_mann 3y ago

It's not just the je ne sais quoi performance aspect that I'd be concerned about, it's something I'd call Wii-Sports-ification: the first time you try out movement-controlled games like they made the Wii console popular, you'll likely put a lot of effort into the movements as if you were wielding a real tennis racket or a golf club. After a while, you realize that the motion sensor is much more coarse, and you do much more efficient, bastardized versions of the original movements. While I acknowledge that this system is supposed to be much more sensitive, I'd expect a similar effect to be happening once athletes get their hands on those machines in training: they'll figure out how to best trigger the detection of each move, and how much flexing of the toes, bending in the back and overall effort is just enough for the machine, and it's going to change the look and feel of the sport.

steveBK123 3y ago
Exactly, its basically a variation of Goodhart's Law - “when a measure becomes a target, it ceases to be a good measure.”
- Nashooo 3y ago
  Not really. It depends on the quality of the measure and any underlying assumptions. If the target is knowing who is the fastest then a simple stopwatch (measure) is 100% aligned with the target. So the measure becoming the target is completely fine. The problem comes when one (or multiple) measure(s) are an approximation of some ideal measure for the target. For example the famous example [1] of the measuring rat tails to determine killed rats (target). The faulty assumption here was that the only way to hand in a rat's tail is by killing them. [1] https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre
  - je42 3y ago
    Unfortunately not true. It is not a single metric. we have extra rules on top: No doping, separation of genders,...
  - IanCal 3y ago

aurareturn 3y ago

Where AI sports judging needs to happen fast is refereeing at the NBA and NFL level. Referees ruin games. Fans are turned off and always skeptical. If an AI can ref a game better, it'd be instant success for the leagues - not to mention getting rid of the unions who protect poorly performing referees. MLB is weird because everything needed to ref a baseball game does not need any advanced AI. MLB simply refuses to let robots ref games.

smackay 3y ago
Be careful what you ask for. FIFA's Video assistant referee was supposed to banish controversial decisions in soccer matches forever. Instead, they simply replaced one form of controversial decisions with another. https://en.wikipedia.org/wiki/Video_assistant_referee https://en.wikipedia.org/wiki/Video_assistant_referee
- dagw 3y ago
  The NBA and NFL already have their own versions of VAR, with all the complaints and problems that the football version has.
  - deleted 3y ago
    [deleted]
- dist-epoch 3y ago
  If people make mistakes watching the event for tens of seconds from multiple cameras with VAR, how could you trust the same decision made without it in a split second.
  - walthamstow 3y ago
    Top soccer refs get >96% of decisions right. That's enough for me. It's sport, it's not criminal justice or healthcare. I want the split second decision. It used to be that the ball would go into the net, you'd glance over at the linesman, see his flag down by his side, the ref indicates for a goal, the fans lose their minds. It would all happen in a second or two, now we have 4 minutes of video pedantry following every goal. This idea that soccer is a 'flowing' game with minimal stoppages is very important to fans.

femto 3y ago

[flagged]

nhinck3 3y ago

I'd be more than a little worried that AI might start picking up on the wrong signals (like uniform or skin colour) to assign points.

penjelly 3y ago

this is fine with me, but it shouldnt replace the existing panel, instead augment the panelists with it. It could analyze the rotations and timings much more closely and replay/study it at a more technical level.

TimSchumann 3y ago

https://archive.md/sJrYj https://archive.md/sJrYj

jerf 3y ago

This is a specific issue of a general problem of AI/computer judging, where humans think they're judging on specific criteria, and write it into the computer, without ever taking a moment to ask whether that's actually what they want. I use speeding in vehicles a lot. The point of a 55MPH speed limit isn't really that cars should be going exactly 55MPH at all times. It's an approximation of a complex situation that could handle only one number. But it's not actually a law of physics. Rigidly handing out tickets for people doing 56MPH is missing the point of speed limit. (As are many of the "You're doing X over" flashing signs, generally put in places that are precisely where the speed limit is too low and what people are doing is actually fine.) Now that we finally have the technology to determine whether a gymnast was or was not one millimeter over some line, the question should be raised, is that what we actually want to judge on? Because just because we humans thought we were judging on that, just because we thought that's what we wanted, is it really? It is this examination step that has been missing in every rush to insert computers into some judging system. We thoughtlessly reify accidents of history and what used to be easy to judge. At least in the case of gymnastics we're not going to send anyone to jail because of such thoughtlessness, but the sport could be destroyed. In the end, it still needs views, however abstracted the sports may seem to be from that, and after the novelty wears off, seeing humans out there competing for the most robotic and soulless performances runs the risk of gradually, but steadily, destroying all support for the sport. Now, personally, I'm not sure this wouldn't be a net gain for humanity in this particular place. But it's certainly not what the gymastics community as a whole wants and they really need to slow down and apply some thought to what they really want before they let someone reify that into code. They won't. Nobody else is either, for things far more important than this. But they should.

fabioz 3y ago
Particularly, I'd say that it would be awesome and could save sports which are graded based on such rules as the current state of affairs is pretty appalling (I hate seeing competitions which are graded based on subjective things and judge biases, for me it's the opposite -- such biases make seeing these sports maddening for me up to the point that I prefer not watching at all as I'm 100% sure it's unfair from the start).
DonsDiscountGas 3y ago

maxglute 3y ago

Computer vision determining if a ball touches the line, or which body part crosses the line first seems fine. Judging artistic merits not so much. If this like 3d capturing the entire routine and analytically evaluating difficulting based on some biomechanical formula them maybe. Otherwise I'd rather biased judging be limited to people / geopolitics than whichever vendor writes the code. Or artistic elements need to weigh down / eliminated.

kbos87 3y ago

I feel like the obvious effect of this is that gymnastics becomes all about testing against and optimizing for an algorithm. That sounds sterile and lacking any of the romance (best word I can think of) that goes into physical competition. Then I remind myself that optimizing for the opinions of judges is the status quo. Judges that are susceptible to wild bias in all sorts of undetectable ways. This will change the sport for sure, but I can't say that I think it will be all bad.

fnordpiglet 3y ago

In discussing AI they lament bias. In discussing human judges they laud the fact nationality and body shape and the judges seat position plays a role. So, when a machine is biased it’s a problem but when nationalism, racism, etc is rooted in a human judge it’s a feature? Technology really shouldn’t be written about by journalism majors.

Kalanos 3y ago

great! now the gymnasts won't have to sleep w the judges, and the judges won't be subject to political pressure

ahaferburg 3y ago

How accurate is this? There's a video embedded in the article. https://www.youtube.com/watch?v=CinAYBZYANg https://www.youtube.com/watch?v=CinAYBZYANg To me it looks very shaky, e. g. at 0:20-0:25. It seems like they're not using an underlying physical model that takes momentum into account. Also it looks like the system is interpreting the movement of the ponytail as head movement.

kouru225 3y ago

I honestly feel like what AI is really doing is just clarifying what we’re actually doing. This whole article reminds me of my reaction when I first saw chatgpt write an essay. I immediately recognized that this essay would’ve been accepted by my teacher when I was a kid, and then remembered just how useless I thought essays were back when I had to write them. It feels like, with AI, I had proof that these essays were just “busy work” rather than anything else. As a student, I was just taking the prompt that my teacher gave me and expanding it into an essay, and there was no information in the essay that wasn’t already in the prompt itself. Why is it useful to take something that could be said in a single sentence and say it in 5 pages? I wasn’t encouraged to think about the book I was writing the essay about because my thoughts were seen as secondary to doing the technical work and simply copying the style and format of the essay, but now, with AI, that part had been automated and all that education was for nothing. It seems like Gymnastics really is, at it’s heart, a very technical act. Your goal is to achieve perfect normal angles for some reason. But as robots begin to dispel the haze around technical work, we stop being so interested in technicality. You can see it as an attack if you’re really invested in that technical stuff, but really you gotta ask yourself why you were so interested in it in the first place.

mort96 3y ago
Do you not believe it to be valuable to be able to write 5 pages of coherent text? Is that not something that schools should teach users how to do?