3 ms·

> At the self-described “heart” of the company’s monitoring software is Monitor AI, a “powerful artificial intelligence engine” that collects facial detection d

by pdkl95 6y ago

> At the self-described “heart” of the company’s monitoring software is Monitor AI, a “powerful artificial intelligence engine” that collects facial detection data [...] to identify “patterns and anomalies associated with cheating.” > "... people who have some sort of facial disfigurement have special challenges; they might get flagged because their face has an unexpected geometry.” So this company is extrapolating "patterns ... associated with cheating" from facial geometry. This is just phrenology laundered through their "powerful artificial intelligence engine" black box. Predicting behavior from the shape of someone's skull is still bullshit pseudoscience, even if the calipers[1] are replaced with a bunch of linear algebra. [1] http://antiquescientifica.com/phrenology_calipers_George_Combe_on_Wells_bust.jpg http://antiquescientifica.com/phrenology_calipers_George_Com...

Balgair 6y ago
> This is just phrenology ... It's probably worse. I'd bet that in addition to humans with non-'normal' heads, it would also flag the blind, deaf, sufferers of Tourette syndrome, certain muslim women, bearded men, humans of african descent, humans of asian descent, etc. > even if the calipers[1] are replaced with a bunch of linear algebra. I love this line. If your head does not fit the space of eigen-heads, throw an error.
- MengerSponge 6y ago
  Yup https://www.technologyreview.com/2020/08/07/1006132/software-algorithms-proctoring-online-tests-ai-ethics/ https://www.technologyreview.com/2020/08/07/1006132/software...
- AnimalMuppet 6y ago
  And the non-neurotypical.
throwaway5752 6y ago
It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers. It's not phrenology. It may not be effective, either, and I don't want to be put in a position of defending this company. But it shouldn't be interpreted as they analyzing if someone looks like a cheater.
- TheDong 6y ago
  It's still not far off. My dad had a pretty severe lazy eye, and I have no doubt eye tracking software would think that he was looking down at his desk constantly. I stare off into the distance (usually sideways) when I'm thinking. Those sorts of things are common and barely conscious. So, what's left for the AI? Trying to detect if people seem nervous? Well, I've got friends who have nervous twitches under normal conditions. What is normal behavior varies wildly person-to-person. Trying to figure out "is someone glancing somewhere before answering" vs "does someone nod to themselves when they think they got an answer in their head" doesn't sound like something AI can do. Figuring out "is this person cheating" from a recording of their face sounds pretty close to phrenology. Determining personality from skull-size, determining cheating from facial-motion... they both sound bogus, and like someone's trying to create a correlation that cannot account for the variety of human behaviors and shapes.
  - throwaway5752 6y ago
    I read the article. The specific cases mentioned involved missed instructions, interruptions where the test taker let their seat, or the test taker talking. This seems invasive, an invasion of privacy, and unreasonable. I object to the use of this software. But I really don't think they are doing or even being accused of anything like phrenology. I think it weakens the case against this type of software to make inaccurate accusations like that.
- heavyset_go 6y ago
  > It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers This assumes that the AI only picks up on patterns between eye movement and cheating, and not correlations between unrelated dimensions of data in the dataset that the model was trained on. Famously, such correlations created AI systems that resulted in disparate impact on legally protected classes in the US. Also, these systems are usually ill-equipped to handle anomalies in an accurate capacity.
- dghlsakjg 6y ago
  I (and a lot of people I know) would do pretty badly on a test that flagged me for staring into space while I think.

by pdkl95 6y ago

Balgair 6y ago

> This is just phrenology ... It's probably worse. I'd bet that in addition to humans with non-'normal' heads, it would also flag the blind, deaf, sufferers of Tourette syndrome, certain muslim women, bearded men, humans of african descent, humans of asian descent, etc. > even if the calipers[1] are replaced with a bunch of linear algebra. I love this line. If your head does not fit the space of eigen-heads, throw an error.

MengerSponge 6y ago
Yup https://www.technologyreview.com/2020/08/07/1006132/software-algorithms-proctoring-online-tests-ai-ethics/ https://www.technologyreview.com/2020/08/07/1006132/software...
AnimalMuppet 6y ago
And the non-neurotypical.

throwaway5752 6y ago

It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers. It's not phrenology. It may not be effective, either, and I don't want to be put in a position of defending this company. But it shouldn't be interpreted as they analyzing if someone looks like a cheater.

TheDong 6y ago
It's still not far off. My dad had a pretty severe lazy eye, and I have no doubt eye tracking software would think that he was looking down at his desk constantly. I stare off into the distance (usually sideways) when I'm thinking. Those sorts of things are common and barely conscious. So, what's left for the AI? Trying to detect if people seem nervous? Well, I've got friends who have nervous twitches under normal conditions. What is normal behavior varies wildly person-to-person. Trying to figure out "is someone glancing somewhere before answering" vs "does someone nod to themselves when they think they got an answer in their head" doesn't sound like something AI can do. Figuring out "is this person cheating" from a recording of their face sounds pretty close to phrenology. Determining personality from skull-size, determining cheating from facial-motion... they both sound bogus, and like someone's trying to create a correlation that cannot account for the variety of human behaviors and shapes.
- throwaway5752 6y ago
  I read the article. The specific cases mentioned involved missed instructions, interruptions where the test taker let their seat, or the test taker talking. This seems invasive, an invasion of privacy, and unreasonable. I object to the use of this software. But I really don't think they are doing or even being accused of anything like phrenology. I think it weakens the case against this type of software to make inaccurate accusations like that.
heavyset_go 6y ago
> It's clearly just looking for people looking at another computer or otherwise glancing elsewhere before entering answers This assumes that the AI only picks up on patterns between eye movement and cheating, and not correlations between unrelated dimensions of data in the dataset that the model was trained on. Famously, such correlations created AI systems that resulted in disparate impact on legally protected classes in the US. Also, these systems are usually ill-equipped to handle anomalies in an accurate capacity.
dghlsakjg 6y ago
I (and a lot of people I know) would do pretty badly on a test that flagged me for staring into space while I think.