4 ms·

The part that annoys me is that students apparently have no right to be told why the AI flagged their work. For any process where an computer is allowed to judg

The part that annoys me is that students apparently have no right to be told why the AI flagged their work. For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person. Now this would effectively kill off the current AI powered solution, because they have no way of explaining, or even understanding, why a paper may be plagiarized or not, but I'm okay with that.

ben_w 2y ago
> For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person. This is a big part of GDPR.
- ckastner 2y ago
  Indeed. Quoting article 22 [1]: > The data subject shall have the right not to be subject to a decision based solely on automated processing [...] [1]: https://gdpr.eu/article-22-automated-individual-decision-making/ https://gdpr.eu/article-22-automated-individual-decision-mak...
  - auggierose 2y ago
    So if an automated decision happens, and the reviewer looks for a second at it, and says, good enough, that will be OK according to GDPR. Don't see what GDPR solves here.
    - lucianbr 2y ago
      Well I guess the theory is that you could go to court, and the court would be reasonable and say "this 1 second look does not fulfill the requirement, you need to actually use human judgement and see what was going on there". Lots of discussions regarding FAANG malicious compliance have shown this is how high courts work in EU. When there is political will. But if you're a nobody, and can't afford to go to court against Deutsche Bank for example, of course you're SOL. EU has some good parts, but it's still a human government. It's especially problematic since a good chunk of those "flagged" are actually doing something nefarious, and both courts and government will consider that "mostly works" is a good outcome. One or ten unlucky citizens are just the way the world works, as long as it's not someone with money or power or fame.
      - auggierose 2y ago
        I don't see that even people with money and power can do anything here. It is like VAR. When has it ever happened that the referee goes to the screen, and does not follow the VAR recommendation? Never. That is how automated decision making will work as well, across the board.
    - ckastner 2y ago
      > So if an automated decision happens, and the reviewer looks for a second at it, and says, good enough, that will be OK according to GDPR. Don't see what GDPR solves here. The assumption is that a human review the conditions that led the automated system to make that decision. I think it would be trivial to argue in court that rubberstamping some scalar value that a deep neural net or whatever spit out does not pass that bar. It's still the automated system's decision, the human is just parroting it. Note that it's easier for the FAANGs to argue such a review has happened because they have massive amounts of heterogenous data where there's bound to be something that would be sufficient to argue with (like having posted something that offended someone). But a single score? I'd say almost impossible to argue. One would have to demonstrate that the system is near-perfect, and virtually never makes mistakes.
      - auggierose 2y ago
        Why a single score? The AI can generate a whole boilerplate of argumentation of why it made the decision. The reviewer will pretend to read and contemplate it, and then press the "OK" button.
        ben_w 2y ago
        There's a lot of "what can you get away with" in the world; but getting caught cheating like that is likely to go worse than getting caught using an LLM for a final exam.
        auggierose 2y ago
        My argument is there is no "getting caught" here. It is perfectly legal.
        ckastner 2y ago
        Fair point.
- mrweasel 2y ago
  I did not know that. Thank you. Reading the rules quickly, it does seem like you're not entitled to know why the computer flagged you, only that you have the right to "obtain human intervention". That seems a little to soft, I'd like to know under which rules exactly I'm being judged.
- 2rsf 2y ago
  And not less importantly the still young EU AI Act
sersi 2y ago
It's a similar problem to people being banned from Google (insert big company name) because of an automated fraud detection system that doesn't give any reason behind the ban. I also thing that there should be laws requiring a clear explanation whenever that happens.
- razakel 2y ago
  What about tipping off? Banks can't tell you that they've closed your account because of fraud or money laundering.
  - tonypace 2y ago
    They should have to tell you that. I can see why it's convenient for them not to, but I believe the larger point is far more important.
  - acdha 2y ago
    That doesn’t seem like a good comparison: it’s a far more serious crime, and while the bank won’t tell that they’re reporting your activity to the authorities the legal process absolutely will and in sensible countries you’re required to be given the opportunity to challenge the evidence. The problem being discussed here feels like it should be similar in that last regard: any time an automated system is making a serious decision they should be required to have an explanation and review process. If they don’t have sufficient evidence to back up the claim, they need to collect that evidence before making further accusations.
- tuetuopay 2y ago
  while it is infuriating, it's common for every place where fraud is an issue. if the company gave feedback, it would open the door to probing and know what is being watched or not. same reason as why a bank will not tell you why you got kicked off.
viraptor 2y ago
> kill off the current AI powered solution, because they have no way of explaining That's not correct. Some solution look at perplexity for specific models, some will look at ngram frequencies, and similar approaches. Almost all of those can produce a heatmap of "what looks suspicious". I wouldn't expect any of the detection systems to be like black boxes relying on LLM over the whole text.
- mrweasel 2y ago
  Sorry if this is "moving the goal post", but I wouldn't call looking at ngram frequencies for AI. Producing a heatmap doesn't tell you why something is suspicious, but it's obviously better than telling you nothing. In any case, if you where to use LLMs, or other black box solutions, you'd have to yank those out, if you where met with a requirement to explain why something is suspicious.
  - viraptor 2y ago
    It's literally the explanation. The only identification we have now is "this local part is often used by an AI model" and "this global structure is often used by an AI model". There's nothing more fancy about it. The heatmap would literally just point out "this part is suspiciously unlikely" - that's the explanation because that's the classification systems use.
- deleted 2y ago
  [deleted]
smartmic 2y ago
I agree with you, but I would go further and turn the tables. An AI should simply not be allowed to evaluate people, in any context whatsoever. For the simple reason that it has been proven not to work (and will also never). Anyone interested to learn more about it, I recommend the recent book "AI Snake Oil" from Arvind Narayanan and Sayash Kapoor [1]. It is a critical but nuanced book and helps to see the whole AI hype a little more clearly. [1] https://press.princeton.edu/books/hardcover/9780691249131/ai-snake-oil?_gl=1*cvkcpz*_up*MQ..*_ga*NjA4OTg1MjI1LjE3MjYwNDk3ODM.*_ga_N1W9JWKLY3*MTcyOTUwNTI2NS4yLjAuMTcyOTUwNTI2NS42MC4wLjEyMzk5NzMxODI https://press.princeton.edu/books/hardcover/9780691249131/ai....
- fullstackchris 2y ago
  I'm definitely no AI hypster, but saying anything will "never" work over an infinite timeline is a big statement... do you have grounds why some sort of AI system could one day "never" work at evaluating some metric about someone? Seems we have reliable systems already doing that in some areas (facial recognition at airport boarding, for example)
  - smartmic 2y ago
    Okay, let me try to be more precise. By "evaluate", I mean using an AI to make predictions about human behavior, either retrospectively (as is the case here in trying to make an accusation of cheating) or prospectively (i.e. automating criminal justice). Even if you could collect all the parameters (features?) that make up a human being, there is the randomness in humans and in nature in general, which simply destroys any ultimate prediction machine. Not to mention the edge cases we wander into. You can try to measure and average a human being, and you will get a certain accuracy well above 50%, but you will never cross the threshold of such high accuracy that a human being should be measured against, especially in life-deciding questions like career decisions or any social matters. Reliable systems in some areas? - Absolutely, and yes, even facial recognition. I agree, it works very well, but that is a different issue as it does not reveal or try to guess anything about the inner person. There are other problems that arise from the fact that it works so well (surveillance, etc.), but I did not mean that part of the equation.
iLoveOncall 2y ago
Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right?
- lcnPylGDnU4H9OF 2y ago
  I don’t think there’s anything to indicate they don’t understand this idea. But this misses the point; in their eyes, the lesser evil is to allow those with false positives to call the reasoning into question.
- baby_souffle 2y ago
  > Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right? This happens anyways, though? Any service that's useful for alternative / shady / illicit purposes is part of a cat/mouse game. Even if you don't tell the $badActors what you're looking for, they'll learn soon enough what you're not looking for just by virtue of their exploitative behavior still working. I'm a little skeptical of any "we fight bad guys!" effort that can be completely tanked by telling the bad guys how they got caught.
4star3star 2y ago
Totally agree. "Your paper is flagged for plagiarism. You get a zero." "But I swear I wrote that 100% on my own. What does it say I plagiarized?" "It doesn't say, but you still get a zero." In what world is this fair? Our court systems certainly don't operate under these assumptions.
kjkjadksj 2y ago
Thats how these tools mostly already work at least on the instructor side. They flag the problem text and will say where it came from. Its up to the teacher to do this due diligence and see if its a quote that merely got flagged or actual plagiarism.
floatrock 2y ago
Must be so demoralizing to be a kid these days. You use AI --> you're told you're cheating, which is immoral. You don't use AI --> you eventually get accused of using it or you get left behind by those who do use it. Figuring out who the hell you are in your high school years was hard enough when Kafka was only a reading assignment.
GJim 2y ago
> For any process where an computer is allowed to judge people.... GDPR to the rescue! https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/rights-related-to-automated-decision-making-including-profiling/?q=public+task https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-re... You must identify whether any of your processing falls under Article 22 [automated decision making, including AI] and, if so, make sure that you: * give individuals information about the processing; * introduce simple ways for them to request human intervention or challenge a decision; * carry out regular checks to make sure that your systems are working as intended. Why in gods name has the USA not adopted similar common sense legislation?

ben_w 2y ago

> For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person. This is a big part of GDPR.

ckastner 2y ago
Indeed. Quoting article 22 [1]: > The data subject shall have the right not to be subject to a decision based solely on automated processing [...] [1]: https://gdpr.eu/article-22-automated-individual-decision-making/ https://gdpr.eu/article-22-automated-individual-decision-mak...
- auggierose 2y ago
  So if an automated decision happens, and the reviewer looks for a second at it, and says, good enough, that will be OK according to GDPR. Don't see what GDPR solves here.
  - lucianbr 2y ago
    Well I guess the theory is that you could go to court, and the court would be reasonable and say "this 1 second look does not fulfill the requirement, you need to actually use human judgement and see what was going on there". Lots of discussions regarding FAANG malicious compliance have shown this is how high courts work in EU. When there is political will. But if you're a nobody, and can't afford to go to court against Deutsche Bank for example, of course you're SOL. EU has some good parts, but it's still a human government. It's especially problematic since a good chunk of those "flagged" are actually doing something nefarious, and both courts and government will consider that "mostly works" is a good outcome. One or ten unlucky citizens are just the way the world works, as long as it's not someone with money or power or fame.
    - auggierose 2y ago
      I don't see that even people with money and power can do anything here. It is like VAR. When has it ever happened that the referee goes to the screen, and does not follow the VAR recommendation? Never. That is how automated decision making will work as well, across the board.
  - ckastner 2y ago
    > So if an automated decision happens, and the reviewer looks for a second at it, and says, good enough, that will be OK according to GDPR. Don't see what GDPR solves here. The assumption is that a human review the conditions that led the automated system to make that decision. I think it would be trivial to argue in court that rubberstamping some scalar value that a deep neural net or whatever spit out does not pass that bar. It's still the automated system's decision, the human is just parroting it. Note that it's easier for the FAANGs to argue such a review has happened because they have massive amounts of heterogenous data where there's bound to be something that would be sufficient to argue with (like having posted something that offended someone). But a single score? I'd say almost impossible to argue. One would have to demonstrate that the system is near-perfect, and virtually never makes mistakes.
    - auggierose 2y ago
      Why a single score? The AI can generate a whole boilerplate of argumentation of why it made the decision. The reviewer will pretend to read and contemplate it, and then press the "OK" button.
      - ben_w 2y ago
        There's a lot of "what can you get away with" in the world; but getting caught cheating like that is likely to go worse than getting caught using an LLM for a final exam.
        auggierose 2y ago
        My argument is there is no "getting caught" here. It is perfectly legal.
      - ckastner 2y ago
        Fair point.
mrweasel 2y ago
I did not know that. Thank you. Reading the rules quickly, it does seem like you're not entitled to know why the computer flagged you, only that you have the right to "obtain human intervention". That seems a little to soft, I'd like to know under which rules exactly I'm being judged.
2rsf 2y ago
And not less importantly the still young EU AI Act

sersi 2y ago

It's a similar problem to people being banned from Google (insert big company name) because of an automated fraud detection system that doesn't give any reason behind the ban. I also thing that there should be laws requiring a clear explanation whenever that happens.

razakel 2y ago
What about tipping off? Banks can't tell you that they've closed your account because of fraud or money laundering.
- tonypace 2y ago
  They should have to tell you that. I can see why it's convenient for them not to, but I believe the larger point is far more important.
- acdha 2y ago
  That doesn’t seem like a good comparison: it’s a far more serious crime, and while the bank won’t tell that they’re reporting your activity to the authorities the legal process absolutely will and in sensible countries you’re required to be given the opportunity to challenge the evidence. The problem being discussed here feels like it should be similar in that last regard: any time an automated system is making a serious decision they should be required to have an explanation and review process. If they don’t have sufficient evidence to back up the claim, they need to collect that evidence before making further accusations.
tuetuopay 2y ago
while it is infuriating, it's common for every place where fraud is an issue. if the company gave feedback, it would open the door to probing and know what is being watched or not. same reason as why a bank will not tell you why you got kicked off.

viraptor 2y ago

> kill off the current AI powered solution, because they have no way of explaining That's not correct. Some solution look at perplexity for specific models, some will look at ngram frequencies, and similar approaches. Almost all of those can produce a heatmap of "what looks suspicious". I wouldn't expect any of the detection systems to be like black boxes relying on LLM over the whole text.

mrweasel 2y ago
Sorry if this is "moving the goal post", but I wouldn't call looking at ngram frequencies for AI. Producing a heatmap doesn't tell you why something is suspicious, but it's obviously better than telling you nothing. In any case, if you where to use LLMs, or other black box solutions, you'd have to yank those out, if you where met with a requirement to explain why something is suspicious.
- viraptor 2y ago
  It's literally the explanation. The only identification we have now is "this local part is often used by an AI model" and "this global structure is often used by an AI model". There's nothing more fancy about it. The heatmap would literally just point out "this part is suspiciously unlikely" - that's the explanation because that's the classification systems use.
deleted 2y ago
[deleted]

smartmic 2y ago

I agree with you, but I would go further and turn the tables. An AI should simply not be allowed to evaluate people, in any context whatsoever. For the simple reason that it has been proven not to work (and will also never). Anyone interested to learn more about it, I recommend the recent book "AI Snake Oil" from Arvind Narayanan and Sayash Kapoor [1]. It is a critical but nuanced book and helps to see the whole AI hype a little more clearly. [1] https://press.princeton.edu/books/hardcover/9780691249131/ai-snake-oil?_gl=1*cvkcpz*_up*MQ..*_ga*NjA4OTg1MjI1LjE3MjYwNDk3ODM.*_ga_N1W9JWKLY3*MTcyOTUwNTI2NS4yLjAuMTcyOTUwNTI2NS42MC4wLjEyMzk5NzMxODI https://press.princeton.edu/books/hardcover/9780691249131/ai....

fullstackchris 2y ago
I'm definitely no AI hypster, but saying anything will "never" work over an infinite timeline is a big statement... do you have grounds why some sort of AI system could one day "never" work at evaluating some metric about someone? Seems we have reliable systems already doing that in some areas (facial recognition at airport boarding, for example)
- smartmic 2y ago
  Okay, let me try to be more precise. By "evaluate", I mean using an AI to make predictions about human behavior, either retrospectively (as is the case here in trying to make an accusation of cheating) or prospectively (i.e. automating criminal justice). Even if you could collect all the parameters (features?) that make up a human being, there is the randomness in humans and in nature in general, which simply destroys any ultimate prediction machine. Not to mention the edge cases we wander into. You can try to measure and average a human being, and you will get a certain accuracy well above 50%, but you will never cross the threshold of such high accuracy that a human being should be measured against, especially in life-deciding questions like career decisions or any social matters. Reliable systems in some areas? - Absolutely, and yes, even facial recognition. I agree, it works very well, but that is a different issue as it does not reveal or try to guess anything about the inner person. There are other problems that arise from the fact that it works so well (surveillance, etc.), but I did not mean that part of the equation.

iLoveOncall 2y ago

Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right?

lcnPylGDnU4H9OF 2y ago
I don’t think there’s anything to indicate they don’t understand this idea. But this misses the point; in their eyes, the lesser evil is to allow those with false positives to call the reasoning into question.
baby_souffle 2y ago
> Surely you understand how any algorithm (regardless of its nature) that gives the cheater the list of reasons why it spotted cheating will only work for a single iteration before the cheaters adapt, right? This happens anyways, though? Any service that's useful for alternative / shady / illicit purposes is part of a cat/mouse game. Even if you don't tell the $badActors what you're looking for, they'll learn soon enough what you're not looking for just by virtue of their exploitative behavior still working. I'm a little skeptical of any "we fight bad guys!" effort that can be completely tanked by telling the bad guys how they got caught.

4star3star 2y ago

Totally agree. "Your paper is flagged for plagiarism. You get a zero." "But I swear I wrote that 100% on my own. What does it say I plagiarized?" "It doesn't say, but you still get a zero." In what world is this fair? Our court systems certainly don't operate under these assumptions.

kjkjadksj 2y ago

Thats how these tools mostly already work at least on the instructor side. They flag the problem text and will say where it came from. Its up to the teacher to do this due diligence and see if its a quote that merely got flagged or actual plagiarism.

floatrock 2y ago

Must be so demoralizing to be a kid these days. You use AI --> you're told you're cheating, which is immoral. You don't use AI --> you eventually get accused of using it or you get left behind by those who do use it. Figuring out who the hell you are in your high school years was hard enough when Kafka was only a reading assignment.

GJim 2y ago

> For any process where an computer is allowed to judge people.... GDPR to the rescue! https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/rights-related-to-automated-decision-making-including-profiling/?q=public+task https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-re... You must identify whether any of your processing falls under Article 22 [automated decision making, including AI] and, if so, make sure that you: * give individuals information about the processing; * introduce simple ways for them to request human intervention or challenge a decision; * carry out regular checks to make sure that your systems are working as intended. Why in gods name has the USA not adopted similar common sense legislation?