5 ms·

AI Detection Tools Falsely Accuse International Students of Cheating

66 points by jyunwai 3y ago

wilg 3y ago
Another reason these AI detection tools are snake oil.
noodlesUK 3y ago
There is (and always has been) a fairly effective way of detecting cheating. Have a conversation with a student. Ask them about some of their recent work. This can be in the form of something more formal like a viva, or it can be a casual chat. These conversations should not be graded, just flag for problems (of which cheating is just one). Combining this with verifying the student's ID is about as good as it gets. I think a mandatory 1-on-1 meeting with a member of staff at least once per term is essential, and serves many purposes. EDIT: I think oftentimes it's known which students are cheating, but it's not easy to do anything about it.
- usea 3y ago
  I've had teachers try this method with me and determined I was cheating when I was not. I don't believe in it.
  - JohnFen 3y ago
    This happened to me in secondary school, and I'm still mad about it.
  - grayhatter 3y ago
    It's not the method you shouldn't believe in, it's the teachers that are malicious. But yours is still an important point to make. The reason that all tests are standardized now, is in large part because of this reduction in bias. plenty of teachers are biased against students and will abuse said power, it's not a defect shared with multiple choice tests. It seems insane to me to return to something we know is bad just because a new calculator was invented. At least before we figure out how to fix it's known issues first.
    - JohnFen 3y ago
      > It's not the method you shouldn't believe in, it's the teachers that are malicious. That seems like leaping to a conclusion. Could it also be that this is not exactly an exact science and it's very easy to get it wrong?
- justrealist 3y ago
  When the student is ESL in the first place (and doing coursework via text), it's pretty hard to determine whether they are cheating or just struggling to communicate verbally.
- pessimizer 3y ago
  Personal attention costs money.
  - abeppu 3y ago
    ... but university students (and especially international students) are paying a lot of money. We're all (understandably) upset if we pay for something much more modest and when it has an issue we're unable to talk to a real human to try to resolve the issue. Imagine paying _many thousands per year_, working on projects and papers, being taught increasingly by adjuncts, struggling to register for all your required classes as administrators admit more students every year (without hiring more faculty), and then when you're accused by a bot of cheating, the response to a suggestion that you should talk with a staff person at least once per term is "oh that's too expensive".
    - rcbdev 3y ago
      Do you think your reasoning also holds up in the rest of western society where university is mostly tax-funded and needs to optimize for efficient use of funds?
      - abeppu 3y ago
        I won't pretend to know anything real about higher education in other countries. But if staff cannot meet once per term with students, then I wonder how effective they can be overall? If a student's interaction with staff is limited to sitting in lecture ... That sounds like a MOOC.
        rcbdev 3y ago
        I have studied in the US (big state college), a small Austrian private college / university of applied science and the University of Vienna (big public university) so maybe my take counts for something: Lecture sizes between European public universities and American universities are very different. While in the US few lectures were "MOOC-style" with most class sizes allowing for 1:1 interaction, in Austria you really generally are "just a number". (There is even some funny German comedy based around students abusing that the professor does not know the name or face to each student ID, many such lectures happen in one of our many so-called Audimax halls) This might be confusing to an American audience but works due to most classes in Europe having no required attendance and the grade being almost wholly determined by one exam at the end. One exception is the small Austrian / German private college (University of Applied Sciences) which limits access much more to allow for small class sizes. These usually have much stricter entrance criteria by necessity but once you're in you generally get a much more personalized experience like I had in the USA. As with everything, ymmv.
- deleted 3y ago
  [deleted]
- MengerSponge 3y ago
  How do you stop charismatic in-group students from getting reduced scrutiny? Biases are hard, and you have to be careful to avoid perpetuating them.
- godelski 3y ago
  As a teacher, I 100% agree with this. It's easy to have false positives and negatives though, so should never be used as more than in evidence building. Students can get nervous when put on the spot and some are smooth talkers. But it is a good start. > but it's not easy to do anything about it. This is actually prolific and a problem. I caught two students cheating once and their excuse was "I didn't know GitHub was on the internet." This was a junior level course... and the professor's focus was on networking... (he put in the syllabus that you can't use the internet for help. Let's put personal opinions aside (I wasn't checking them against SO fwiw)). The other student couldn't speak a lick on English. Yet, our department didn't want us to do anything about it. Despite being about as clear cut of a case as you can get, they even admitted it. In my other post I mention Goodhart's Hell. The reason the department didn't want to do anything is because the university evaluates a department's level of cheating by the number of reports that they make. So obviously, no one fucking reports it. But besides this, I think we have to ask why students cheat in the first place. It's because despite GPA not having meaningful outcome [0], they are critical for students so get into grad school or get their first job. No one knows that you took the hard professor, learned twice as much, but barely got a C which would easily be an A in the other course. So of course they cheat. I design my class around this actually. I make it easy to get a good grade on purpose, but get students to work. To do projects, explore, and be creative. If I don't, they just cheat and that undoes any benefit from a hard and challenging homework. I let them know too. But I also am purposefully late to return grades, which makes them sweat. It might sound dumb, but you got to adapt to the environment. My goal is to make students learn. I can't make someone choose to learn, and I don't bother with students that are checked out. But I do make my class engaging and highly encourage creativity, and I've found that a lot of students that previously were checked out end up checking in (I give often give them opportunities to make up their slack, but require more from them). And the students that do work hard, they have good projects in their portfolios which I've seen them use to get hired. Its dumb, but it is the only way I've found to re-align the a class with the intended goals: to teach students. [0] https://web.archive.org/web/20230107050342/https://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html https://web.archive.org/web/20230107050342/https://www.nytim...
- grayhatter 3y ago
  I'm not sure it's honest* to say it's known, which students are cheating. you shouldn't say you know someone is guilty of a malicious offense without clear evidence. Where, if you had clear evidence it wouldn't be hard to do something about. Or if you don't have clear evidence the most you should charitably consider is suspect. IMO, in a hypothetical like that, it's unethical to accuse someone at all. It's impossible for me to prove I didn't cheat. *: honest meaning in this specific context I think otherwise, without any accusation or assumption of maliciousness, you're probably right
mustafa_pasi 3y ago
Classifying people using AI is always going to lead to discrimination as long as there are differences between genders/ethnicities/socio-economic backgrounds. This is not preventable and you cannot do anything about it.
- gotoeleven 3y ago
  You can thumb the scale until you get the answer you want.
godelski 3y ago
This will always happen and will happen much more as we move forward. People specifically tune their models to sound human. People specifically use detectors to turn models, adversarially. I keep proclaiming that we live in Goodhart's Hell, and this is yet another great example. First, you have to remember that metrics are never a complete measure, but they are guides. This doesn't matter if you're cutting lumber or training a ML model. Metrics are __always__ proxies. For the vast majority of things you are intending to evaluate, you don't actually have a measurable function to express that. Every fucking measurement has error/noise, and this is why you measure twice and cut once. It is absolutely insane that in the age of advanced algorithms we've just tossed caution to the wind and we're instead doing a half assed measurement and then getting upset when our cuts are bad, blaming the ruler or blaming the saw. Second, you have to ask how closely your metric aligns with the thing you're actually trying to measure. Even cutting lumber you have to check that your measurement aligns. Accounting for angles and grain. We should be doing the same in algorithms, but with __much__ more care, but we don't for some reason. All metrics must be continually scrutinized. This is why we measure twice (or more), why we double check our calculations, and we do frequent sanity checks and build in redundancy, because you can't fucking trust your metrics. For some reason we seem to have this belief that empirical measurements are absolute and generalized. We seem to have this belief that noise is a thing that can be decoupled from the system and precisely removed. We seem to have this belief that measurements are unhackable. What the fuck is going on. This isn't just ML, it is academia, it is bureaucracy, it is the workplace, it is governments, it is economics. So I gotta ask, what the fuck is going on? In the past we treated things as hard, so there was an inherent uncertainty built into mental models but now we treat everything like they are simple to understand and a quick reading or video can make you an expert. We may have never had nuance from non-experts, but now everyone acts like an expert and that math is unbiased and objective. Why do we keep shooting ourselves in the foot and deciding the next best action is to reload the gun?
rondrabkin 3y ago
I worked for a big testing company. There are all sorts of cheating and many are easy to detect with the right tools. Maybe this is just the wrong tool.
tkgally 3y ago
I can give an example of why some English learners’ writing might seem to be ChatGPT-generated. (I have been involved with education and research on second-language academic writing in Japan for the past two decades.) When you ask ChatGPT to write an essay on a topic, it produces a series of well-formed paragraphs. Very often, many of the paragraphs begin with a transitional or signposting expression, such as “However,” “On the other hand,” “In addition,” “Conversely,” “Furthermore,” and, for the last paragraph, “In conclusion.” Books and websites that teach academic writing to second-language learners of English often explicitly tell the students to use such signposting. Some examples aimed at Japanese learners are at [1], [2], and [3]. While such paragraph-initial signposting is sometimes used in real-life English writing, it isn’t very common. You won’t find many newspaper op-ed essays or HN comments, for example, with final paragraphs beginning “In conclusion....” Since last December, whenever I see writing with a lot of paragraph-initial signposts I immediately suspect that the author has used ChatGPT. The detection software is probably looking for them, too. It’s important to remember that many learners of English have been taught to write that way. [1] https://etsjapan.jp/criterion/pdf/essaywordsphrases.pdf https://etsjapan.jp/criterion/pdf/essaywordsphrases.pdf [2] https://solo-ielts-toefl.com/link-paragraph/ https://solo-ielts-toefl.com/link-paragraph/ [3] http://unique-experience.xyz/?p=1129 http://unique-experience.xyz/?p=1129
- baybal2 3y ago
  The amount of academic publications outside of English native language countries have long surpassed the amount of publications in US+UK No wonder, these tools just gather word statistics from some large online publications repository.
radicality 3y ago
Hasn’t even OpenAI themselves turned off the feature/API which would try and tell the likelihood that a given text was AI generated, since it was too low accuracy? If even OpenAI thinks such a thing is too low accuracy, how can a third party service like TurnItIn think they can do a better job at it?
janalsncm 3y ago
I made this (non) tool as a public service, for specifically this purpose. https://isthiswrittenbyai.surge.sh/ https://isthiswrittenbyai.surge.sh/
LatteLazy 3y ago
The more I see of AI the more is just seems like automated, opaque, statistics. And while statistics are generally useful and efficient we also know the human side of "statistics tell us people who look like you are the problem"... I also think this is just one my symptom of a much bigger problem: big US institutions trying to automate human things. I was lucky to be part of a class of 60 (Exeter, UK, Physics). Core modules were a max of 60 people in the class and options ranged from 10 to 30. I knew basically all the professors. The concept of anticheat was anathema to us because people each other. If a student gave in top grade work but couldn't hold a conversation or pass an exam it would have been pretty obvious. This also likely discouraged people from trying. Now that's not possible in a class of 600...
constantcrying 3y ago
The concept of an "AI detector" for written works is absurd. The explicit modelling goal of ChatGPT is to write as humans do and it is extremely good at that. ChatGPT if not promted specifically has a very particular tone, which you can surely detect. But writing in that careful, somewhat formulaic english can not be taken as evidence of anything. I can absolutely believe that ESL students tend towards that tone. But that only shows how absolutely useless the detector is.

66 points by jyunwai 3y ago

wilg 3y ago

Another reason these AI detection tools are snake oil.

There is (and always has been) a fairly effective way of detecting cheating. Have a conversation with a student. Ask them about some of their recent work. This can be in the form of something more formal like a viva, or it can be a casual chat. These conversations should not be graded, just flag for problems (of which cheating is just one). Combining this with verifying the student's ID is about as good as it gets. I think a mandatory 1-on-1 meeting with a member of staff at least once per term is essential, and serves many purposes. EDIT: I think oftentimes it's known which students are cheating, but it's not easy to do anything about it.

usea 3y ago
I've had teachers try this method with me and determined I was cheating when I was not. I don't believe in it.
- JohnFen 3y ago
  This happened to me in secondary school, and I'm still mad about it.
- grayhatter 3y ago
  It's not the method you shouldn't believe in, it's the teachers that are malicious. But yours is still an important point to make. The reason that all tests are standardized now, is in large part because of this reduction in bias. plenty of teachers are biased against students and will abuse said power, it's not a defect shared with multiple choice tests. It seems insane to me to return to something we know is bad just because a new calculator was invented. At least before we figure out how to fix it's known issues first.
  - JohnFen 3y ago
    > It's not the method you shouldn't believe in, it's the teachers that are malicious. That seems like leaping to a conclusion. Could it also be that this is not exactly an exact science and it's very easy to get it wrong?
justrealist 3y ago
When the student is ESL in the first place (and doing coursework via text), it's pretty hard to determine whether they are cheating or just struggling to communicate verbally.
pessimizer 3y ago
Personal attention costs money.
- abeppu 3y ago
  ... but university students (and especially international students) are paying a lot of money. We're all (understandably) upset if we pay for something much more modest and when it has an issue we're unable to talk to a real human to try to resolve the issue. Imagine paying _many thousands per year_, working on projects and papers, being taught increasingly by adjuncts, struggling to register for all your required classes as administrators admit more students every year (without hiring more faculty), and then when you're accused by a bot of cheating, the response to a suggestion that you should talk with a staff person at least once per term is "oh that's too expensive".
  - rcbdev 3y ago
    Do you think your reasoning also holds up in the rest of western society where university is mostly tax-funded and needs to optimize for efficient use of funds?
    - abeppu 3y ago
      I won't pretend to know anything real about higher education in other countries. But if staff cannot meet once per term with students, then I wonder how effective they can be overall? If a student's interaction with staff is limited to sitting in lecture ... That sounds like a MOOC.
      - rcbdev 3y ago
        I have studied in the US (big state college), a small Austrian private college / university of applied science and the University of Vienna (big public university) so maybe my take counts for something: Lecture sizes between European public universities and American universities are very different. While in the US few lectures were "MOOC-style" with most class sizes allowing for 1:1 interaction, in Austria you really generally are "just a number". (There is even some funny German comedy based around students abusing that the professor does not know the name or face to each student ID, many such lectures happen in one of our many so-called Audimax halls) This might be confusing to an American audience but works due to most classes in Europe having no required attendance and the grade being almost wholly determined by one exam at the end. One exception is the small Austrian / German private college (University of Applied Sciences) which limits access much more to allow for small class sizes. These usually have much stricter entrance criteria by necessity but once you're in you generally get a much more personalized experience like I had in the USA. As with everything, ymmv.
deleted 3y ago
[deleted]
MengerSponge 3y ago
How do you stop charismatic in-group students from getting reduced scrutiny? Biases are hard, and you have to be careful to avoid perpetuating them.
godelski 3y ago
As a teacher, I 100% agree with this. It's easy to have false positives and negatives though, so should never be used as more than in evidence building. Students can get nervous when put on the spot and some are smooth talkers. But it is a good start. > but it's not easy to do anything about it. This is actually prolific and a problem. I caught two students cheating once and their excuse was "I didn't know GitHub was on the internet." This was a junior level course... and the professor's focus was on networking... (he put in the syllabus that you can't use the internet for help. Let's put personal opinions aside (I wasn't checking them against SO fwiw)). The other student couldn't speak a lick on English. Yet, our department didn't want us to do anything about it. Despite being about as clear cut of a case as you can get, they even admitted it. In my other post I mention Goodhart's Hell. The reason the department didn't want to do anything is because the university evaluates a department's level of cheating by the number of reports that they make. So obviously, no one fucking reports it. But besides this, I think we have to ask why students cheat in the first place. It's because despite GPA not having meaningful outcome [0], they are critical for students so get into grad school or get their first job. No one knows that you took the hard professor, learned twice as much, but barely got a C which would easily be an A in the other course. So of course they cheat. I design my class around this actually. I make it easy to get a good grade on purpose, but get students to work. To do projects, explore, and be creative. If I don't, they just cheat and that undoes any benefit from a hard and challenging homework. I let them know too. But I also am purposefully late to return grades, which makes them sweat. It might sound dumb, but you got to adapt to the environment. My goal is to make students learn. I can't make someone choose to learn, and I don't bother with students that are checked out. But I do make my class engaging and highly encourage creativity, and I've found that a lot of students that previously were checked out end up checking in (I give often give them opportunities to make up their slack, but require more from them). And the students that do work hard, they have good projects in their portfolios which I've seen them use to get hired. Its dumb, but it is the only way I've found to re-align the a class with the intended goals: to teach students. [0] https://web.archive.org/web/20230107050342/https://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html https://web.archive.org/web/20230107050342/https://www.nytim...
grayhatter 3y ago
I'm not sure it's honest* to say it's known, which students are cheating. you shouldn't say you know someone is guilty of a malicious offense without clear evidence. Where, if you had clear evidence it wouldn't be hard to do something about. Or if you don't have clear evidence the most you should charitably consider is suspect. IMO, in a hypothetical like that, it's unethical to accuse someone at all. It's impossible for me to prove I didn't cheat. *: honest meaning in this specific context I think otherwise, without any accusation or assumption of maliciousness, you're probably right

mustafa_pasi 3y ago

Classifying people using AI is always going to lead to discrimination as long as there are differences between genders/ethnicities/socio-economic backgrounds. This is not preventable and you cannot do anything about it.

gotoeleven 3y ago
You can thumb the scale until you get the answer you want.

godelski 3y ago

This will always happen and will happen much more as we move forward. People specifically tune their models to sound human. People specifically use detectors to turn models, adversarially. I keep proclaiming that we live in Goodhart's Hell, and this is yet another great example. First, you have to remember that metrics are never a complete measure, but they are guides. This doesn't matter if you're cutting lumber or training a ML model. Metrics are __always__ proxies. For the vast majority of things you are intending to evaluate, you don't actually have a measurable function to express that. Every fucking measurement has error/noise, and this is why you measure twice and cut once. It is absolutely insane that in the age of advanced algorithms we've just tossed caution to the wind and we're instead doing a half assed measurement and then getting upset when our cuts are bad, blaming the ruler or blaming the saw. Second, you have to ask how closely your metric aligns with the thing you're actually trying to measure. Even cutting lumber you have to check that your measurement aligns. Accounting for angles and grain. We should be doing the same in algorithms, but with __much__ more care, but we don't for some reason. All metrics must be continually scrutinized. This is why we measure twice (or more), why we double check our calculations, and we do frequent sanity checks and build in redundancy, because you can't fucking trust your metrics. For some reason we seem to have this belief that empirical measurements are absolute and generalized. We seem to have this belief that noise is a thing that can be decoupled from the system and precisely removed. We seem to have this belief that measurements are unhackable. What the fuck is going on. This isn't just ML, it is academia, it is bureaucracy, it is the workplace, it is governments, it is economics. So I gotta ask, what the fuck is going on? In the past we treated things as hard, so there was an inherent uncertainty built into mental models but now we treat everything like they are simple to understand and a quick reading or video can make you an expert. We may have never had nuance from non-experts, but now everyone acts like an expert and that math is unbiased and objective. Why do we keep shooting ourselves in the foot and deciding the next best action is to reload the gun?

rondrabkin 3y ago

I worked for a big testing company. There are all sorts of cheating and many are easy to detect with the right tools. Maybe this is just the wrong tool.

tkgally 3y ago

I can give an example of why some English learners’ writing might seem to be ChatGPT-generated. (I have been involved with education and research on second-language academic writing in Japan for the past two decades.) When you ask ChatGPT to write an essay on a topic, it produces a series of well-formed paragraphs. Very often, many of the paragraphs begin with a transitional or signposting expression, such as “However,” “On the other hand,” “In addition,” “Conversely,” “Furthermore,” and, for the last paragraph, “In conclusion.” Books and websites that teach academic writing to second-language learners of English often explicitly tell the students to use such signposting. Some examples aimed at Japanese learners are at [1], [2], and [3]. While such paragraph-initial signposting is sometimes used in real-life English writing, it isn’t very common. You won’t find many newspaper op-ed essays or HN comments, for example, with final paragraphs beginning “In conclusion....” Since last December, whenever I see writing with a lot of paragraph-initial signposts I immediately suspect that the author has used ChatGPT. The detection software is probably looking for them, too. It’s important to remember that many learners of English have been taught to write that way. [1] https://etsjapan.jp/criterion/pdf/essaywordsphrases.pdf https://etsjapan.jp/criterion/pdf/essaywordsphrases.pdf [2] https://solo-ielts-toefl.com/link-paragraph/ https://solo-ielts-toefl.com/link-paragraph/ [3] http://unique-experience.xyz/?p=1129 http://unique-experience.xyz/?p=1129

baybal2 3y ago
The amount of academic publications outside of English native language countries have long surpassed the amount of publications in US+UK No wonder, these tools just gather word statistics from some large online publications repository.

radicality 3y ago

Hasn’t even OpenAI themselves turned off the feature/API which would try and tell the likelihood that a given text was AI generated, since it was too low accuracy? If even OpenAI thinks such a thing is too low accuracy, how can a third party service like TurnItIn think they can do a better job at it?

janalsncm 3y ago

I made this (non) tool as a public service, for specifically this purpose. https://isthiswrittenbyai.surge.sh/ https://isthiswrittenbyai.surge.sh/

LatteLazy 3y ago

The more I see of AI the more is just seems like automated, opaque, statistics. And while statistics are generally useful and efficient we also know the human side of "statistics tell us people who look like you are the problem"... I also think this is just one my symptom of a much bigger problem: big US institutions trying to automate human things. I was lucky to be part of a class of 60 (Exeter, UK, Physics). Core modules were a max of 60 people in the class and options ranged from 10 to 30. I knew basically all the professors. The concept of anticheat was anathema to us because people each other. If a student gave in top grade work but couldn't hold a conversation or pass an exam it would have been pretty obvious. This also likely discouraged people from trying. Now that's not possible in a class of 600...

constantcrying 3y ago

The concept of an "AI detector" for written works is absurd. The explicit modelling goal of ChatGPT is to write as humans do and it is extremely good at that. ChatGPT if not promted specifically has a very particular tone, which you can surely detect. But writing in that careful, somewhat formulaic english can not be taken as evidence of anything. I can absolutely believe that ESL students tend towards that tone. But that only shows how absolutely useless the detector is.