4 ms·

As a professor at a community college, I am very hesitant to just declare something is AI generated unless is it very clear. If I suspect something is AI genera

As a professor at a community college, I am very hesitant to just declare something is AI generated unless is it very clear. If I suspect something is AI generated I run it through 3 different checkers that I found online and if they don't all agree then I don't call it plagiarism. Normally the writing is so vague that it isn't more than a C anyway.

natdempk 3y ago
You shouldn’t be using AI detectors. They aren’t accurate and other approaches to teaching need to be used instead of leaning on old methods: https://help.openai.com/en/articles/8313351-how-can-educators-respond-to-students-presenting-ai-generated-content-as-their-own https://help.openai.com/en/articles/8313351-how-can-educator...
- paulddraper 3y ago
  What are those "other approaches". Just in-person exams, right?
  - danShumway 3y ago
    It's frustrating that I can't know for certain who's going to win the 2024 election, but that doesn't mean I'd be justified in making a prediction based on a magic 8 ball. AI detectors don't work. If there aren't any other tools and methods that do detect AI, that genuinely stinks, it's a difficult problem. I definitely sympathize. But all the sympathizing in the world won't change the fact that AI detectors don't work. There are lots of unsolved problems in the world, and they don't justify putting weight behind non-solutions. I don't know how to cure Alzheimer's disease, but I know that a TV healer can't cure it, and if doctors started prescribing TV healers it would be medical malpractice. Similarly, I don't know how to tell if an essay was AI written... but neither do the AI detectors, and it is unethical for professors to rely on them to make decisions about plagiarism. When you have an unsolved problem and someone proposes a bad solution with bad consequences, what you end up with is the same unsolved problem, still unsolved, plus a newly created additional problem on top of it from your bad solution. You started with the problem of rampant plagiarism and cheating, now you have both a problem with rampant plagiarism and cheating, and a new problem with making baseless accusations of cheating against potentially honest students with no real evidence.
    - paulddraper 3y ago
      That is perhaps longest possible way I've ever seen to say "I don't know"
      - danShumway 3y ago
        If that is really your only takeaway from what I wrote then I apologize that the comment didn't elaborate more.
lostdog 3y ago
There are no accurate checkers for whether something was written by an AI.
BoorishBears 3y ago
This is a depressing comment, because it shows even someone who's aware these detectors are flawed is still willing to use them.
sigmar 3y ago
>3 different checkers that I found online and if they don't all agree then I don't call it plagiarism How do you verify these "different checkers" aren't all going to the same api or using the same software? How do you know they aren't using a similar heuristic method? What is the false positive rate of each? Have you seen the research that shows text from non-native speakers are more often falsely detected as being AI[1]? [1] https://twitter.com/emollick/status/1705057915625697712 https://twitter.com/emollick/status/1705057915625697712
Veen 3y ago
They don't work. If three services that don't work agree, it doesn't mean they are right. It just means whatever spurious metrics they use happen to align in that case.
danShumway 3y ago
"I used a magic 8 ball to predict the future." "They don't predict the future." "I know their predictions are statistically quite bad, so I used a magic 8 ball three times to make sure there wasn't any doubt." ---- How much allure for these products is there that a college professor can be on HN, know that the tools don't work, and still use them anyway? And how are students supposed to combat that kind of misuse if even educating professors about the problem isn't enough to make them stop trusting the detectors? And for a professor to then to brag about it on HN that they're being responsible as if they expect to be praised for it? It's just so frustrating to read... What do you do if you're a student in that situation, do you have to sit your professor down and not only show that the tools have problems but also explain the concept of correlated error? I kind of feel like the only way this changes is with legal challenges to colleges until they eventually ban the tools from being used in grading. Public pushback isn't going to change it, schools aren't going to care about that. And if educating about the problems isn't going to change behavior, I can't expect students to walk every one of their professors through a stats class. I don't know what if any legal rights at all students actually have in these situations, I'm guessing most colleges have boilerplate that if they decide to randomly fail you tough luck. But at the very least, @jccalhoun you should stop using these programs yourself; you have the individual ability to make your own classroom better, and you should.
sebzim4500 3y ago
So you know the detectors are worthless, and you are using them to fail students anyway? jfc education is dead

natdempk 3y ago

You shouldn’t be using AI detectors. They aren’t accurate and other approaches to teaching need to be used instead of leaning on old methods: https://help.openai.com/en/articles/8313351-how-can-educators-respond-to-students-presenting-ai-generated-content-as-their-own https://help.openai.com/en/articles/8313351-how-can-educator...

paulddraper 3y ago
What are those "other approaches". Just in-person exams, right?
- danShumway 3y ago
  It's frustrating that I can't know for certain who's going to win the 2024 election, but that doesn't mean I'd be justified in making a prediction based on a magic 8 ball. AI detectors don't work. If there aren't any other tools and methods that do detect AI, that genuinely stinks, it's a difficult problem. I definitely sympathize. But all the sympathizing in the world won't change the fact that AI detectors don't work. There are lots of unsolved problems in the world, and they don't justify putting weight behind non-solutions. I don't know how to cure Alzheimer's disease, but I know that a TV healer can't cure it, and if doctors started prescribing TV healers it would be medical malpractice. Similarly, I don't know how to tell if an essay was AI written... but neither do the AI detectors, and it is unethical for professors to rely on them to make decisions about plagiarism. When you have an unsolved problem and someone proposes a bad solution with bad consequences, what you end up with is the same unsolved problem, still unsolved, plus a newly created additional problem on top of it from your bad solution. You started with the problem of rampant plagiarism and cheating, now you have both a problem with rampant plagiarism and cheating, and a new problem with making baseless accusations of cheating against potentially honest students with no real evidence.
  - paulddraper 3y ago
    That is perhaps longest possible way I've ever seen to say "I don't know"
    - danShumway 3y ago
      If that is really your only takeaway from what I wrote then I apologize that the comment didn't elaborate more.

lostdog 3y ago

There are no accurate checkers for whether something was written by an AI.

BoorishBears 3y ago

This is a depressing comment, because it shows even someone who's aware these detectors are flawed is still willing to use them.

sigmar 3y ago

>3 different checkers that I found online and if they don't all agree then I don't call it plagiarism How do you verify these "different checkers" aren't all going to the same api or using the same software? How do you know they aren't using a similar heuristic method? What is the false positive rate of each? Have you seen the research that shows text from non-native speakers are more often falsely detected as being AI[1]? [1] https://twitter.com/emollick/status/1705057915625697712 https://twitter.com/emollick/status/1705057915625697712

Veen 3y ago

They don't work. If three services that don't work agree, it doesn't mean they are right. It just means whatever spurious metrics they use happen to align in that case.

danShumway 3y ago

"I used a magic 8 ball to predict the future." "They don't predict the future." "I know their predictions are statistically quite bad, so I used a magic 8 ball three times to make sure there wasn't any doubt." ---- How much allure for these products is there that a college professor can be on HN, know that the tools don't work, and still use them anyway? And how are students supposed to combat that kind of misuse if even educating professors about the problem isn't enough to make them stop trusting the detectors? And for a professor to then to brag about it on HN that they're being responsible as if they expect to be praised for it? It's just so frustrating to read... What do you do if you're a student in that situation, do you have to sit your professor down and not only show that the tools have problems but also explain the concept of correlated error? I kind of feel like the only way this changes is with legal challenges to colleges until they eventually ban the tools from being used in grading. Public pushback isn't going to change it, schools aren't going to care about that. And if educating about the problems isn't going to change behavior, I can't expect students to walk every one of their professors through a stats class. I don't know what if any legal rights at all students actually have in these situations, I'm guessing most colleges have boilerplate that if they decide to randomly fail you tough luck. But at the very least, @jccalhoun you should stop using these programs yourself; you have the individual ability to make your own classroom better, and you should.

sebzim4500 3y ago

So you know the detectors are worthless, and you are using them to fail students anyway? jfc education is dead