6 ms·

The ability to 'cheat' on written assignments has always been a possibility. With the rise of the Internet, it became easier to source it from more places - wit

by thadt 3y ago

The ability to 'cheat' on written assignments has always been a possibility. With the rise of the Internet, it became easier to source it from more places - with a counter balancing rise of plagiarism detectors. Now with Google, Grammarly and ChatGPT, it is trivial to copy/paste a pipeline with enough entropy to defeat many if not most detectors. I suspect the next generation of tools will be finish this cycle - we won't be able to build an arbitrary text distinguisher that works with acceptable accuracy. Educators have to adapt. If you want to "know what someone knows" - then ask them face to face, or have them write it down on paper. We used to call these tests, and they work pretty well. If you want someone to produce a piece of writing about something, how they produced it is less important than the result. If it is correct, clear, and gets the job done, then good. It's be the same for programming assignments - Copilot and ChatGPT make all un-proctored creation assignments a black box. You can evaluate the output, but it is nearly impossible to know what's in the box.

tnel77 3y ago
“But I’m a bad test taker!” I understand that everyone is different and that we should try to be accommodating overall, but it always cracked me up when select people could only thrive when they were at home with a take home test.
vehemenz 3y ago
It's not even a matter of adaptation in some cases. Written tests have been around for a long time, even during the era of word processors and computers. Whether instructors wanted to use has been a matter of pedagogy. Some of my philosophy exams 20+ years ago were written tests. You were given about twice as much time as you needed to write your arguments, which was fair to the slowest students. My CS courses for C programming were all written tests. You were expected to know the basic control structures and the standard libraries well enough to do this. To be fair, this was before Blackboard and other learning management systems, but there's no reason we can't go back to this system across the board.
whateveracct 3y ago
> The ability to 'cheat' on written assignments has always been a possibility. Right, and you have also always gotten a big fat F when you cheat.
- AnimalMuppet 3y ago
  When you got caught, which isn't the same thing.
- hammyhavoc 3y ago
  An `F` assumes it was actually issued a grade. Cheating usually results in disqualification and a lack of grade issued.
- nicenewtemp84 3y ago
  I used to pay a classmate to write my papers. He'd just write two different ones. I even had a classmate take a physics midterm for me, after taking his own earlier in the day. The fact that professors don't check the identity of the person handing a paper in vs the name on the test is insanity. But thanks for the good grade I guess.
  - dehrmann 3y ago
    All the freshman engineering classes at my college checked ids when turning in exams.
    - nicenewtemp84 3y ago
      Makes sense. 2003-2004 my specific UC system did not.
      - dehrmann 3y ago
        My college wasn't as selective as UCs, and it took pride in low GPA inflation and failing students in their freshman year.
- 3y ago
macleginn 3y ago
Seriously, though. Q to ChatGPT: ‘Was this text generated by ChatGPT? "The ability to 'cheat' on written assignments has always been a possibility. With the rise of the Internet, it became easier to source it from more places - with a counter balancing rise of plagiarism detectors. Now with Google, Grammarly and ChatGPT, it is trivial to copy/paste a pipeline with enough entropy to defeat many if not most detectors. I suspect the next generation of tools will be finish this cycle - we won't be able to build an arbitrary text distinguisher that works with acceptable accuracy."’ ChatGPT's answer: ‘Yes, the text you provided appears to have been generated by ChatGPT or a similar language model.’ Screenshot: https://imgur.com/jifRNE2 https://imgur.com/jifRNE2
- hgsgm 3y ago
  Downvote the witch!
  - hutzlibu 3y ago
    So not making any spelling errors is now equivalent to having red hair in the old times?
- qingdao99 3y ago
  It says the same for the comment you replied to. I don't think it's accurate for this task.
- Reason077 3y ago
  I just had ChatGPT write me a poem, and then in a new session, pasted the poem back in and asked it if it wrote it. The response? "No, I did not write that poem. I am an AI language model, and I generate responses based on patterns and information in the data I have been trained on. I don't have the capability to write original poems or claim authorship of any specific text."
  - hammyhavoc 3y ago
    This is the key thing with the limitations of LLMs at present, it can't "remember" what it has or hasn't done, by design.
thomasahle 3y ago
In the first exam I wrote, while doing a Postdoc at a university, I made a question with a really neat algorithm I'd thought of. I spent a lot of time making it super clean, ramping up to the algorithm using hints, etc. I was very happy when I read the first exam reply and the student had correctly solved the entire question! But then the second student had the same solution, and the third, and the fourth... Turns out somebody had written the exact same algorithm on stackoverflow, and I hadn't noticed. The students were allowed to use the internet, but they were also required to say what material they had used. Only one student actually listed the stackoverflow answer. My first instinct was to report all the students for cheating. But then, there were so many of them that I couldn't be bothered. I just changed the weighting of the question to 0.
falcolas 3y ago
> how they produced it is less important than the result A correct result includes how the result was reached. Thanks to patent and copyright laws, not all paths to a "right" answer are correct in the real world either. In college, this means that the professor's rules against ChatGPT need to be respected to create a "right" result.
nonethewiser 3y ago
> Educators have to adapt. If you want to "know what someone knows" - then ask them face to face, or have them write it down on paper. We used to call these tests, and they work pretty well. I actually dont think this is the right adaptation. The right adaptation is to design tests around the assumption that ChatGPT will be used.
bentcorner 3y ago
The very existence of "AI Detectors" imply that there is some set of text that is AI generated, some set of text that is human generated, there is no overlap, and it is possible to determine the set of text that is AI generated. And this doesn't even get into the fact that ChatGPT just lies about anything factual because of how it works.
2devnull 3y ago
Better yet! Use gpt and other ml to increase efficiency of assessments so that students can be measured more accurately, frequently and inexpensively than with human developed traditional paper/pencil items. Next, increase instructional efficiency by using gpt to tutor students individually. Lastly, remove the costly overhead of educational administration by automating the paperwork of school and district offices. But, like MOOCs, few of those great outcomes for learning will ever materialize because the industry more than educating will circle the wagons to protect it’s turf.
- drewcoo 3y ago
  Or just do blue book exams . . . https://en.wikipedia.org/wiki/Blue_book_exam https://en.wikipedia.org/wiki/Blue_book_exam
JohnFen 3y ago
> If you want someone to produce a piece of writing about something, how they produced it is less important than the result. I very much disagree with this in an educational setting. Part of the point of writing in the classroom is to learn how to write. How it's produced matters a great deal. Same with programming assignments. Using CoPilot and ChatGPT means that you're avoiding some of the very things that you're there to learn.
shalmanese 3y ago
> If you want someone to produce a piece of writing about something, how they produced it is less important than the result. If it is correct, clear, and gets the job done, then good. No, this fundamentally misunderstands the difference between work and pedagogy. With a work product, the end result is all that matters because it is a fundamentally new contribution for which the contribution is prized. If a marketer can get a 0.1% better conversion rate using ChatGPT, then that's the best course of action. Pedagogy is a deliberate simulacrum of work under artificial conditions where learning is done via the process. Nobody cares that you've built the 10 millionth version of tetris, the work product at the end of it is fundamentally uninteresting. The work product is solely there to force you to engage in the process and deliberately useless so it can be shaped towards maximal learning. Nobody mistakes, for example, that someone who uses a forklift to lift a 300lb bar or pays a friend to do it for them is getting their reps in at the gym because it's intrinsically obvious that the work of moving metal has no value but is instead, meant to induce an internal change in the athlete. Using ChatGPT on an assignment is no different from hiring a forklift to do your workout but people seem to want to get confused about it.
- jdenning 3y ago
  To play devils advocate: if, in the near future, AI is commonly used to produce the actual work product, then shouldn’t AI be part of this pedagogical practice?
  - shalmanese 3y ago
    The pedagogical process is whatever it says on the tin, same as how a sport is whatever the rules of the sport say it is. If the assignment is to use AI to do an X, then obviously AI is fine. Absent that, AI is cheating.
- koonsolo 3y ago
  Did you not read they also talked about tests? If you want to control the circumstances, you do a test. If you care about the output, you do an assignment. We had this problem for years already with calculators, this is no different. And in your case of lifting weights: the test is to lift or do x pullups. The military has been doing those TESTS for years. Or do you think they should give assignments instead?

by thadt 3y ago

tnel77 3y ago

“But I’m a bad test taker!” I understand that everyone is different and that we should try to be accommodating overall, but it always cracked me up when select people could only thrive when they were at home with a take home test.

vehemenz 3y ago

It's not even a matter of adaptation in some cases. Written tests have been around for a long time, even during the era of word processors and computers. Whether instructors wanted to use has been a matter of pedagogy. Some of my philosophy exams 20+ years ago were written tests. You were given about twice as much time as you needed to write your arguments, which was fair to the slowest students. My CS courses for C programming were all written tests. You were expected to know the basic control structures and the standard libraries well enough to do this. To be fair, this was before Blackboard and other learning management systems, but there's no reason we can't go back to this system across the board.

whateveracct 3y ago

> The ability to 'cheat' on written assignments has always been a possibility. Right, and you have also always gotten a big fat F when you cheat.

AnimalMuppet 3y ago
When you got caught, which isn't the same thing.
hammyhavoc 3y ago
An `F` assumes it was actually issued a grade. Cheating usually results in disqualification and a lack of grade issued.
nicenewtemp84 3y ago
I used to pay a classmate to write my papers. He'd just write two different ones. I even had a classmate take a physics midterm for me, after taking his own earlier in the day. The fact that professors don't check the identity of the person handing a paper in vs the name on the test is insanity. But thanks for the good grade I guess.
- dehrmann 3y ago
  All the freshman engineering classes at my college checked ids when turning in exams.
  - nicenewtemp84 3y ago
    Makes sense. 2003-2004 my specific UC system did not.
    - dehrmann 3y ago
      My college wasn't as selective as UCs, and it took pride in low GPA inflation and failing students in their freshman year.
3y ago

macleginn 3y ago

Seriously, though. Q to ChatGPT: ‘Was this text generated by ChatGPT? "The ability to 'cheat' on written assignments has always been a possibility. With the rise of the Internet, it became easier to source it from more places - with a counter balancing rise of plagiarism detectors. Now with Google, Grammarly and ChatGPT, it is trivial to copy/paste a pipeline with enough entropy to defeat many if not most detectors. I suspect the next generation of tools will be finish this cycle - we won't be able to build an arbitrary text distinguisher that works with acceptable accuracy."’ ChatGPT's answer: ‘Yes, the text you provided appears to have been generated by ChatGPT or a similar language model.’ Screenshot: https://imgur.com/jifRNE2 https://imgur.com/jifRNE2

hgsgm 3y ago
Downvote the witch!
- hutzlibu 3y ago
  So not making any spelling errors is now equivalent to having red hair in the old times?
qingdao99 3y ago
It says the same for the comment you replied to. I don't think it's accurate for this task.
Reason077 3y ago
I just had ChatGPT write me a poem, and then in a new session, pasted the poem back in and asked it if it wrote it. The response? "No, I did not write that poem. I am an AI language model, and I generate responses based on patterns and information in the data I have been trained on. I don't have the capability to write original poems or claim authorship of any specific text."
- hammyhavoc 3y ago
  This is the key thing with the limitations of LLMs at present, it can't "remember" what it has or hasn't done, by design.

thomasahle 3y ago

In the first exam I wrote, while doing a Postdoc at a university, I made a question with a really neat algorithm I'd thought of. I spent a lot of time making it super clean, ramping up to the algorithm using hints, etc. I was very happy when I read the first exam reply and the student had correctly solved the entire question! But then the second student had the same solution, and the third, and the fourth... Turns out somebody had written the exact same algorithm on stackoverflow, and I hadn't noticed. The students were allowed to use the internet, but they were also required to say what material they had used. Only one student actually listed the stackoverflow answer. My first instinct was to report all the students for cheating. But then, there were so many of them that I couldn't be bothered. I just changed the weighting of the question to 0.

falcolas 3y ago

> how they produced it is less important than the result A correct result includes how the result was reached. Thanks to patent and copyright laws, not all paths to a "right" answer are correct in the real world either. In college, this means that the professor's rules against ChatGPT need to be respected to create a "right" result.

nonethewiser 3y ago

> Educators have to adapt. If you want to "know what someone knows" - then ask them face to face, or have them write it down on paper. We used to call these tests, and they work pretty well. I actually dont think this is the right adaptation. The right adaptation is to design tests around the assumption that ChatGPT will be used.

bentcorner 3y ago

The very existence of "AI Detectors" imply that there is some set of text that is AI generated, some set of text that is human generated, there is no overlap, and it is possible to determine the set of text that is AI generated. And this doesn't even get into the fact that ChatGPT just lies about anything factual because of how it works.

2devnull 3y ago

Better yet! Use gpt and other ml to increase efficiency of assessments so that students can be measured more accurately, frequently and inexpensively than with human developed traditional paper/pencil items. Next, increase instructional efficiency by using gpt to tutor students individually. Lastly, remove the costly overhead of educational administration by automating the paperwork of school and district offices. But, like MOOCs, few of those great outcomes for learning will ever materialize because the industry more than educating will circle the wagons to protect it’s turf.

drewcoo 3y ago
Or just do blue book exams . . . https://en.wikipedia.org/wiki/Blue_book_exam https://en.wikipedia.org/wiki/Blue_book_exam

JohnFen 3y ago

> If you want someone to produce a piece of writing about something, how they produced it is less important than the result. I very much disagree with this in an educational setting. Part of the point of writing in the classroom is to learn how to write. How it's produced matters a great deal. Same with programming assignments. Using CoPilot and ChatGPT means that you're avoiding some of the very things that you're there to learn.

shalmanese 3y ago

> If you want someone to produce a piece of writing about something, how they produced it is less important than the result. If it is correct, clear, and gets the job done, then good. No, this fundamentally misunderstands the difference between work and pedagogy. With a work product, the end result is all that matters because it is a fundamentally new contribution for which the contribution is prized. If a marketer can get a 0.1% better conversion rate using ChatGPT, then that's the best course of action. Pedagogy is a deliberate simulacrum of work under artificial conditions where learning is done via the process. Nobody cares that you've built the 10 millionth version of tetris, the work product at the end of it is fundamentally uninteresting. The work product is solely there to force you to engage in the process and deliberately useless so it can be shaped towards maximal learning. Nobody mistakes, for example, that someone who uses a forklift to lift a 300lb bar or pays a friend to do it for them is getting their reps in at the gym because it's intrinsically obvious that the work of moving metal has no value but is instead, meant to induce an internal change in the athlete. Using ChatGPT on an assignment is no different from hiring a forklift to do your workout but people seem to want to get confused about it.

jdenning 3y ago
To play devils advocate: if, in the near future, AI is commonly used to produce the actual work product, then shouldn’t AI be part of this pedagogical practice?
- shalmanese 3y ago
  The pedagogical process is whatever it says on the tin, same as how a sport is whatever the rules of the sport say it is. If the assignment is to use AI to do an X, then obviously AI is fine. Absent that, AI is cheating.
koonsolo 3y ago
Did you not read they also talked about tests? If you want to control the circumstances, you do a test. If you care about the output, you do an assignment. We had this problem for years already with calculators, this is no different. And in your case of lifting weights: the test is to lift or do x pullups. The military has been doing those TESTS for years. Or do you think they should give assignments instead?