4 ms·

I recently had to check code from some of my students at the university as I suspected plagiarism. I discovered JPlag which works like a charm and generates nic

I recently had to check code from some of my students at the university as I suspected plagiarism. I discovered JPlag which works like a charm and generates nice reports

jaimex2 2y ago
Kinda rare these days with ChatGPT
- i_am_proteus 2y ago
  You might be surprised. Many students who use ChatGPT for assignments end up turning in code identical (or nearly identical) to other students who use ChatGPT.
  - hatmatrix 2y ago
    Surprising because you get different answers each time you ask ChatGPT.
    - mmcwilliams 2y ago
      Different in an exact string match but code that is copied and pasted from ChatGPT has a lot of similarities in the way that it is (over) commented. I've seen a lot of Python where the student who "authored" it cannot tell me how a method works or why it was implemented despite having the comments prefixed to every line in the file.
      - Joker_vD 2y ago
        Ah yes, good old "Did you even read the essay before handing it in? Next time, please do."
      - elashri 2y ago
        > (over) commented From my experience using ChatGPT, It usually remove most of my already written comments when I ask questions about code I wrote myself. It usually give you outline comments. So unless you are supporter of the self documented code idea, I don't think ChatGPT over comments.
        mmcwilliams 2y ago
        It's obviously down to taste, but what I've seen over and over is a comment per line which to me is excessive outside it being requested of absolute beginners. That happens and also the model can't decide if it wants the comment on the line before the code or if everything should be appended to the line itself so when I see both styles within a single project it's another signal. People generally have a style that they stick with.
    - pdntspa 2y ago
      ChatGPT answers don't differ that much without being prompted to do so
      - serf 2y ago
        yeah but the prompt itself generally adds sufficient randomness to avoid the same verbatim answer each time. as an example just go ask it to write any sufficiently average function. use different names and phrases for what the function should do; you'll generally get a different flavor of answer each time, even if the functions all output the same thing. sometimes the prompt even forces the thing to output the most naive implementation possible due to the ordering or perceived priority of things within the requesting prompt. it's fun to use as a tool to nudge it into what you want once you get the hang of the preconceptions it falls into.
- franga2000 2y ago
  MOSS seems to be pretty good finding multiple people using LLM-generated code and flagging them as copies of each other. I imagine it would also be a good idea to throw the assignment text into the few most popular LLMs and feed that in as well, but I don't know of anyone who has tried this.
  - emeryberger 2y ago
    FWIW the attack we describe in the paper works against MOSS, too (that was the original inspiration for the name, “Mossad”).
BossingAround 2y ago
How do you deal with disputes? One's code is flagged even if the student in question didn't actually cheat. What then? Do you trust tools over the students' word? In addition, do things like stack overflow and using LLM-generated code count as cheating? Because that is horrible in and of itself, though a separate concern.
- deleted 2y ago
  [deleted]
- ablob 2y ago
  If you talk about the written code to the student in question it should become clear whether it was copied or not.
- ActualTeacher 2y ago
  The output of plagiarism tools should only serve as a hint to look at a pair of solutions more closely. All judgement should be derived entirely from similarities between solutions and not some artificial similarity score computed by some program.
  - ziddoap 2y ago
    Unfortunately, this is not really what happens in my experience. The output of plagiarism tools is taken as fact (especially at high school levels). Without extraordinary evidence of the tool being incorrect, students have no recourse, even if they could sit and explain the thought process behind every word/line of code/whatever.
    - deleted 2y ago
      [deleted]
    - nsajko 2y ago
      Lousy high school.
  - drapado 2y ago
    Indeed, this is exactly what I did.
beeboobaa3 2y ago
Next time just ask them a few questions about the programming choices they made. Far easier.
thi341 2y ago
An archival video of all coding sessions (locally, hosted by the student), starting with a visible outline of pseudo-code and ending with debugging should be sufficient. In case of a false positive from a faulty detector this is extraordinary evidence.
- wildzzz 2y ago
  We had a professor require us to use git as a timestamped log of our progress. Of course you could fake it but stealing work and basically redoing it piece by piece with fake timestamps is a lot of work for cheaters.

I recently had to check code from some of my students at the university as I suspected plagiarism. I discovered JPlag which works like a charm and generates nice reports

jaimex2 2y ago

Kinda rare these days with ChatGPT

i_am_proteus 2y ago
You might be surprised. Many students who use ChatGPT for assignments end up turning in code identical (or nearly identical) to other students who use ChatGPT.
- hatmatrix 2y ago
  Surprising because you get different answers each time you ask ChatGPT.
  - mmcwilliams 2y ago
    Different in an exact string match but code that is copied and pasted from ChatGPT has a lot of similarities in the way that it is (over) commented. I've seen a lot of Python where the student who "authored" it cannot tell me how a method works or why it was implemented despite having the comments prefixed to every line in the file.
    - Joker_vD 2y ago
      Ah yes, good old "Did you even read the essay before handing it in? Next time, please do."
    - elashri 2y ago
      > (over) commented From my experience using ChatGPT, It usually remove most of my already written comments when I ask questions about code I wrote myself. It usually give you outline comments. So unless you are supporter of the self documented code idea, I don't think ChatGPT over comments.
      - mmcwilliams 2y ago
        It's obviously down to taste, but what I've seen over and over is a comment per line which to me is excessive outside it being requested of absolute beginners. That happens and also the model can't decide if it wants the comment on the line before the code or if everything should be appended to the line itself so when I see both styles within a single project it's another signal. People generally have a style that they stick with.
  - pdntspa 2y ago
    ChatGPT answers don't differ that much without being prompted to do so
    - serf 2y ago
      yeah but the prompt itself generally adds sufficient randomness to avoid the same verbatim answer each time. as an example just go ask it to write any sufficiently average function. use different names and phrases for what the function should do; you'll generally get a different flavor of answer each time, even if the functions all output the same thing. sometimes the prompt even forces the thing to output the most naive implementation possible due to the ordering or perceived priority of things within the requesting prompt. it's fun to use as a tool to nudge it into what you want once you get the hang of the preconceptions it falls into.
franga2000 2y ago
MOSS seems to be pretty good finding multiple people using LLM-generated code and flagging them as copies of each other. I imagine it would also be a good idea to throw the assignment text into the few most popular LLMs and feed that in as well, but I don't know of anyone who has tried this.
- emeryberger 2y ago
  FWIW the attack we describe in the paper works against MOSS, too (that was the original inspiration for the name, “Mossad”).

BossingAround 2y ago

How do you deal with disputes? One's code is flagged even if the student in question didn't actually cheat. What then? Do you trust tools over the students' word? In addition, do things like stack overflow and using LLM-generated code count as cheating? Because that is horrible in and of itself, though a separate concern.

deleted 2y ago
[deleted]
ablob 2y ago
If you talk about the written code to the student in question it should become clear whether it was copied or not.
ActualTeacher 2y ago
The output of plagiarism tools should only serve as a hint to look at a pair of solutions more closely. All judgement should be derived entirely from similarities between solutions and not some artificial similarity score computed by some program.
- ziddoap 2y ago
  Unfortunately, this is not really what happens in my experience. The output of plagiarism tools is taken as fact (especially at high school levels). Without extraordinary evidence of the tool being incorrect, students have no recourse, even if they could sit and explain the thought process behind every word/line of code/whatever.
  - deleted 2y ago
    [deleted]
  - nsajko 2y ago
    Lousy high school.
- drapado 2y ago
  Indeed, this is exactly what I did.

beeboobaa3 2y ago

Next time just ask them a few questions about the programming choices they made. Far easier.

thi341 2y ago

An archival video of all coding sessions (locally, hosted by the student), starting with a visible outline of pseudo-code and ending with debugging should be sufficient. In case of a false positive from a faulty detector this is extraordinary evidence.

wildzzz 2y ago
We had a professor require us to use git as a timestamped log of our progress. Of course you could fake it but stealing work and basically redoing it piece by piece with fake timestamps is a lot of work for cheaters.