10 ms·
JPlag – Detecting Software Plagiarism
- drapado 2y agoI recently had to check code from some of my students at the university as I suspected plagiarism. I discovered JPlag which works like a charm and generates nice reports
- jaimex2 2y agoKinda rare these days with ChatGPT
- i_am_proteus 2y agoYou might be surprised. Many students who use ChatGPT for assignments end up turning in code identical (or nearly identical) to other students who use ChatGPT.
- hatmatrix 2y agoSurprising because you get different answers each time you ask ChatGPT.
- mmcwilliams 2y agoDifferent in an exact string match but code that is copied and pasted from ChatGPT has a lot of similarities in the way that it is (over) commented. I've seen a lot of Python where the student who "authored" it cannot tell me how a method works or why it was implemented despite having the comments prefixed to every line in the file.
- Joker_vD 2y agoAh yes, good old "Did you even read the essay before handing it in? Next time, please do."
- elashri 2y ago> (over) commented From my experience using ChatGPT, It usually remove most of my already written comments when I ask questions about code I wrote myself. It usually give you outline comments. So unless you are supporter of the self documented code idea, I don't think ChatGPT over comments.
- mmcwilliams 2y agoIt's obviously down to taste, but what I've seen over and over is a comment per line which to me is excessive outside it being requested of absolute beginners. That happens and also the model can't decide if it wants the comment on the line before the code or if everything should be appended to the line itself so when I see both styles within a single project it's another signal. People generally have a style that they stick with.
- pdntspa 2y agoChatGPT answers don't differ that much without being prompted to do so
- serf 2y agoyeah but the prompt itself generally adds sufficient randomness to avoid the same verbatim answer each time. as an example just go ask it to write any sufficiently average function. use different names and phrases for what the function should do; you'll generally get a different flavor of answer each time, even if the functions all output the same thing. sometimes the prompt even forces the thing to output the most naive implementation possible due to the ordering or perceived priority of things within the requesting prompt. it's fun to use as a tool to nudge it into what you want once you get the hang of the preconceptions it falls into.
- franga2000 2y agoMOSS seems to be pretty good finding multiple people using LLM-generated code and flagging them as copies of each other. I imagine it would also be a good idea to throw the assignment text into the few most popular LLMs and feed that in as well, but I don't know of anyone who has tried this.
- emeryberger 2y agoFWIW the attack we describe in the paper works against MOSS, too (that was the original inspiration for the name, “Mossad”).
- BossingAround 2y agoHow do you deal with disputes? One's code is flagged even if the student in question didn't actually cheat. What then? Do you trust tools over the students' word? In addition, do things like stack overflow and using LLM-generated code count as cheating? Because that is horrible in and of itself, though a separate concern.
- deleted 2y ago[deleted]
- ablob 2y agoIf you talk about the written code to the student in question it should become clear whether it was copied or not.
- ActualTeacher 2y agoThe output of plagiarism tools should only serve as a hint to look at a pair of solutions more closely. All judgement should be derived entirely from similarities between solutions and not some artificial similarity score computed by some program.
- ziddoap 2y agoUnfortunately, this is not really what happens in my experience. The output of plagiarism tools is taken as fact (especially at high school levels). Without extraordinary evidence of the tool being incorrect, students have no recourse, even if they could sit and explain the thought process behind every word/line of code/whatever.
- deleted 2y ago[deleted]
- nsajko 2y agoLousy high school.
- drapado 2y agoIndeed, this is exactly what I did.
- beeboobaa3 2y agoNext time just ask them a few questions about the programming choices they made. Far easier.
- thi341 2y agoAn archival video of all coding sessions (locally, hosted by the student), starting with a visible outline of pseudo-code and ending with debugging should be sufficient. In case of a false positive from a faulty detector this is extraordinary evidence.
- wildzzz 2y agoWe had a professor require us to use git as a timestamped log of our progress. Of course you could fake it but stealing work and basically redoing it piece by piece with fake timestamps is a lot of work for cheaters.
- playingalong 2y agoCan I use it to detect Copy & Paste within my company's own codebase?
- chuckadams 2y agoJetbrains IDEs will flag duplicate code of N lines (configurable) or more. Just run the inspection manually and there you go. You can even do it in batch in CI nowadays.
- maleldil 2y agoI've had some success with PMD/CMD [1]. It's Java-based, though, which could be a pain, depending on your setup. If your codebase uses Python, you can use Pylint for this, too. [1] https://pmd.github.io/pmd/pmd_userdocs_cpd.html https://pmd.github.io/pmd/pmd_userdocs_cpd.html
- willy_k 2y agoWhat do you need that for? It’s not really cheating when it’s going into a product, if it works it works. In a hypothetical ideal corporate environment wouldn’t it be preferred if one could save company time by copy and pasting?
- Retr0id 2y agoPresumably, this needs a corpus of software to check against. Does it include one, or do you have to bring your own?
- sakjur 2y ago> Just to make it clear: JPlag does not compare to the internet! It is designed to find similarities among the student solutions, which is usually sufficient for computer programs. It seems like the latter based on their wiki, but also that that corpus can be relatively small.
- deleted 2y ago[deleted]
- emeryberger 2y agoJPlag, like similar plagiarism detectors, is vulnerable to attack. We outline the attack in this paper and show its effectiveness against JPlag and another widely used plagiarism detector, Moss. Note that this was written in 2020, in the pre “CheatGPT” era! https://arxiv.org/abs/2010.01700 https://arxiv.org/abs/2010.01700 Mossad: Defeating Software Plagiarism Detection Breanna Devore-McDonald, Emery D. Berger Automatic software plagiarism detection tools are widely used in educational settings to ensure that submitted work was not copied. These tools have grown in use together with the rise in enrollments in computer science programs and the widespread availability of code on-line. Educators rely on the robustness of plagiarism detection tools; the working assumption is that the effort required to evade detection is as high as that required to actually do the assigned work. This paper shows this is not the case. It presents an entirely automatic program transformation approach, Mossad, that defeats popular software plagiarism detection tools. Mossad comprises a framework that couples techniques inspired by genetic programming with domain-specific knowledge to effectively undermine plagiarism detectors. Mossad is effective at defeating four plagiarism detectors, including Moss and JPlag. Mossad is both fast and effective: it can, in minutes, generate modified versions of programs that are likely to escape detection. More insidiously, because of its non-deterministic approach, Mossad can, from a single program, generate dozens of variants, which are classified as no more suspicious than legitimate assignments. A detailed study of Mossad across a corpus of real student assignments demonstrates its efficacy at evading detection. A user study shows that graduate student assistants consistently rate Mossad-generated code as just as readable as authentic student code. This work motivates the need for both research on more robust plagiarism detection tools and greater integration of naturally plagiarism-resistant methodologies like code review into computer science education.
- diggan 2y ago> A user study shows that graduate student assistants consistently rate Mossad-generated code as just as readable as authentic student code. Do you have any small examples on a program that was transformed/generated with Mossad that we could compare against the original? As far as I can tell, the paper just have a really tiny example function.
- 2y ago
- hooverd 2y agoIt's funny how we drill the idea that everything must be reimplemented from first principles into students, only to flip that when they join the workforce.
- jffhn 2y agoexam: implement add. "return a + b": plagiarism, disqualified. "return a + 1 + b - 1": A+
- TuringNYC 2y agoThe exam questions themselves would be plagiarism also
- deleted 2y ago[deleted]
- donatj 2y agoExactly this. If my coworker implemented sort themselves we'd have questions about their qualifications.
- diggan 2y ago> It's funny how we drill the idea that everything must be reimplemented from first principles into students, only to flip that when they join the workforce. Kind of makes sense, doesn't it? While in school, you want to learn as much as possible (ideally?), while in the workforce, you want (or the company wants you) to be as efficient as possible. Different goals leads to different workflows.
- dataflow 2y ago> It's funny how we drill the idea that everything must be reimplemented from first principles into students Where on earth did you go to school that you couldn't just call list.sort() after your first algorithms class? I've never seen a class that teaches students that everything must be implemented from first principles. They only do that for the concepts taught in that class, or 1-2 classes prior, to make sure you are actually learning and have a clue what you're doing. Which... should makes sense? You should want engineers to cut costs based on understanding rather than ignorance.
- geraldwhen 2y ago[flagged]
- alsodumb 2y agoIt’s no secret that huge portion of all students are cheating or plagiarizing - let’s not pretend it’s only international students. My few data points being a TA for many years at a top engineering school: about 70% of the cases I reported for violations were domestic students.
- tossandthrow 2y agoDepending on the distribution of international / domestic students, this also sounds wild and in congruence with the other commenter. If there is 10 out of 100 international students, and they do 30% of 10 cases, then 30% (!!) of international students cheat while only 7% of domestic students cheat - in conclusion, international students do most of the cheating. Regardless of this statistical thought experiment, I don't know what to do of data like this. It is not really ethical to act on it.
- smokel 2y agoThis is a silly game. At MIT, 29.1% of students are international. [1] [1] https://facts.mit.edu/enrollment-statistics/ https://facts.mit.edu/enrollment-statistics/
- TwiztidK 2y agoSame link confirms that only 11% of undergrads are international students.
- ActualTeacher 2y agoIn my experience as a TA, ChatGPT has leveled the playing field. Previously, some nationalities cheated significantly more often, but now almost everyone (at least 85 %, probably more) cheat with ChatGPT. There is not much point in providing homework assignments anymore.
- EveOffline 2y ago[dead]
- viralpraxis 2y agoOne of the key outcomes of my master's thesis was the development of an extendable solution for Code Clone Detection (CCD), primarily focused on code and tested with undergraduates at my university [1]. Although I didn't have time to complete the adapter for JPlag, I believe it would be highly beneficial. Interestingly, whenever I discussed my thesis, the first reaction from others often revolved around moral concerns. [1] https://github.com/studyfair/studyfair https://github.com/studyfair/studyfair
- michaelmior 2y agoThis looks cool, but for me one of the big wins with JPlag is that I just download and run a single JAR file.
- JosephRedfern 2y agoIs this also effective at detecting code duplication within a codebase?
- thiht 2y agoI was actually looking for something like this a few days ago! There’s an open source tool which I love the idea of (basically a tool for declarative integration tests), but I really don’t like it’s implementation. I tried to contribute to improve it, but it’s too much work and it will never fit my ideal. So I basically decided to "redo it but better", and I’m also tempted to make it a paid, proprietary tool because my implementation diverges enough that I consider it a different codebase altogether (and it would bring legitimate value to companies). I wrote my code from scratch but still had some knowledge of the original code base so I’d be interested in running something like JPlag to make sure I didn’t accidentally plagiarize open source code. I hope I find a way to make it compare 2 codebases :)
- meiraleal 2y ago> to make sure I didn’t accidentally plagiarize open source code. If you didn't plagiarize, you don't need to run the tool. If you did plagiarize and want to hide it, tho...
- thiht 2y agoThere’s big sections of code I wrote in the original open source lib. I didn’t copy paste the code but the implementation in this component is obviously pretty close. I’m the copyright holder of this code anyway so it should not be an issue, but I’d rather not take the risk. Plagiarism is not always clear cut because life is messy. That’s why Wine doesn’t allow contributions from people who have seen Windows source code[1] for instance, even though it could be good faith contributions with experience instead of plagiarism [1]: https://wiki.winehq.org/Developer_FAQ#Who_can't_contribute_to_Wine https://wiki.winehq.org/Developer_FAQ#Who_can't_contribute_t...?
- westurner 2y agoShould a plagiarism score be considered when generating code with an infinite monkeys algorithm with selection or better? Would that result in inability to write code in a clean room, even; because eventually all possible code strings and mutations thereof would already be patented. For example, are three notes or chords copyrightable?
- ocean_moist 2y agopro tip: change variable/function names, switch from if/else to switch, invert if/else statements, switch for and while loops, group code differently, create helper functions or collapse helper functions, rewrite loops as streams/ranges/list comprehensions, etc. The ide can do most automatically. It is pretty much impossible to detect software plagiarism, especially on leetcode style questions as only 1 style or pattern is the most efficient answer. Though if a student changes it sufficiently enough, they might begin to actually see the invariants and ideas and actually learn the material.