11 ms·

That sounds great and all but I honestly have doubts about this software that detects similarities… there’s only so many ways to solve the bland questions that

That sounds great and all but I honestly have doubts about this software that detects similarities… there’s only so many ways to solve the bland questions that professors lift from books; kind of ironic. I’m assuming it’s basicallly doing AST analysis and it’s no smarter than eliminating things like variables being renamed. They are basically stating that this “software” is 100% accurate. Furthermore it’s then left to whims of some TAs? No algorithm can detect cheating unless the number of permutations are very very large (I.e being struck by lightening). Maybe one way to offset would be to use data as the student is entering the solution but that was never the case for us; just upload the source code to their custom made Windows app.

TheApexTheater 4y ago
From doing some cursory research, it appears the software in question is called MOSS (Measure of Software Similarity) and is currently being provided as a service [0]. Since it is intended to be used by instructors and staff, the source is restricted (though "anyone may create a MOSS account"). According to the paper describing how it's used [1], "False positives have never been reported, and all false negatives were quickly traced back to the source, which was either an implementation or a user misunderstanding." Sources: [0]: https://theory.stanford.edu/~aiken/moss https://theory.stanford.edu/~aiken/moss [1]: http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf http://theory.stanford.edu/~aiken/publications/papers/sigmod...
dskloet 4y ago
I used something similar when I was a TA 20 years ago and while your assumption seems reasonable, there are actually a lot of different ways to solve even quite simple tasks and most cheating is very obvious on manual inspection.
- Sirened 4y ago
  Yep... If you're going to go through the effort of completely rewriting a piece of code to try and dodge an AST analysis algorithm, you've effectively just done 70% of the work and put your grade/position at the institution on the line. It's not worth it, and so people don't tend to do that. It's the same thing with plagiarism—students could very well resynthesize a stolen work in their own words. It would still be plagiarism, sure, but it's also putting in a large amount of effort while still being risky.
  - Aeolun 4y ago
    If you rewrite everything you steal (e.g. never copy/paste), it’s no different from using an especially well written source.
    - Sirened 4y ago
      Well, no. It's still plagiarized if you fail to communicate that it isn't your original work. You can't just steal ideas from someone else's paper, even if you rewrite everything. If you rely on another paper for inspiration, you have to cite it. If a student submitted a paper that was just another paper entirely rephrased, that would not be acceptable in the least even if they cited their source because the expectation of writing a paper is that you contribute something novel and not just regurgitate someone else's argument
- bhargav 4y ago
  If the problem is large enough, I do submit that there are multiple (even many) ways of solving it. I will also say that there’s problems where that is not the case. For example, we were told to write simulators for scheduling schemes (RR, MLFQ). Other than using different data structures (even that’s a bit of a stretch) not sure how much variance there will be. Using the right tool for the right job is important. Just above your post another author posted/cited results of a system that “never produced false positives”. I think that cited number another author shared is probably correct but presumably, the tool is used in cases where problems are big enough to warrant it.
  - dskloet 4y ago
    The problems we had were way way simpler than anything deserving an acronym. You'd think there was only one way to do it and yet it was not hard to distinguish plagiarism.
    - bhargav 4y ago
      Do you happen to have a few examples? I’m super curious! How many students were taking the course?
lolinder 4y ago
I implemented the widely used MOSS algorithm (mentioned by a sibling) for my CS department in my senior year. That algorithm doesn't do AST analysis, it just looks at the plain text in a way that is resistant to most small refactorings. MOSS compares sets of k-grams (strings of k characters) between every pair of projects that are under test and produces the number of shared k-grams for each pair of projects. On any given assignment in a given semester, there's a baseline amount of of similarity that is "normal". You then test for outliers, and that gives you the projects that need closer scrutiny. On the test data we were given (anonymized assignments from prior semesters together with known public git repos), we never had a false positive. On the flip side, small refactorings like variable renames or method re-ordering still turned up above the "suspicious" threshold because there would be enough remaining matching k-grams to make that pair of projects an outlier. Our school explicitly did not use the algorithm's numbers as evidence of cheating and did not involve the TAs--the numbers were used only to point the professor in the right direction. We excluded all k-grams that featured in the professor's materials (slides, examples, boilerplate code). It also helped that they only used it on the more complex assignments that should have had unique source code (our test data was a client and server for an Android app). My sense was that this was a pretty good system. Cheaters stood out in the outliers test by several orders of magnitude, so false positives are extremely unlikely. At the same time, the k-gram approach means that if you actually manage to mangle your project enough that it's not detected as copied, you had to perform refactorings in the process that clearly show you know how the program works--anything less still leaves you above the safe zone of shared k-grams.
temp3738868585 4y ago
Speaking from experience using similar software on students assignments, it is often blatantly obvious when cheating is occurring. To start with, at an undergrad level, most students had fairly distinct coding styles - usually with quirks of not "proper" coding. Some cheaters had the exact same quirks in multiple students assignments. Also, some cheaters had the exact same mistakes in their code, on top of the same code style. Yes the software picks up people that write correct solutions with perfect syntax, but those are the ones that you just toss out because there isn't any proof there. The people that get caught cheating generally don't know what correct solutions and good code look like, so they don't understand how obvious it is when they copy paste their friends mediocre code.
- cudgy 4y ago
  Assuming that the software detects a similarity between two or more student’s submissions, how do you know which students cheated? What if one of the students (the one that actually did the work) had their program stolen/copied somehow (eg left screen open in lab or print out of code)?
  - GoOnThenDoTell 4y ago
    You ask them to do it again in front of you
  - rrobukef 4y ago
    It doesn't matter, both violated academic integrity by letting the copy happen. (Submissions are never stolen) If you think letting copying happen is less severe, you ask them and rebalance based on the work. Most of the time 'they made it together'.
    - tomp 4y ago
      How is sharing knowledge "violating academic integrity"? Unless given specific and explicit instructions not to reveal working solutions, then sharing your code is literally just "helping" others, it's up to them to either study it and produce their own versions, or jut blatantly copy & cheat.

TheApexTheater 4y ago

From doing some cursory research, it appears the software in question is called MOSS (Measure of Software Similarity) and is currently being provided as a service [0]. Since it is intended to be used by instructors and staff, the source is restricted (though "anyone may create a MOSS account"). According to the paper describing how it's used [1], "False positives have never been reported, and all false negatives were quickly traced back to the source, which was either an implementation or a user misunderstanding." Sources: [0]: https://theory.stanford.edu/~aiken/moss https://theory.stanford.edu/~aiken/moss [1]: http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf http://theory.stanford.edu/~aiken/publications/papers/sigmod...

dskloet 4y ago

I used something similar when I was a TA 20 years ago and while your assumption seems reasonable, there are actually a lot of different ways to solve even quite simple tasks and most cheating is very obvious on manual inspection.

Sirened 4y ago
Yep... If you're going to go through the effort of completely rewriting a piece of code to try and dodge an AST analysis algorithm, you've effectively just done 70% of the work and put your grade/position at the institution on the line. It's not worth it, and so people don't tend to do that. It's the same thing with plagiarism—students could very well resynthesize a stolen work in their own words. It would still be plagiarism, sure, but it's also putting in a large amount of effort while still being risky.
- Aeolun 4y ago
  If you rewrite everything you steal (e.g. never copy/paste), it’s no different from using an especially well written source.
  - Sirened 4y ago
    Well, no. It's still plagiarized if you fail to communicate that it isn't your original work. You can't just steal ideas from someone else's paper, even if you rewrite everything. If you rely on another paper for inspiration, you have to cite it. If a student submitted a paper that was just another paper entirely rephrased, that would not be acceptable in the least even if they cited their source because the expectation of writing a paper is that you contribute something novel and not just regurgitate someone else's argument
bhargav 4y ago
If the problem is large enough, I do submit that there are multiple (even many) ways of solving it. I will also say that there’s problems where that is not the case. For example, we were told to write simulators for scheduling schemes (RR, MLFQ). Other than using different data structures (even that’s a bit of a stretch) not sure how much variance there will be. Using the right tool for the right job is important. Just above your post another author posted/cited results of a system that “never produced false positives”. I think that cited number another author shared is probably correct but presumably, the tool is used in cases where problems are big enough to warrant it.
- dskloet 4y ago
  The problems we had were way way simpler than anything deserving an acronym. You'd think there was only one way to do it and yet it was not hard to distinguish plagiarism.
  - bhargav 4y ago
    Do you happen to have a few examples? I’m super curious! How many students were taking the course?

lolinder 4y ago

I implemented the widely used MOSS algorithm (mentioned by a sibling) for my CS department in my senior year. That algorithm doesn't do AST analysis, it just looks at the plain text in a way that is resistant to most small refactorings. MOSS compares sets of k-grams (strings of k characters) between every pair of projects that are under test and produces the number of shared k-grams for each pair of projects. On any given assignment in a given semester, there's a baseline amount of of similarity that is "normal". You then test for outliers, and that gives you the projects that need closer scrutiny. On the test data we were given (anonymized assignments from prior semesters together with known public git repos), we never had a false positive. On the flip side, small refactorings like variable renames or method re-ordering still turned up above the "suspicious" threshold because there would be enough remaining matching k-grams to make that pair of projects an outlier. Our school explicitly did not use the algorithm's numbers as evidence of cheating and did not involve the TAs--the numbers were used only to point the professor in the right direction. We excluded all k-grams that featured in the professor's materials (slides, examples, boilerplate code). It also helped that they only used it on the more complex assignments that should have had unique source code (our test data was a client and server for an Android app). My sense was that this was a pretty good system. Cheaters stood out in the outliers test by several orders of magnitude, so false positives are extremely unlikely. At the same time, the k-gram approach means that if you actually manage to mangle your project enough that it's not detected as copied, you had to perform refactorings in the process that clearly show you know how the program works--anything less still leaves you above the safe zone of shared k-grams.

temp3738868585 4y ago

Speaking from experience using similar software on students assignments, it is often blatantly obvious when cheating is occurring. To start with, at an undergrad level, most students had fairly distinct coding styles - usually with quirks of not "proper" coding. Some cheaters had the exact same quirks in multiple students assignments. Also, some cheaters had the exact same mistakes in their code, on top of the same code style. Yes the software picks up people that write correct solutions with perfect syntax, but those are the ones that you just toss out because there isn't any proof there. The people that get caught cheating generally don't know what correct solutions and good code look like, so they don't understand how obvious it is when they copy paste their friends mediocre code.

cudgy 4y ago
Assuming that the software detects a similarity between two or more student’s submissions, how do you know which students cheated? What if one of the students (the one that actually did the work) had their program stolen/copied somehow (eg left screen open in lab or print out of code)?
- GoOnThenDoTell 4y ago
  You ask them to do it again in front of you
- rrobukef 4y ago
  It doesn't matter, both violated academic integrity by letting the copy happen. (Submissions are never stolen) If you think letting copying happen is less severe, you ask them and rebalance based on the work. Most of the time 'they made it together'.
  - tomp 4y ago
    How is sharing knowledge "violating academic integrity"? Unless given specific and explicit instructions not to reveal working solutions, then sharing your code is literally just "helping" others, it's up to them to either study it and produce their own versions, or jut blatantly copy & cheat.