6 ms·

I recently took a CS class at Stanford with an interesting policy on cheating. While cheating almost certainly happened during the course, at the end of the qua

by carls 4y ago

I recently took a CS class at Stanford with an interesting policy on cheating. While cheating almost certainly happened during the course, at the end of the quarter the course staff made a public post allowing any student who cheated to make a private message to the staff admitting they've done so. If a student admitted to cheating, while they would face academic disciplinary action (i.e. receiving a failing or low grade), they would not be brought up to the administrative office that deals with issues of academic integrity, and therefore would not face consequences like expulsion or being on official academic probation. However if a cheating student decided to risk it and not admit their guilt, they were at risk of a potentially even greater degree punishment. The course staff would run all students code through a piece of software to detect similarities between each other, as well as online solutions. Students who were flagged by this software would then have their code hand-checked by at least one course staff, who would make a judgement call as to whether it seemed like cheating. I found this policy quite interesting. As a former high school teacher, I've certain encountered teaching in my own classes, and have historically oscillated between taking a very harsh stance, or perhaps an overly permissive one. The one taken by the lecturers of this course offered a "second chance" to cheaters in a way I hadn't seen before.

bhargav 4y ago
That sounds great and all but I honestly have doubts about this software that detects similarities… there’s only so many ways to solve the bland questions that professors lift from books; kind of ironic. I’m assuming it’s basicallly doing AST analysis and it’s no smarter than eliminating things like variables being renamed. They are basically stating that this “software” is 100% accurate. Furthermore it’s then left to whims of some TAs? No algorithm can detect cheating unless the number of permutations are very very large (I.e being struck by lightening). Maybe one way to offset would be to use data as the student is entering the solution but that was never the case for us; just upload the source code to their custom made Windows app.
- TheApexTheater 4y ago
  From doing some cursory research, it appears the software in question is called MOSS (Measure of Software Similarity) and is currently being provided as a service [0]. Since it is intended to be used by instructors and staff, the source is restricted (though "anyone may create a MOSS account"). According to the paper describing how it's used [1], "False positives have never been reported, and all false negatives were quickly traced back to the source, which was either an implementation or a user misunderstanding." Sources: [0]: https://theory.stanford.edu/~aiken/moss https://theory.stanford.edu/~aiken/moss [1]: http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf http://theory.stanford.edu/~aiken/publications/papers/sigmod...
- dskloet 4y ago
  I used something similar when I was a TA 20 years ago and while your assumption seems reasonable, there are actually a lot of different ways to solve even quite simple tasks and most cheating is very obvious on manual inspection.
  - Sirened 4y ago
    Yep... If you're going to go through the effort of completely rewriting a piece of code to try and dodge an AST analysis algorithm, you've effectively just done 70% of the work and put your grade/position at the institution on the line. It's not worth it, and so people don't tend to do that. It's the same thing with plagiarism—students could very well resynthesize a stolen work in their own words. It would still be plagiarism, sure, but it's also putting in a large amount of effort while still being risky.
    - Aeolun 4y ago
      If you rewrite everything you steal (e.g. never copy/paste), it’s no different from using an especially well written source.
      - Sirened 4y ago
        Well, no. It's still plagiarized if you fail to communicate that it isn't your original work. You can't just steal ideas from someone else's paper, even if you rewrite everything. If you rely on another paper for inspiration, you have to cite it. If a student submitted a paper that was just another paper entirely rephrased, that would not be acceptable in the least even if they cited their source because the expectation of writing a paper is that you contribute something novel and not just regurgitate someone else's argument
  - bhargav 4y ago
    If the problem is large enough, I do submit that there are multiple (even many) ways of solving it. I will also say that there’s problems where that is not the case. For example, we were told to write simulators for scheduling schemes (RR, MLFQ). Other than using different data structures (even that’s a bit of a stretch) not sure how much variance there will be. Using the right tool for the right job is important. Just above your post another author posted/cited results of a system that “never produced false positives”. I think that cited number another author shared is probably correct but presumably, the tool is used in cases where problems are big enough to warrant it.
- lolinder 4y ago
  I implemented the widely used MOSS algorithm (mentioned by a sibling) for my CS department in my senior year. That algorithm doesn't do AST analysis, it just looks at the plain text in a way that is resistant to most small refactorings. MOSS compares sets of k-grams (strings of k characters) between every pair of projects that are under test and produces the number of shared k-grams for each pair of projects. On any given assignment in a given semester, there's a baseline amount of of similarity that is "normal". You then test for outliers, and that gives you the projects that need closer scrutiny. On the test data we were given (anonymized assignments from prior semesters together with known public git repos), we never had a false positive. On the flip side, small refactorings like variable renames or method re-ordering still turned up above the "suspicious" threshold because there would be enough remaining matching k-grams to make that pair of projects an outlier. Our school explicitly did not use the algorithm's numbers as evidence of cheating and did not involve the TAs--the numbers were used only to point the professor in the right direction. We excluded all k-grams that featured in the professor's materials (slides, examples, boilerplate code). It also helped that they only used it on the more complex assignments that should have had unique source code (our test data was a client and server for an Android app). My sense was that this was a pretty good system. Cheaters stood out in the outliers test by several orders of magnitude, so false positives are extremely unlikely. At the same time, the k-gram approach means that if you actually manage to mangle your project enough that it's not detected as copied, you had to perform refactorings in the process that clearly show you know how the program works--anything less still leaves you above the safe zone of shared k-grams.
- temp3738868585 4y ago
  Speaking from experience using similar software on students assignments, it is often blatantly obvious when cheating is occurring. To start with, at an undergrad level, most students had fairly distinct coding styles - usually with quirks of not "proper" coding. Some cheaters had the exact same quirks in multiple students assignments. Also, some cheaters had the exact same mistakes in their code, on top of the same code style. Yes the software picks up people that write correct solutions with perfect syntax, but those are the ones that you just toss out because there isn't any proof there. The people that get caught cheating generally don't know what correct solutions and good code look like, so they don't understand how obvious it is when they copy paste their friends mediocre code.
  - cudgy 4y ago
    Assuming that the software detects a similarity between two or more student’s submissions, how do you know which students cheated? What if one of the students (the one that actually did the work) had their program stolen/copied somehow (eg left screen open in lab or print out of code)?
    - GoOnThenDoTell 4y ago
      You ask them to do it again in front of you
    - rrobukef 4y ago
      It doesn't matter, both violated academic integrity by letting the copy happen. (Submissions are never stolen) If you think letting copying happen is less severe, you ask them and rebalance based on the work. Most of the time 'they made it together'.
      - tomp 4y ago
        How is sharing knowledge "violating academic integrity"? Unless given specific and explicit instructions not to reveal working solutions, then sharing your code is literally just "helping" others, it's up to them to either study it and produce their own versions, or jut blatantly copy & cheat.
deleted 4y ago
[deleted]
bhrgunatha 4y ago
> I've certain encountered teaching in my own classes, This kind of mistyping reminds me of those example of people whose names fit their job but it's rare to find such an apropos example. Sorry to derail your point but the juxtaposition of "ch" and "t" here is perfect.
- mkl 4y ago
  Not juxtaposition, transposition. https://en.wiktionary.org/wiki/juxtaposition https://en.wiktionary.org/wiki/juxtaposition
  - bhrgunatha 4y ago
    Yes, my mistake, thanks.
kurthr 4y ago
I noticed a swap in your prose (still comprehensible), but just realized that cheating and teaching are semi-spoonerisms (swapping the sound order of a single word)... how appropos!
donkeybeer 4y ago
The results of checking against existing and other test takers solutions must be taken with a strong human judgment. Programming problems such as would be asked in tests are essentially like mathematical formulas/algorithms, and there isn't much variation in how a given formula or algorithm can be implemented.
- tsimionescu 4y ago
  I don't think these techniques are often applied to problems in tests - there are other, simpler ways of catching cheaters there. They are much more likely to be applied to homework assignments, where the opportunity for copying is large, but the chance of two students producing the exact same >500-1000 line program is slim to none. Perhaps once in a while a critical function will be copied and no one will realize or similarities in a trivial function will be unnecessarily flagged, but this will be relatively rare and quickly discovered in manual review.
- rrobukef 4y ago
  There is a lot of syntactic variation possible, both for formula and algorithms. Even for something as simple as quicksort there is enough natural variation for a class of 30 maybe even 100 (if no references can be used). Anything more complex and even with references it should be unique.
voiper1 4y ago
It's not _just_ trying to be lenient and offer a second chance - it's a way to catch more cheaters. "Turn yourself in and we'll go easy on you... because we might not catch you."
Claude_Shannon 4y ago
I have the same policy at my uni in Poland. Admit to cheating without being called out? Depending on the professor mood, they either allowed you to retake exam (though the best grade you'd get was the lowest passing one), or you just fail the course and try again next year. Both they had less paperwork, and well, they wouldn't report that person.

by carls 4y ago

bhargav 4y ago

That sounds great and all but I honestly have doubts about this software that detects similarities… there’s only so many ways to solve the bland questions that professors lift from books; kind of ironic. I’m assuming it’s basicallly doing AST analysis and it’s no smarter than eliminating things like variables being renamed. They are basically stating that this “software” is 100% accurate. Furthermore it’s then left to whims of some TAs? No algorithm can detect cheating unless the number of permutations are very very large (I.e being struck by lightening). Maybe one way to offset would be to use data as the student is entering the solution but that was never the case for us; just upload the source code to their custom made Windows app.

TheApexTheater 4y ago
From doing some cursory research, it appears the software in question is called MOSS (Measure of Software Similarity) and is currently being provided as a service [0]. Since it is intended to be used by instructors and staff, the source is restricted (though "anyone may create a MOSS account"). According to the paper describing how it's used [1], "False positives have never been reported, and all false negatives were quickly traced back to the source, which was either an implementation or a user misunderstanding." Sources: [0]: https://theory.stanford.edu/~aiken/moss https://theory.stanford.edu/~aiken/moss [1]: http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf http://theory.stanford.edu/~aiken/publications/papers/sigmod...
dskloet 4y ago
I used something similar when I was a TA 20 years ago and while your assumption seems reasonable, there are actually a lot of different ways to solve even quite simple tasks and most cheating is very obvious on manual inspection.
- Sirened 4y ago
  Yep... If you're going to go through the effort of completely rewriting a piece of code to try and dodge an AST analysis algorithm, you've effectively just done 70% of the work and put your grade/position at the institution on the line. It's not worth it, and so people don't tend to do that. It's the same thing with plagiarism—students could very well resynthesize a stolen work in their own words. It would still be plagiarism, sure, but it's also putting in a large amount of effort while still being risky.
  - Aeolun 4y ago
    If you rewrite everything you steal (e.g. never copy/paste), it’s no different from using an especially well written source.
    - Sirened 4y ago
      Well, no. It's still plagiarized if you fail to communicate that it isn't your original work. You can't just steal ideas from someone else's paper, even if you rewrite everything. If you rely on another paper for inspiration, you have to cite it. If a student submitted a paper that was just another paper entirely rephrased, that would not be acceptable in the least even if they cited their source because the expectation of writing a paper is that you contribute something novel and not just regurgitate someone else's argument
- bhargav 4y ago
  If the problem is large enough, I do submit that there are multiple (even many) ways of solving it. I will also say that there’s problems where that is not the case. For example, we were told to write simulators for scheduling schemes (RR, MLFQ). Other than using different data structures (even that’s a bit of a stretch) not sure how much variance there will be. Using the right tool for the right job is important. Just above your post another author posted/cited results of a system that “never produced false positives”. I think that cited number another author shared is probably correct but presumably, the tool is used in cases where problems are big enough to warrant it.
lolinder 4y ago
I implemented the widely used MOSS algorithm (mentioned by a sibling) for my CS department in my senior year. That algorithm doesn't do AST analysis, it just looks at the plain text in a way that is resistant to most small refactorings. MOSS compares sets of k-grams (strings of k characters) between every pair of projects that are under test and produces the number of shared k-grams for each pair of projects. On any given assignment in a given semester, there's a baseline amount of of similarity that is "normal". You then test for outliers, and that gives you the projects that need closer scrutiny. On the test data we were given (anonymized assignments from prior semesters together with known public git repos), we never had a false positive. On the flip side, small refactorings like variable renames or method re-ordering still turned up above the "suspicious" threshold because there would be enough remaining matching k-grams to make that pair of projects an outlier. Our school explicitly did not use the algorithm's numbers as evidence of cheating and did not involve the TAs--the numbers were used only to point the professor in the right direction. We excluded all k-grams that featured in the professor's materials (slides, examples, boilerplate code). It also helped that they only used it on the more complex assignments that should have had unique source code (our test data was a client and server for an Android app). My sense was that this was a pretty good system. Cheaters stood out in the outliers test by several orders of magnitude, so false positives are extremely unlikely. At the same time, the k-gram approach means that if you actually manage to mangle your project enough that it's not detected as copied, you had to perform refactorings in the process that clearly show you know how the program works--anything less still leaves you above the safe zone of shared k-grams.
temp3738868585 4y ago
Speaking from experience using similar software on students assignments, it is often blatantly obvious when cheating is occurring. To start with, at an undergrad level, most students had fairly distinct coding styles - usually with quirks of not "proper" coding. Some cheaters had the exact same quirks in multiple students assignments. Also, some cheaters had the exact same mistakes in their code, on top of the same code style. Yes the software picks up people that write correct solutions with perfect syntax, but those are the ones that you just toss out because there isn't any proof there. The people that get caught cheating generally don't know what correct solutions and good code look like, so they don't understand how obvious it is when they copy paste their friends mediocre code.
- cudgy 4y ago
  Assuming that the software detects a similarity between two or more student’s submissions, how do you know which students cheated? What if one of the students (the one that actually did the work) had their program stolen/copied somehow (eg left screen open in lab or print out of code)?
  - GoOnThenDoTell 4y ago
    You ask them to do it again in front of you
  - rrobukef 4y ago
    It doesn't matter, both violated academic integrity by letting the copy happen. (Submissions are never stolen) If you think letting copying happen is less severe, you ask them and rebalance based on the work. Most of the time 'they made it together'.
    - tomp 4y ago
      How is sharing knowledge "violating academic integrity"? Unless given specific and explicit instructions not to reveal working solutions, then sharing your code is literally just "helping" others, it's up to them to either study it and produce their own versions, or jut blatantly copy & cheat.

deleted 4y ago

[deleted]

bhrgunatha 4y ago

> I've certain encountered teaching in my own classes, This kind of mistyping reminds me of those example of people whose names fit their job but it's rare to find such an apropos example. Sorry to derail your point but the juxtaposition of "ch" and "t" here is perfect.

mkl 4y ago
Not juxtaposition, transposition. https://en.wiktionary.org/wiki/juxtaposition https://en.wiktionary.org/wiki/juxtaposition
- bhrgunatha 4y ago
  Yes, my mistake, thanks.

kurthr 4y ago

I noticed a swap in your prose (still comprehensible), but just realized that cheating and teaching are semi-spoonerisms (swapping the sound order of a single word)... how appropos!

donkeybeer 4y ago

The results of checking against existing and other test takers solutions must be taken with a strong human judgment. Programming problems such as would be asked in tests are essentially like mathematical formulas/algorithms, and there isn't much variation in how a given formula or algorithm can be implemented.

tsimionescu 4y ago
I don't think these techniques are often applied to problems in tests - there are other, simpler ways of catching cheaters there. They are much more likely to be applied to homework assignments, where the opportunity for copying is large, but the chance of two students producing the exact same >500-1000 line program is slim to none. Perhaps once in a while a critical function will be copied and no one will realize or similarities in a trivial function will be unnecessarily flagged, but this will be relatively rare and quickly discovered in manual review.
rrobukef 4y ago
There is a lot of syntactic variation possible, both for formula and algorithms. Even for something as simple as quicksort there is enough natural variation for a class of 30 maybe even 100 (if no references can be used). Anything more complex and even with references it should be unique.

voiper1 4y ago

It's not _just_ trying to be lenient and offer a second chance - it's a way to catch more cheaters. "Turn yourself in and we'll go easy on you... because we might not catch you."

Claude_Shannon 4y ago

I have the same policy at my uni in Poland. Admit to cheating without being called out? Depending on the professor mood, they either allowed you to retake exam (though the best grade you'd get was the lowest passing one), or you just fail the course and try again next year. Both they had less paperwork, and well, they wouldn't report that person.