3 ms·

Hi everyone, article author here. I was also interviewed by The Register about this topic: https://www.theregister.com/2022/08/19/copilot_github_students/ https

Hi everyone, article author here. I was also interviewed by The Register about this topic: https://www.theregister.com/2022/08/19/copilot_github_students/ https://www.theregister.com/2022/08/19/copilot_github_studen....

stew-j 4y ago
I might have missed it today in your articles or comments here--it's been a hectic day--but has there been some study of just how different code would be given that the students are using the same text from questions? Is there randomization intrinsic to Copilot, or is it just because minor variations in textual input causes code to be so different? My wife taught CS, she did catch cheaters pre-Copilot, and my first thought it that she probably would enter test questions and print out a reference sheet for Copilot generated results.
- emeryberger 4y ago
  I haven't seen a study yet, but yes, Copilot deliberately (AFAICT) incorporates randomness. For longer code fragments, I believe this would thwart most plagiarism detectors. I'll note that defeating plagiarism detectors is easier than many people think. I and one of my students wrote a paper on an automatic technique to defeat plagiarism detectors, and it was highly effective: "Mossad: Defeating Software Plagiarism Detection", Devore-McDonald and Berger, OOPSLA 2020 (talk and paper link here: https://2020.splashcon.org/details/splash-2020-oopsla/14/Mossad-Defeating-Software-Plagiarism-Detection https://2020.splashcon.org/details/splash-2020-oopsla/14/Mos...).
  - stew-j 4y ago
    From the OOPSLA paper abstract: > techniques inspired by genetic programming with domain-specific knowledge to effectively undermine plagiarism detectors. My creaky memory of CS Theory makes me suspect that, ultimately, Mr. Turing's halting problem may make resistance to cheating futile. [1] It's interesting that the paper is in OOPSLA, I used to follow the work from it more, especially when I listened to IEEE's Software Engineering Radio. Thanks for your work in this area and your reply. DieHard and DieHarder are funny, creative names for fault tolerant memory managers. It's also nice to see a fellow humorist. At least we can rest assured that cheaters get what they deserve, like my classmate at Columbia. [2] Regards. [1] > He, for one, welcomes our new AI overlords. Yes, let's suggest a Borg-like partnership with them. I'd be willing to drive "self-driving-car" tow truck--if they haven't invented a self-driving one yet. [2] https://news.ycombinator.com/item?id=32532896 https://news.ycombinator.com/item?id=32532896
Barrin92 4y ago
>"Copilot is different, he said, "It actually generates novel solutions. Not like they're super-crazy, sophisticated, genius solutions. But it makes new solutions that are superficially different enough that they plausibly could have come from a student."" virtually every cheating student has always slightly altered their solutions. At best copilot saves you the five minutes of Googling and refactoring. For the committed cheater who wants to avoid a week's worth of homework and learning that's hardly meaningful. And in fact probably not even that because you still need to actually read what copilot outputs because it makes obvious bot-like mistakes quite often.

stew-j 4y ago

I might have missed it today in your articles or comments here--it's been a hectic day--but has there been some study of just how different code would be given that the students are using the same text from questions? Is there randomization intrinsic to Copilot, or is it just because minor variations in textual input causes code to be so different? My wife taught CS, she did catch cheaters pre-Copilot, and my first thought it that she probably would enter test questions and print out a reference sheet for Copilot generated results.

emeryberger 4y ago
I haven't seen a study yet, but yes, Copilot deliberately (AFAICT) incorporates randomness. For longer code fragments, I believe this would thwart most plagiarism detectors. I'll note that defeating plagiarism detectors is easier than many people think. I and one of my students wrote a paper on an automatic technique to defeat plagiarism detectors, and it was highly effective: "Mossad: Defeating Software Plagiarism Detection", Devore-McDonald and Berger, OOPSLA 2020 (talk and paper link here: https://2020.splashcon.org/details/splash-2020-oopsla/14/Mossad-Defeating-Software-Plagiarism-Detection https://2020.splashcon.org/details/splash-2020-oopsla/14/Mos...).
- stew-j 4y ago
  From the OOPSLA paper abstract: > techniques inspired by genetic programming with domain-specific knowledge to effectively undermine plagiarism detectors. My creaky memory of CS Theory makes me suspect that, ultimately, Mr. Turing's halting problem may make resistance to cheating futile. [1] It's interesting that the paper is in OOPSLA, I used to follow the work from it more, especially when I listened to IEEE's Software Engineering Radio. Thanks for your work in this area and your reply. DieHard and DieHarder are funny, creative names for fault tolerant memory managers. It's also nice to see a fellow humorist. At least we can rest assured that cheaters get what they deserve, like my classmate at Columbia. [2] Regards. [1] > He, for one, welcomes our new AI overlords. Yes, let's suggest a Borg-like partnership with them. I'd be willing to drive "self-driving-car" tow truck--if they haven't invented a self-driving one yet. [2] https://news.ycombinator.com/item?id=32532896 https://news.ycombinator.com/item?id=32532896

Barrin92 4y ago

>"Copilot is different, he said, "It actually generates novel solutions. Not like they're super-crazy, sophisticated, genius solutions. But it makes new solutions that are superficially different enough that they plausibly could have come from a student."" virtually every cheating student has always slightly altered their solutions. At best copilot saves you the five minutes of Googling and refactoring. For the committed cheater who wants to avoid a week's worth of homework and learning that's hardly meaningful. And in fact probably not even that because you still need to actually read what copilot outputs because it makes obvious bot-like mistakes quite often.