7 ms·

Adjunct instructor here. Before I came into the program, cheating was on the rise. When given programming assignments, students would go to Geeks for geeks and

Adjunct instructor here. Before I came into the program, cheating was on the rise. When given programming assignments, students would go to Geeks for geeks and download solutions, some of them mostly correct, and submit those for credit. One remedy was to try to make programming assignments harder and harder. Eventually, they got so hard that we were asking people new to computer science to implement a full arbitrary length integer calculator using nothing but a single tape Turing machine (in Java, not BF). Eventually, some of us came to the realization that you can never prove that somebody is cheating. People have been known to hire tutors to do their assignments for them. There is just no end. As a result, we evaluate students on four dimensions. Programming assignments,* homework assignments, but also class discussions and group participation. Those last two count for a small, but non-trivial percentage of the grade, and are usually enough to separate and identify those who understand what they are doing, from those who are just "following along" solutions that they find on the internet. * one addition to programming assignments includes an analysis write up. Tell me in human words what is happening. Why do you see that effect. What is the running time. And separately, comment your code to tell me how it works. Those two parts count significantly towards the grade. Any other professors here, what have you found that works?

leeoniya 4y ago
> one addition to programming assignments includes an analysis write up. when i was interviewing candidates this was the fastest way to filter those who understood and those who didnt. ask someone to talk you through how to solve some real-life problem. to look at a stack trace and describe what they see, or to run a profile and trace through the cause of a hotspot/contention. show them an issue and have them live debug to root-cause and fix. it's okay if they use the internet, SO, etc. -- that's how we all do it. see what they have to look up! just listening to the amount of depth someone can verbally communicate during novel problem solving (including additional questions they ask you) turns a 60min interview that wastes time into a 10min one that tells you if you can move on. a favorite one of mine was to ask the candidate to describe in as much detail as possible what happens between a keystroke typed into a google search box and the search results appearing. the diversity of replies is facinating. some will tell you "google returns results from its database", others will ask you if you want them to first describe how the keyboard works at an electrical level through the USB or bluetooth driver stack. "Never memorize something that you can look up." --someone smart
- nkingsy 4y ago
  Ugh I had someone ask me to debug a verbally communicated error message in an interview. Like they picked some random port configuration issue that had stumped them for days in the past and thought cool let’s remove the internet, the command line, and ask people to solve it on the spot. I offered plenty of plans for how I would go about debugging, but I didn’t know the one simple trick. I agree with trying to use challenges that are closer to the real work, but it’s really hard to do that without over-testing domain knowledge. For instance, I ask candidates to do some asynchronous control flow. These are all candidates with js listed as their best language and I offer to let them look things up or to show them the promise apis they need, but a certain percent just refuse to engage with the problem because they don’t have the domain knowledge and seem to feel they’re being tested unfairly. The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
  - leeoniya 4y ago
    > Ugh I had someone ask me to debug a verbally communicated error message in an interview. i dont mean that the problem is only verbally communicated. i mean that the debugging process the candidate does is verbally communicated. in the scenario you're describing you'd be sitting at that machine with access to the internet on another machine. like, you know, in real life. > The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks. async stuff in js is pretty good, but it's also easy to go too deep on it with some bizzarely poorly architected code. many js devs still fail to grasp all the implications of closures, or how to avoid memory leaks, or how to work with the GC rather than against it.
    - deleted 4y ago
      [deleted]
- matheusmoreira 4y ago
  That's a great way to evaluate someone! Individualized and personal, uses concrete problems, no artificial restrictions. It's a shame our mass education systems cannot use it. They simply cannot apply such a humane method to hundreds or thousands of students. They are reduced to applying their bullshit test questions because it's the best they can come up with at their scale.
mdaniel 4y ago
> but also class discussions and group participation I really, really hated those parts of school, second only to group projects. I do appreciate why teachers want that, and I hear you about how it can surface those who are cheating, but I do wonder if it's causing the class to suffer for the alleged misdeeds of a few
- ethbr0 4y ago
  I disliked them in school, for the usual reasons, but found them the most useful exercises for developing career skills. No person an island. And if they are, I don't want to be on a work team with them.
  - qsort 4y ago
    School assignments are so different from the dynamics of a real workplace that I find this point very moot. Always hated them with a passion, but otoh I always enjoyed team math and programming competitions, situations where you have to do stuff that's actually hard, work with people you actually respect and trust them to do their job.
    - NikolaNovak 4y ago
      That's the thing. For me at least, the dynamics of "School Group Project" are basically completely different and separate from "Work team". I happen to do well in both, but I do not enjoy both: the motivations and structure and dynamics and goals and timelines of "School Group Project" are so much more artificial and ultimately pointless, and obviously to all from the start. Same things with discussions - I happen to be an engaged student, usually front row, hand always up, discussing with instructor and team and colleagues and everybody. I like to be engaged and figure things out together. But I'll never be half the developer than my colleague who barely speaks a word unless asked. He's friendly, meek, polite, and excellent team player and developer - he just does not initiate conversations, especially in group settings. I can imagine he'd have a nice big 0 in that category if that was a grade criteria. Which is not to criticize the professors who try to use group discussions; just to point out, this is not a solved problem at scale any more than interviewing/hiring. There's too much humanity and too few absolutes :)
philipswood 4y ago
When playing with GPT-3, I've found it to be impressively good at explaining what code does in English...
- drran 4y ago
  When teachers require recording of timelapses, we will have AI which will be able to generate text and a video of a user typing it. IMHO, they should start to require timelapses right now.
  - NoraCodes 4y ago
    That requires either trusting that students haven't edited out the evidence of cheating, which is pointless, or forcing them to use specific, proprietary software to record the video, which is unacceptable.
    - drran 4y ago
      Timelapse can be recorded online, just like a security camera with upload to cloud does, so it will be impossible to edit later. Cheating will be visible because of sudden changes in text of the program.
      - NoraCodes 4y ago
        That is, as I said, unacceptable. Among other things, it means students can't use their editor of choice, and students with disabilities who use assistive technologies will be much more likely to be flagged as cheaters. It also won't stop anyone from cheating by doing the exercise with Copilot and typing it up a second time for the time lapse.
        drran 4y ago
        Examples: https://www.youtube.com/watch?v=YCutBaSt6tM https://www.youtube.com/watch?v=YCutBaSt6tM https://www.youtube.com/watch?v=DmaKc0I1VAo https://www.youtube.com/watch?v=DmaKc0I1VAo https://www.youtube.com/watch?v=N14T_4gUH-c https://www.youtube.com/watch?v=N14T_4gUH-c
aaomidi 4y ago
I do want to say that, those criteria exclude neurodivergent folk (who for many reasons may have never even been diagnosed). Honestly I think moving away from grades is going to be the way forward. Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.
- nyanpasu64 4y ago
  Mass-produced education is like a train, once you fall off you'll get hurt trying to get back on. I've come to the realization that given the inhumane pace, difficult and poorly-taught curriculum, and ambiguous/out-of-scope homework assignments in a top-tier university's CS program, the majority of my university classes were either not worth taking, or I'd be better off self-studying if I found an intrinsically rewarding project (definitely less panic attacks and psychological trauma, sometimes even similar or deeper learning). Perhaps I'd have a better time in a less prestigious university that doesn't see its purpose as weeding out and breaking lesser students.
- DiggyJohnson 4y ago
  Do you also believe that we should move away from evaluation entirely? Genuinely asking.
  - aaomidi 4y ago
    I think during education yes, during execution of the skills learned in education - not really but also depends. A lot of people hate how tech interviews are done, right? Cause they have nothing to do with the job the majority of the time. Those are places where I can see improvements. However, some jobs do genuinely need you to interact with a diverse set of people all the time, for those you need to make sure for the sake of your company & the candidate that they match the criteria. Other jobs, you need only maintain strong relations with small groups, and the requirements for those are a lot easier to hit - even the most introverted people can do well in smaller groups that don't change a lot. I do really think a lot of evaluations that happen during education though makes a confrontational relationship with learning. You're constantly judged about your _ability to learn_and that has far reaching impacts beyond your first twoish decades of your life. You get scared of making mistakes, you get scared of taking risks, etc. So, evaluate when it's important for the function & safety of the job, and the candidate. Skip evaluation when its evaluation for the sake of evaluation.
jstanley 4y ago
> One remedy was to try to make programming assignments harder and harder. How is that going to stop cheating? Surely that will just make people more inclined to cheat, since they have less chance of solving it on their own.
- gfaster 4y ago
  It would prevent people from simply going and downloading solutions from the internet. A lot of easier assignments are fairly well treaded and already have solutions floating around.
clusterhacks 4y ago
Hi, sometime adjunct here. Cheating on programming assignments has been rampant forever at every undergrad institution I have experience with and somewhat present among graduate students. My experience spans about 25 years in that space now. When I taught an introductory class, I gave open-book exams with no laptops or phones allowed. About 1/3 of the class was unable to write a syntactically correct for loop in Python despite our textbook being an introductory Python-based book chock full of examples. It was pretty clear that a subset of students were either working together on project assignments or out-and-out having someone else do the assignments for them. I mainly compensated for this by having a large part of the grade being a 1-1 meeting with me to talk me through the code. That and the exam had the effect of actually making cheating somewhat less worthwhile. But this approach simply doesn't scale these days - my max class was around 32-33 students and the time I spent meeting with students was insane. I haven't taught in a few years and understand that class sizes of 150+ are not uncommon. I could have never used the same approach with that many students. I probably would have doubled-down on exams and made exam length darn near impossible to finish without actually knowing the material well enough to do without referring to the textbook. I was teaching more from a practice-based viewpoint so mostly I came up with "weird" projects that mirrored problems I spend time on (data cleaning, using existing libraries to do neat little things, and also having students pick a personal project to implement that I helped them scope appropriately). We bitch and moan about interview whiteboarding but given grade inflation its kind of hard to trust university credentials. Grade inflation was kind of disheartening in that the worst students didn't really get a poor grade. But I also didn't have any problems with the top 1/3 of the class getting a very high grade - these students were motivated, understood the material, and often impressed me with where they got in a single semester.
- SoftTalker 4y ago
  I recently worked at a large state university. I remember a conversation with an instructor in a master-level operating systems course. One of the assignments was to implement a simple filesystem. The amount of cheating was insane. He started academic dishonesty proceedings against many of the students but the department pressured him to "work it out." He had to basically interview each student individually and ask for an explanation of the code. Most could not explain what their code was doing (because they didn't write it). They were all given an opportunity to resubmit original work. Many could not do it, because they simply didn't understand the basics of programming, or the language they were using, or how computers even worked. These were Masters students.
sireat 4y ago
I teach data structures and algorithms among other subjects. My assignments are similar to everyone else's (CLRS, JeffE, Leetcode) yet different enough that simply copy and pasting will not work. All I ask is that they comment their code, analyze complexity and cite sources. I've been using copilot for the last year in my live coding sessions. It saves so much time (especially in comments). If students want to use Copilot I do not see a problem. It is just another tool. Copilot is near useless without decent programming knowledge. 80% of time it is fantastic. About 10% of time Copilot gives you subpar snippets. O(n2) instead of O(n) etc. Then 10% of time it just gives you wrong snippets. Just like GPT-3, once you go beyond a paragraph it loses context.
emeryberger 4y ago
I think grading on multiple dimensions makes sense, though scaling is a challenge. As for analysis write-up: turns out, language models can also explain what the code does really well, in convincing vernacular. They generate reasonable comments. https://twitter.com/emeryberger/status/1560618825208320001?s=20&t=_5G6G7bLzc_Y-_xuEQwLkg https://twitter.com/emeryberger/status/1560618825208320001?s... As an illustrative example, I just typed in a little test into Copilot. I only typed the first line and the text that starts with "a paragraph explaining...", and typed a "#" for each comment. Copilot filled in the rest. def compute_sum_recursively(n): """A paragraph explaining what this function does and why, and explaining its running time: This function computes the sum of the first n natural numbers. The running time of this function is O(n). """ # Base case: n = 0 if n == 0: # The sum of the first 0 natural numbers is 0 return 0 else: # Recursive case: n > 0 # The sum of the first n natural numbers is n + the sum of the first n-1 natural numbers return n + compute_sum_recursively(n-1)
- contravariant 4y ago
  Now I'm curious if it can find the O(1) version of that same function.
  - emeryberger 4y ago
    Yes, it can. This is in the original article. Look for “output_sum_closed_form”. All I typed in was the function name.
scott00 4y ago
Not a professor, but my undergrad classes were generally graded as 10-15% homework and 85-90% exams. With grading like that cheating on the homework is basically pointless, so few did it.

leeoniya 4y ago

> one addition to programming assignments includes an analysis write up. when i was interviewing candidates this was the fastest way to filter those who understood and those who didnt. ask someone to talk you through how to solve some real-life problem. to look at a stack trace and describe what they see, or to run a profile and trace through the cause of a hotspot/contention. show them an issue and have them live debug to root-cause and fix. it's okay if they use the internet, SO, etc. -- that's how we all do it. see what they have to look up! just listening to the amount of depth someone can verbally communicate during novel problem solving (including additional questions they ask you) turns a 60min interview that wastes time into a 10min one that tells you if you can move on. a favorite one of mine was to ask the candidate to describe in as much detail as possible what happens between a keystroke typed into a google search box and the search results appearing. the diversity of replies is facinating. some will tell you "google returns results from its database", others will ask you if you want them to first describe how the keyboard works at an electrical level through the USB or bluetooth driver stack. "Never memorize something that you can look up." --someone smart

nkingsy 4y ago
Ugh I had someone ask me to debug a verbally communicated error message in an interview. Like they picked some random port configuration issue that had stumped them for days in the past and thought cool let’s remove the internet, the command line, and ask people to solve it on the spot. I offered plenty of plans for how I would go about debugging, but I didn’t know the one simple trick. I agree with trying to use challenges that are closer to the real work, but it’s really hard to do that without over-testing domain knowledge. For instance, I ask candidates to do some asynchronous control flow. These are all candidates with js listed as their best language and I offer to let them look things up or to show them the promise apis they need, but a certain percent just refuse to engage with the problem because they don’t have the domain knowledge and seem to feel they’re being tested unfairly. The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks.
- leeoniya 4y ago
  > Ugh I had someone ask me to debug a verbally communicated error message in an interview. i dont mean that the problem is only verbally communicated. i mean that the debugging process the candidate does is verbally communicated. in the scenario you're describing you'd be sitting at that machine with access to the internet on another machine. like, you know, in real life. > The problem does a really good job of showing a candidates grasp of all the tricky parts of js, so I keep using it despite the drawbacks. async stuff in js is pretty good, but it's also easy to go too deep on it with some bizzarely poorly architected code. many js devs still fail to grasp all the implications of closures, or how to avoid memory leaks, or how to work with the GC rather than against it.
  - deleted 4y ago
    [deleted]
matheusmoreira 4y ago
That's a great way to evaluate someone! Individualized and personal, uses concrete problems, no artificial restrictions. It's a shame our mass education systems cannot use it. They simply cannot apply such a humane method to hundreds or thousands of students. They are reduced to applying their bullshit test questions because it's the best they can come up with at their scale.

mdaniel 4y ago

> but also class discussions and group participation I really, really hated those parts of school, second only to group projects. I do appreciate why teachers want that, and I hear you about how it can surface those who are cheating, but I do wonder if it's causing the class to suffer for the alleged misdeeds of a few

ethbr0 4y ago
I disliked them in school, for the usual reasons, but found them the most useful exercises for developing career skills. No person an island. And if they are, I don't want to be on a work team with them.
- qsort 4y ago
  School assignments are so different from the dynamics of a real workplace that I find this point very moot. Always hated them with a passion, but otoh I always enjoyed team math and programming competitions, situations where you have to do stuff that's actually hard, work with people you actually respect and trust them to do their job.
  - NikolaNovak 4y ago
    That's the thing. For me at least, the dynamics of "School Group Project" are basically completely different and separate from "Work team". I happen to do well in both, but I do not enjoy both: the motivations and structure and dynamics and goals and timelines of "School Group Project" are so much more artificial and ultimately pointless, and obviously to all from the start. Same things with discussions - I happen to be an engaged student, usually front row, hand always up, discussing with instructor and team and colleagues and everybody. I like to be engaged and figure things out together. But I'll never be half the developer than my colleague who barely speaks a word unless asked. He's friendly, meek, polite, and excellent team player and developer - he just does not initiate conversations, especially in group settings. I can imagine he'd have a nice big 0 in that category if that was a grade criteria. Which is not to criticize the professors who try to use group discussions; just to point out, this is not a solved problem at scale any more than interviewing/hiring. There's too much humanity and too few absolutes :)

philipswood 4y ago

When playing with GPT-3, I've found it to be impressively good at explaining what code does in English...

drran 4y ago
When teachers require recording of timelapses, we will have AI which will be able to generate text and a video of a user typing it. IMHO, they should start to require timelapses right now.
- NoraCodes 4y ago
  That requires either trusting that students haven't edited out the evidence of cheating, which is pointless, or forcing them to use specific, proprietary software to record the video, which is unacceptable.
  - drran 4y ago
    Timelapse can be recorded online, just like a security camera with upload to cloud does, so it will be impossible to edit later. Cheating will be visible because of sudden changes in text of the program.
    - NoraCodes 4y ago
      That is, as I said, unacceptable. Among other things, it means students can't use their editor of choice, and students with disabilities who use assistive technologies will be much more likely to be flagged as cheaters. It also won't stop anyone from cheating by doing the exercise with Copilot and typing it up a second time for the time lapse.
      - drran 4y ago
        Examples: https://www.youtube.com/watch?v=YCutBaSt6tM https://www.youtube.com/watch?v=YCutBaSt6tM https://www.youtube.com/watch?v=DmaKc0I1VAo https://www.youtube.com/watch?v=DmaKc0I1VAo https://www.youtube.com/watch?v=N14T_4gUH-c https://www.youtube.com/watch?v=N14T_4gUH-c

aaomidi 4y ago

I do want to say that, those criteria exclude neurodivergent folk (who for many reasons may have never even been diagnosed). Honestly I think moving away from grades is going to be the way forward. Part of this is also making university, etc cheaper so you’re not financially incentivized to not take a class one, or multiple more times.

nyanpasu64 4y ago
Mass-produced education is like a train, once you fall off you'll get hurt trying to get back on. I've come to the realization that given the inhumane pace, difficult and poorly-taught curriculum, and ambiguous/out-of-scope homework assignments in a top-tier university's CS program, the majority of my university classes were either not worth taking, or I'd be better off self-studying if I found an intrinsically rewarding project (definitely less panic attacks and psychological trauma, sometimes even similar or deeper learning). Perhaps I'd have a better time in a less prestigious university that doesn't see its purpose as weeding out and breaking lesser students.
DiggyJohnson 4y ago
Do you also believe that we should move away from evaluation entirely? Genuinely asking.
- aaomidi 4y ago
  I think during education yes, during execution of the skills learned in education - not really but also depends. A lot of people hate how tech interviews are done, right? Cause they have nothing to do with the job the majority of the time. Those are places where I can see improvements. However, some jobs do genuinely need you to interact with a diverse set of people all the time, for those you need to make sure for the sake of your company & the candidate that they match the criteria. Other jobs, you need only maintain strong relations with small groups, and the requirements for those are a lot easier to hit - even the most introverted people can do well in smaller groups that don't change a lot. I do really think a lot of evaluations that happen during education though makes a confrontational relationship with learning. You're constantly judged about your _ability to learn_and that has far reaching impacts beyond your first twoish decades of your life. You get scared of making mistakes, you get scared of taking risks, etc. So, evaluate when it's important for the function & safety of the job, and the candidate. Skip evaluation when its evaluation for the sake of evaluation.

jstanley 4y ago

> One remedy was to try to make programming assignments harder and harder. How is that going to stop cheating? Surely that will just make people more inclined to cheat, since they have less chance of solving it on their own.

gfaster 4y ago
It would prevent people from simply going and downloading solutions from the internet. A lot of easier assignments are fairly well treaded and already have solutions floating around.

clusterhacks 4y ago

Hi, sometime adjunct here. Cheating on programming assignments has been rampant forever at every undergrad institution I have experience with and somewhat present among graduate students. My experience spans about 25 years in that space now. When I taught an introductory class, I gave open-book exams with no laptops or phones allowed. About 1/3 of the class was unable to write a syntactically correct for loop in Python despite our textbook being an introductory Python-based book chock full of examples. It was pretty clear that a subset of students were either working together on project assignments or out-and-out having someone else do the assignments for them. I mainly compensated for this by having a large part of the grade being a 1-1 meeting with me to talk me through the code. That and the exam had the effect of actually making cheating somewhat less worthwhile. But this approach simply doesn't scale these days - my max class was around 32-33 students and the time I spent meeting with students was insane. I haven't taught in a few years and understand that class sizes of 150+ are not uncommon. I could have never used the same approach with that many students. I probably would have doubled-down on exams and made exam length darn near impossible to finish without actually knowing the material well enough to do without referring to the textbook. I was teaching more from a practice-based viewpoint so mostly I came up with "weird" projects that mirrored problems I spend time on (data cleaning, using existing libraries to do neat little things, and also having students pick a personal project to implement that I helped them scope appropriately). We bitch and moan about interview whiteboarding but given grade inflation its kind of hard to trust university credentials. Grade inflation was kind of disheartening in that the worst students didn't really get a poor grade. But I also didn't have any problems with the top 1/3 of the class getting a very high grade - these students were motivated, understood the material, and often impressed me with where they got in a single semester.

SoftTalker 4y ago
I recently worked at a large state university. I remember a conversation with an instructor in a master-level operating systems course. One of the assignments was to implement a simple filesystem. The amount of cheating was insane. He started academic dishonesty proceedings against many of the students but the department pressured him to "work it out." He had to basically interview each student individually and ask for an explanation of the code. Most could not explain what their code was doing (because they didn't write it). They were all given an opportunity to resubmit original work. Many could not do it, because they simply didn't understand the basics of programming, or the language they were using, or how computers even worked. These were Masters students.

sireat 4y ago

I teach data structures and algorithms among other subjects. My assignments are similar to everyone else's (CLRS, JeffE, Leetcode) yet different enough that simply copy and pasting will not work. All I ask is that they comment their code, analyze complexity and cite sources. I've been using copilot for the last year in my live coding sessions. It saves so much time (especially in comments). If students want to use Copilot I do not see a problem. It is just another tool. Copilot is near useless without decent programming knowledge. 80% of time it is fantastic. About 10% of time Copilot gives you subpar snippets. O(n2) instead of O(n) etc. Then 10% of time it just gives you wrong snippets. Just like GPT-3, once you go beyond a paragraph it loses context.

emeryberger 4y ago

I think grading on multiple dimensions makes sense, though scaling is a challenge. As for analysis write-up: turns out, language models can also explain what the code does really well, in convincing vernacular. They generate reasonable comments. https://twitter.com/emeryberger/status/1560618825208320001?s=20&t=_5G6G7bLzc_Y-_xuEQwLkg https://twitter.com/emeryberger/status/1560618825208320001?s... As an illustrative example, I just typed in a little test into Copilot. I only typed the first line and the text that starts with "a paragraph explaining...", and typed a "#" for each comment. Copilot filled in the rest. def compute_sum_recursively(n): """A paragraph explaining what this function does and why, and explaining its running time: This function computes the sum of the first n natural numbers. The running time of this function is O(n). """ # Base case: n = 0 if n == 0: # The sum of the first 0 natural numbers is 0 return 0 else: # Recursive case: n > 0 # The sum of the first n natural numbers is n + the sum of the first n-1 natural numbers return n + compute_sum_recursively(n-1)

contravariant 4y ago
Now I'm curious if it can find the O(1) version of that same function.
- emeryberger 4y ago
  Yes, it can. This is in the original article. Look for “output_sum_closed_form”. All I typed in was the function name.

scott00 4y ago

Not a professor, but my undergrad classes were generally graded as 10-15% homework and 85-90% exams. With grading like that cheating on the homework is basically pointless, so few did it.