4 ms·

CS Professor here: just yesterday I did the discussion of a course projects' (Parallel Computing), and one of the three groups that I did yesterday have clearly

CS Professor here: just yesterday I did the discussion of a course projects' (Parallel Computing), and one of the three groups that I did yesterday have clearly gone the ChatGPT way. They couldn't even understand the choices the LLM made regarding the architecture, etc. The way to "catch" these students is similar to what we did in the past when students copied from other students which is "to give them rope to hang" - ask for clarifications until they follow unintended paths that lead nowhere. To fellow professors, when you're suspicious my suggestion is to appeal to their honesty (like "let's be honest, how much of this code is yours, and how much is ChatGPT's?") and offer some empathy and understanding (like understanding they may had multiple deadlines in the same week, etc.). Nevertheless, don't miss the chance to give them the lesson on how is the correct way of doing things. The way to catch these students is to find the same signs of yesteryear copying from other students (which in essence is what copying from an LLM is, although the number has increased because they found us professors unprepared for the volume). The other two groups also used LLM but in a high-level and architectural way. They were clearly responsible for the code (even if they didn't wrote it 100% manually) and could explain their reasoning and strategies used to solve the problems. Me and my colleagues still have a lot of projects to review, and I asked them to keep the score of the number of projects like these, but so far, the score is 1 in 3 (33%).

kuschku 4mo ago
> The other two groups also used LLM but in a high-level and architectural way. Sounds more like the score is 3/3 (100%) Would you have accepted them cooy-pasting code from libraries together to build their project? If not, why is using LLM generated code different?
- yubblegum 4mo ago
  He explains why in his comment. Read it again, carefully. Or ask an LLM for an "explainer"..
- jventura 4mo ago
  > Would you have accepted them cooy-pasting code from libraries together to build their project? Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc. In this case, the students had to invent multiple strategies to solve a specific problem. The "successful" groups did a mix of generated and hand-crafted code (don't know percentages), implemented different strategies and knew their plus and minuses, could change the code in a timely manner to accommodate some of my requests, etc. The "unsuccessful" group couldn't do any of that. I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students. But in reality I've been slowly transitioning from group projects (for a subset of the grade) to "practical tests", where they must implement a significant subset of a larger project in a 2h class. Still experimenting though.
  - cushychicken 4mo ago
    > Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc. This is a good principle to maintain, I think. I'm not a professor, but I manage a team of about a dozen people. The maxim I have is: "You're responsible for anything that hits git." Don't care if the LLM generated it, or the LLM told you if it's a good idea. If you commit it, you are endorsing it as a good idea - so you're the one I'm going to ask about it. I see the same principle at work in your pedagogy. > I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students. This hits. Especially this part: > and really, what could I do if I were? My completely unsolicited opinion: you're doing a responsible thing by teaching these students how to use AI as a reference, and keeping them honest about not using it as a substitute for their own critical thinking.
  - lokar 4mo ago
    I'd say most of my CS classes (in the 90s) had us write out code by hand, on paper, during in-class exams. It was fine.
    - dysoco 4mo ago
- newswangerd 4mo ago
  Using AI to inform architecture doesn’t seem so different from googling architecture in this case. Architectural patterns are mostly well understood and well documented these days and are something that you could piece together via Google search pre AI. The thing that AI brings to the table that wasn’t google able in the past is code generation. Previously you had to understand the architecture patterns to implement them yourself, but now the AI can just do it for you.
bo1024 4mo ago
What is the policy and guidance you gave the students regarding LLM use?
- jventura 4mo ago
  This is all so new, and caught us completely unprepared that there's no official university-level policies. Most of us are still navigating the waters and seeing what works and what doesn't work anymore. I have colleagues that are teaching for more than 30 years, few years away from retirement, who suddenly have been confronted with a new way of doing things. Those are the ones that are still insisting on doing practical projects, etc. I've only been doing this for 20 years, and I'm quite lazy (worked previously as software engineer), so I've moved to those practical tests. I guess that there should probably exist a class or workshop to teach these students how to use LLMs effectively, but as I said, this technology and its implications is quite new. Personally, what I did was to give them the "lecture" in the line of that they do not understand what the machine has generated, that is not the way a true engineer does, try to do some parallel with things like an LLM designing a bridge and civil engineers building that bridge, and a fatal flaw collapsing the all thing, etc. In other words, we do not have a formal system in place, it's all talking and convincing them. Obviously it's a big enough problem that should deserve more investment in solutions, but we are all overwhelmed by other tasks. Maybe LLM studios should be held responsible for all these "disruptions" and provide solutions to problems they created! :)
LovelyButterfly 4mo ago
As a professor, what's you opinion regard the Socratic method with LLMs? Would that be preferable over simply "give me the answer" prompt?
- jventura 4mo ago
  Everything that provides students with a workflow to think and to try to find solutions to a problem is much better than giving the answer directly! Unfortunately there will always be students that prefer to take the shortcut.. How could we "force" the students to use an LLM that confronted their doubts with more questions? We could tell them to start each chat with a specific prompt (to use the socratic method, etc), but they could eventually jail-break it.. But nevertheless, I like your idea! This is something that a document highlighting methodologies for students on how to use LLMs effectively could/should contain..
  - e-topy 4mo ago
    I'm curious on your opinion, as a professor, how should universities (not individual professors) adapt to LLMs? As an undergrad, I hope schools move to educating students to use LLMs in a more responsible way. You can't put the genie back in the bottle, and resisting progress is futile, might as well use the tool we now have to help students learn even faster and better (e.g., making feedback instant and not answers, helping digest or split up material, checking answers). I know opinions about AI at (not only at) my faculty are very mixed, but I think the answer is going to be in the rational mean, just like how technorealism reacted to the internet[0]. In our last program board sitting, some teachers said that they think programming as a job will be completely irrelevant in two years, while other pushed for more adoption. And meanwhile I know of some students that are basically only passing because of LLMs, and it's bad, like "leaving claude output in markdown files and finished source code on the faculty server in /tmp because opencode did so" bad. And our first year classes completely prohibit even sharing tests or talking about the solutions, which in my opinion a) makes people extremely asocial and atomized b) doesn't prepare students for real life c) promotes dishonesty. Still, I think our university's thinking is in a stalemate, not wanting pure AI output and useless students, while also wanting to move with the times, and I doubt it's the only one. [0]: https://web.archive.org/web/20081009111415/https://artefaktum.hu/it2/millarch.html https://web.archive.org/web/20081009111415/https://artefaktu... (absolutely amazing read, recommend it)
rahimnathwani 4mo ago
So all groups outsourced their thinking to an LLM. Will they have learned anything they can apply to different types of projects in future?
- jventura 4mo ago
  Not in the same level, see my other comments..
  - rahimnathwani 4mo ago
    I think we're using two standards without naming the difference. - "Responsible for the code", i.e. understands it, can explain it, can modify it on request, is the right bar for a working engineer using an LLM to go faster. But it's the wrong bar for a student, because in a course the deliverable isn't the point, the practice is. I presume that as course like 'Parallel Computing' exists to build the habits of reasoning about decomposition, contention, race conditions, strategy tradeoffs. Those habits come from doing the reps, the same way you don't learn to write by reading good prose. Your two "successful" groups outsourced exactly the reps the course is (or should be) there to build. Being able to follow and tweak a solution proves comprehension, but comprehension is a much lower bar than producing the solution. Your students may understand this particular solution well enough to explain it. But did they learn enough that they could tackle a different problem with a different shape, without using an LLM?
    - jventura 4mo ago
      I'm not a native EN speaker (I'm Portuguese), therefore a lot is lost in my attempt to find the right words. I'm writing these words without any use of AI or even a sentence-checker (only the spell-checker in Firefox). Of course that the "succesfull" groups implemented their solutions, tweaked them and, tried their best to escape from race conditions and were able to talk about the performance/memory/etc. tradeoffs. We have a grid to grade for those and other items. They did not outsource all of their thinking to an LLM.. That is what I mean as taking responsability for their code, maybe not the correct word, but the one that came to my head when I wrote it..
estetlinus 4mo ago
I did a course of parallell computing 6 years ago. I still dont understand anything about it. Probably would if I had used an LLM, since I would have used it as a private tutor.
ygouzerh 4mo ago
How do you see impact of using LLM on the teaching content? (genuine curiosity) In the industry, almost everyone here is doing the exact same thing, outsourcing more and more of their thinking to LLM: so even if students learned how to manually write code, they will probably loose it later on (happening to us already, mostly can be seen when people are working on new solutions/frameworks... they now have the same issue than students).

kuschku 4mo ago

> The other two groups also used LLM but in a high-level and architectural way. Sounds more like the score is 3/3 (100%) Would you have accepted them cooy-pasting code from libraries together to build their project? If not, why is using LLM generated code different?

yubblegum 4mo ago
He explains why in his comment. Read it again, carefully. Or ask an LLM for an "explainer"..
jventura 4mo ago
> Would you have accepted them cooy-pasting code from libraries together to build their project? Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc. In this case, the students had to invent multiple strategies to solve a specific problem. The "successful" groups did a mix of generated and hand-crafted code (don't know percentages), implemented different strategies and knew their plus and minuses, could change the code in a timely manner to accommodate some of my requests, etc. The "unsuccessful" group couldn't do any of that. I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students. But in reality I've been slowly transitioning from group projects (for a subset of the grade) to "practical tests", where they must implement a significant subset of a larger project in a 2h class. Still experimenting though.
- cushychicken 4mo ago
  > Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc. This is a good principle to maintain, I think. I'm not a professor, but I manage a team of about a dozen people. The maxim I have is: "You're responsible for anything that hits git." Don't care if the LLM generated it, or the LLM told you if it's a good idea. If you commit it, you are endorsing it as a good idea - so you're the one I'm going to ask about it. I see the same principle at work in your pedagogy. > I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students. This hits. Especially this part: > and really, what could I do if I were? My completely unsolicited opinion: you're doing a responsible thing by teaching these students how to use AI as a reference, and keeping them honest about not using it as a substitute for their own critical thinking.
- lokar 4mo ago
  I'd say most of my CS classes (in the 90s) had us write out code by hand, on paper, during in-class exams. It was fine.
  - dysoco 4mo ago
newswangerd 4mo ago
Using AI to inform architecture doesn’t seem so different from googling architecture in this case. Architectural patterns are mostly well understood and well documented these days and are something that you could piece together via Google search pre AI. The thing that AI brings to the table that wasn’t google able in the past is code generation. Previously you had to understand the architecture patterns to implement them yourself, but now the AI can just do it for you.

bo1024 4mo ago

What is the policy and guidance you gave the students regarding LLM use?

jventura 4mo ago
This is all so new, and caught us completely unprepared that there's no official university-level policies. Most of us are still navigating the waters and seeing what works and what doesn't work anymore. I have colleagues that are teaching for more than 30 years, few years away from retirement, who suddenly have been confronted with a new way of doing things. Those are the ones that are still insisting on doing practical projects, etc. I've only been doing this for 20 years, and I'm quite lazy (worked previously as software engineer), so I've moved to those practical tests. I guess that there should probably exist a class or workshop to teach these students how to use LLMs effectively, but as I said, this technology and its implications is quite new. Personally, what I did was to give them the "lecture" in the line of that they do not understand what the machine has generated, that is not the way a true engineer does, try to do some parallel with things like an LLM designing a bridge and civil engineers building that bridge, and a fatal flaw collapsing the all thing, etc. In other words, we do not have a formal system in place, it's all talking and convincing them. Obviously it's a big enough problem that should deserve more investment in solutions, but we are all overwhelmed by other tasks. Maybe LLM studios should be held responsible for all these "disruptions" and provide solutions to problems they created! :)

LovelyButterfly 4mo ago

As a professor, what's you opinion regard the Socratic method with LLMs? Would that be preferable over simply "give me the answer" prompt?

jventura 4mo ago
Everything that provides students with a workflow to think and to try to find solutions to a problem is much better than giving the answer directly! Unfortunately there will always be students that prefer to take the shortcut.. How could we "force" the students to use an LLM that confronted their doubts with more questions? We could tell them to start each chat with a specific prompt (to use the socratic method, etc), but they could eventually jail-break it.. But nevertheless, I like your idea! This is something that a document highlighting methodologies for students on how to use LLMs effectively could/should contain..
- e-topy 4mo ago
  I'm curious on your opinion, as a professor, how should universities (not individual professors) adapt to LLMs? As an undergrad, I hope schools move to educating students to use LLMs in a more responsible way. You can't put the genie back in the bottle, and resisting progress is futile, might as well use the tool we now have to help students learn even faster and better (e.g., making feedback instant and not answers, helping digest or split up material, checking answers). I know opinions about AI at (not only at) my faculty are very mixed, but I think the answer is going to be in the rational mean, just like how technorealism reacted to the internet[0]. In our last program board sitting, some teachers said that they think programming as a job will be completely irrelevant in two years, while other pushed for more adoption. And meanwhile I know of some students that are basically only passing because of LLMs, and it's bad, like "leaving claude output in markdown files and finished source code on the faculty server in /tmp because opencode did so" bad. And our first year classes completely prohibit even sharing tests or talking about the solutions, which in my opinion a) makes people extremely asocial and atomized b) doesn't prepare students for real life c) promotes dishonesty. Still, I think our university's thinking is in a stalemate, not wanting pure AI output and useless students, while also wanting to move with the times, and I doubt it's the only one. [0]: https://web.archive.org/web/20081009111415/https://artefaktum.hu/it2/millarch.html https://web.archive.org/web/20081009111415/https://artefaktu... (absolutely amazing read, recommend it)

rahimnathwani 4mo ago

So all groups outsourced their thinking to an LLM. Will they have learned anything they can apply to different types of projects in future?

jventura 4mo ago
Not in the same level, see my other comments..
- rahimnathwani 4mo ago
  I think we're using two standards without naming the difference. - "Responsible for the code", i.e. understands it, can explain it, can modify it on request, is the right bar for a working engineer using an LLM to go faster. But it's the wrong bar for a student, because in a course the deliverable isn't the point, the practice is. I presume that as course like 'Parallel Computing' exists to build the habits of reasoning about decomposition, contention, race conditions, strategy tradeoffs. Those habits come from doing the reps, the same way you don't learn to write by reading good prose. Your two "successful" groups outsourced exactly the reps the course is (or should be) there to build. Being able to follow and tweak a solution proves comprehension, but comprehension is a much lower bar than producing the solution. Your students may understand this particular solution well enough to explain it. But did they learn enough that they could tackle a different problem with a different shape, without using an LLM?
  - jventura 4mo ago
    I'm not a native EN speaker (I'm Portuguese), therefore a lot is lost in my attempt to find the right words. I'm writing these words without any use of AI or even a sentence-checker (only the spell-checker in Firefox). Of course that the "succesfull" groups implemented their solutions, tweaked them and, tried their best to escape from race conditions and were able to talk about the performance/memory/etc. tradeoffs. We have a grid to grade for those and other items. They did not outsource all of their thinking to an LLM.. That is what I mean as taking responsability for their code, maybe not the correct word, but the one that came to my head when I wrote it..

estetlinus 4mo ago

I did a course of parallell computing 6 years ago. I still dont understand anything about it. Probably would if I had used an LLM, since I would have used it as a private tutor.

ygouzerh 4mo ago

How do you see impact of using LLM on the teaching content? (genuine curiosity) In the industry, almost everyone here is doing the exact same thing, outsourcing more and more of their thinking to LLM: so even if students learned how to manually write code, they will probably loose it later on (happening to us already, mostly can be seen when people are working on new solutions/frameworks... they now have the same issue than students).