3 ms·

I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not

I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not seem like a majority of the students are using ChatGPT at all, even though I did encourage the use of it at the beginning of the course. For my own course, I think several factors contributed to students not utilizing ChatGPT as much: - The assignments are not in English, and performance of ChatGPT in languages other than English is subpar. - The programming language that I'm teaching is C, I'd imagine Python/Javascript and other more popular languages might lead to different outcomes - I did specifically design the assignments so that copy/pasting the assignment to ChatGPT does not lead to a usable answer (by restricting use of certain standard library functions, making the assignment more complicated) - The course is not introductory, i.e. a previous course already taught the basic syntax of C and basics of programming, so I can make my assignments much more advanced It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.

alach11 3y ago
> from what I can see right now, LLMs still have trouble solving novel problems Have you been using 3.5 or 4?
- TheOnly92 3y ago
  I have been using 3.5, but when I was designing the assignments, I asked a friend who had access to 4 to check it, and the code it produced was still incorrect.
nsxwolf 3y ago
I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door". I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
- TheOnly92 3y ago
  Interesting, I'd suppose it makes sense that rephrasing the problem in a different way and also adding a bunch of nouns that have no relations with the problem at hand will definitely confuse LLMs. It will be interesting to see how LLMs will adapt to these as more and more of these techniques develop. In my own assignments however, I focus less on algorithmic stuff but more on adding and mixing several things together. E.g. instead of just sorting, do group & sort, and a combination of a bunch of other practical stuff like reading big-endian binary files.
- flangola7 3y ago
  3.5 or 4? March or June model?
cfn 3y ago
I've been using ChatGPT and Bard with C++ and it is quite helpful for boilerplate and reference (replacing Google/Stackoverflow). Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.
- TheOnly92 3y ago
  It definitely depends on the task at hand, but when you're teaching programming you don't teach stuff with boilerplate. Using ChatGPT for reference to replace Google/Stackoverflow was definitely one of the ways I'd expected the students to utilize it, but it probably wasn't providing answers in ways a beginner/novice could understand. I'd expect simple tasks like calculating RMSE to definitely be within the abilities of LLM, you might combine things like actually reading the vectors from a CSV file (or a custom format) and calculating RMSE then sorting them etc to see the limitations of LLMs. Most students have no issues with calculating RMSE, they have issues with trying to do all the other stuff that leads to it, and then the combination of sorting and other tasks. Regarding the restrictions, most of them are just don't use itoa/strtod or strcpy or some other standard library functions.
  - cfn 3y ago
    Thanks, yes, RMSE is a simple task, I was focusing on its ability to translate the name (raiz quadrada do erro quadrático médio) correctly. It is funny that the Portuguese code has Portuguese comments but the name of the function is calculateRMSE even though I don't mention RMSE in the prompt. I agree with you, in my experience, ChatGPT is a better search engine but it is not capable of composing the various parts of an application in a cogent manner. I also think that the current UI is not appropriate for software development and I am sure there are efforts going on to create something closer to Jupyter notebooks for programming. That may be a game changer for your students (and you).
    - TheOnly92 3y ago

alach11 3y ago

> from what I can see right now, LLMs still have trouble solving novel problems Have you been using 3.5 or 4?

TheOnly92 3y ago
I have been using 3.5, but when I was designing the assignments, I asked a friend who had access to 4 to check it, and the code it produced was still incorrect.

nsxwolf 3y ago

I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door". I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.

TheOnly92 3y ago
Interesting, I'd suppose it makes sense that rephrasing the problem in a different way and also adding a bunch of nouns that have no relations with the problem at hand will definitely confuse LLMs. It will be interesting to see how LLMs will adapt to these as more and more of these techniques develop. In my own assignments however, I focus less on algorithmic stuff but more on adding and mixing several things together. E.g. instead of just sorting, do group & sort, and a combination of a bunch of other practical stuff like reading big-endian binary files.
flangola7 3y ago
3.5 or 4? March or June model?

cfn 3y ago

I've been using ChatGPT and Bard with C++ and it is quite helpful for boilerplate and reference (replacing Google/Stackoverflow). Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.

TheOnly92 3y ago
It definitely depends on the task at hand, but when you're teaching programming you don't teach stuff with boilerplate. Using ChatGPT for reference to replace Google/Stackoverflow was definitely one of the ways I'd expected the students to utilize it, but it probably wasn't providing answers in ways a beginner/novice could understand. I'd expect simple tasks like calculating RMSE to definitely be within the abilities of LLM, you might combine things like actually reading the vectors from a CSV file (or a custom format) and calculating RMSE then sorting them etc to see the limitations of LLMs. Most students have no issues with calculating RMSE, they have issues with trying to do all the other stuff that leads to it, and then the combination of sorting and other tasks. Regarding the restrictions, most of them are just don't use itoa/strtod or strcpy or some other standard library functions.
- cfn 3y ago
  Thanks, yes, RMSE is a simple task, I was focusing on its ability to translate the name (raiz quadrada do erro quadrático médio) correctly. It is funny that the Portuguese code has Portuguese comments but the name of the function is calculateRMSE even though I don't mention RMSE in the prompt. I agree with you, in my experience, ChatGPT is a better search engine but it is not capable of composing the various parts of an application in a cogent manner. I also think that the current UI is not appropriate for software development and I am sure there are efforts going on to create something closer to Jupyter notebooks for programming. That may be a game changer for your students (and you).
  - TheOnly92 3y ago