6 ms·
Today I had to submit my final grades for my Java class, meaning I had to grade all the final exams for it last night. During the exam (which I had to administ
by tombert 3y ago
Today I had to submit my final grades for my Java class, meaning I had to grade all the final exams for it last night.
During the exam (which I had to administer remotely this time) I made it abundantly clear that they cannot use any kind of AI assistance for their work. No ChatGPT, no Copilot, no Bing AI, no Google Bard, and also no Googling, etc. I repeated this several times and also wrote it in bold font on the top of the test. I really didn't have any way to enforce this, but I was hoping people would be honest.
I'm pretty sure that most of the students were honest on this; the answers I got generally fine, but had grammatical mistakes and were "basically correct but had light factual errors that are common with people new to programming but aren't bad enough to count as 'wrong'". One student, however, who has submitted broken sentences and broken code the entire semester, managed to suddenly have decent writing skills, decent explanations of everything, and his code was clean and concise.
I'm about 95% sure he used ChatGPT to generate answers to the questions. I tried getting ChatGPT (and Bard and Bing AI) to give me a word-for-word copy of what he submitted, but I couldn't. It got somewhat close, but never an exact match.
Technically, it's possible that he just studied very very hard and his code and grammar improved. It's also technically possible that he used Grammarly to make sure his writing was ok (which was technically against the rules but I wouldn't really consider cheating in a Java class), and so I just had to swallow my pride and grade the test assuming he was being honest.
It's kind of upset me all day; I have worked pretty hard trying my best to be available to students if they have questions, and I worked pretty hard to try and make sure that the final exam was a reasonable level of difficulty. I think most of my students were fine, but one bad apple is enough to really ruin my day.
- elliotto 3y agoWhilst it's unfortunate this kid might be cheating the system, I don't think it's worth being upset if they got through. The point of these exams is to allow students to study and learn these topics, which it sounds like they did. It sounds like while maybe you failed in gatekeeping the cheaters (an impossible post gpt task) you have succeeded in the real goal which was to help these kids learn and improve.
- tombert 3y agoOh I largely agree; I'm kind of learning that you have to view teaching as somewhat more of a "statistical success" than anything else. It's just one of those things that while you're totally right, the one or two bits of failure kind of nag at me. It's extremely easy to take these things personally (especially for a nascent teacher like me) when in reality I should likely view things as transactional.
- yowzadave 3y agoA good and reasonable sentiment. It's a very human thing for people who are largely successful at something to fixate unproductively on failures--and in this case you can't even conclusively say that it was a "failure".
- i2cmaster 3y agoI think it's good. This sort of thing will move Universities back to being about teaching and research instead of vending credentials.
- tombert 3y agoI've gone back and forth about how I feel about grades as a whole. I do feel that by having these tests and grades that it does, indeed, sort of become just a transactional way to "vend credentials" instead of focusing on learning. I also think that often homework and tests aren't great measurements; I was bad about doing homework in college the first time around, and it was frustrating to the professors because I was doing horrible grade-wise, but was extremely active during class and generally did fine on the tests. I was mostly learning the material (at least well enough to get an A on the exams), but because I wasn't submitting my assignments, they would be forced to give me a C or a D. At the same time, I really haven't figured out a better system. I would be open alternative systems, but I think grades and grading and tests do the best job for the "average case". Maybe I should email some of my old professors and apologize for the headaches I probably gave them...
- the_only_law 3y ago> This sort of thing will move Universities back to being about teaching and research instead of vending credentials. I don’t see this happening unless jobs stop caring about credentials or become more willing accept other forms of credentials. Of course I’m assuming you aren’t specifically talking about software, which doesn’t have nearly the amount of credentialism as many other fields.
- sumtechguy 3y agoToday you learned a lesson. You will need to add to your course work how GPT can get things wildly wrong. Just a simple example should not be too terribly hard to find. Then plant the seed 'if it has that wrong what else is it getting wrong? It is a good tool of getting the general idea but one you need to audit. This class is where you are going to learn the basics and know when this thing is wrong when you use it' That thing is not going away any time soon.
- tombert 3y agoI actually did say something more or less exactly like that pretty early into the course. Something to the effect of "ChatGPT is pretty cool, I like it, but it will just make stuff up sometimes in extremely convincing way. It's a tool that's dangerous to newbies and powerful to professionals, and I'm hoping this class will help you get closer to the latter".
- sumtechguy 3y agoAh good. A very small demonstration probably would drive the point home. It is one thing to hear that but another to see it yourself.
- DangitBobby 3y agoBut clearly it gets things right often enough to be worth using to cheat on a test or assignment.
- TheRealPomax 3y agoAlso, how GPT can get things right, and that if students use GPT (because it's the early days of the graphing calculator all over again but now for code) then the exercise is not "write the code" but "explain the conditions under which this code doesn't work, and why" because if you're taking a programming class it doesn't matter what writes the code, what matters is whether you understand that code. So if you're turning the exercise into a code audit by using GPT, you better damn well be able to explain what problems are left in the code it generated.
- 3y ago
- saalweachter 3y agoI wonder if it would be worth it to give some sort of in person feedback to convey (a) you noticed the dramatic improvement (b) you were even worried they might be cheating (c) you decided to give them the benefit of the doubt (d) congratulations on their hard work paying off (e) but if they did cheat they got lucky and if they keep it up it's going to backfire on them one day.
- tombert 3y agoYeah, I've drafted a few emails that I have not sent stating more or less what you listed, in "decreasingly-pissy" tones. I think by tonight I'll have cooled off enough to write an appropriate email.
- sebzim4500 3y agoI really don't think this is a good idea. If I got an email half accusing me of cheating I would complain to anyone who would listen.
- Spivak 3y agoHonestly, unless the grade for that class translates into taking something directly tangible from other students, and a "java" class sounds pretty intro, I wouldn't give any thought to it. Cramming for exams and then immediately forgetting everything you hastily memorized is a-okay even though you didn't learn anything either. A random college class, to me, isn't the kind of thing where cheating really matters. The stakes are low, it's expected that most students will do well, and everyone gets the same degree at the end. If someone wants to sabotage their own education then fine, and if it doesn't bite them later good for them.
- tombert 3y agoYeah, I'll admit that the problem is somewhat on my end, but I find it very hard not to take these kinds of things personally. I agree that I shouldn't take them personally, but that's easier said than done.
- cogman10 3y agoI'm starting to see this come up in coding interviews. I had one candidate that struggled MIGHTILY on a coding problem that was easy (kept running into syntax errors) but then pulled out an esoteric solution for the second problem in no time flat. I can't prove that he was cheating but it very highly suggestive that he is. I hate it, but I'm starting to think I have to do an AI detection question now. It's not hard, just ask someone to do something impossible. However, I don't like the fact that now I need to be "tricky". I've never believed in making coding challenges hard, I just want to see if you can write code.
- cgearhart 3y agoI saw something similar in a recent interview. I brought in a candidate who did not meet all of the typical resume check boxes, but who had a long history of success on highly technical problems. These type of candidates can bring very new perspectives to our team, but they’re often a long shot in interviews. In this case the candidate chose to use a language that I don’t know well and they said they weren’t especially familiar with either. Then they struggled a lot to even talk through the question and it was a grind to work through each line of code. I couldn’t offer much help, and they were very clearly unsure at each step. But when we clicked “run” on the code, we were both surprised to see that it ran on the first try. Not only that, but it already worked for all of the “but what about…” follow up edge cases. And the candidate didn’t know why… “oh, I guess the language just works that way.” I don’t know if they were cheating, but the entire thing was far outside any other interview experience I’ve ever had — and fits the pattern of what I’d expect an LLM to produce.
- giraffe_lady 3y agothis is me using ocaml
- littlekey 3y agoSeems like there's a silver lining here. It reinforces the idea that the answer to the problem is not necessarily the important part. When you can trivially provide a correct answer, it places more emphasis on the skill of being able to actually explain and work through the logic as a way to distinguish yourself from other candidates.
- bachmeier 3y agoUniversities are going to have to adjust. That's really all there is to it. It's not that hard for an exam, since you can put them in a properly designed classroom with human monitors. Homework? Projects? Much more difficult. The current system worked in 1960. Not in 2023, and probably not since the late 1990s, when technology started to make our evaluation processes obsolete.
- nickff 3y agoExams used to be a larger part of school grades. In my area, they were devalued and the reason that was given was that exams favored males over females.
- throwawayffffas 3y agoI would suggest you ask him. Offer him immunity and ask him. For your peace of mind and so that in the future you can be better prepared.
- joefourier 3y agoCan I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? As a software developer I use these tools daily, and if I or my colleagues stopped our productivity would suffer - it sounds like the same genre of thinking that leads to having students write out code by hand because using an IDE is “cheating”. If it would be possible to take a bottom-tier CS student and turn them into a decent programmer using AI tools, the hiring landscape would be /very/ different. These tools aren’t magic, and to me it sounds like the tests are failing at measuring the students competence if they can so easily be gamed by using an AI/old fashioned googling. Maybe the solution should be to move towards a style of exam/grading that actually measures the competence of the student in a situation closer to what a professional developer will be in, rather than an old fashioned artificial exam setting?
- yuliyp 3y agoAn evaluation for a course should measure how much of the subjects of the course the students learned. It's not a job success simulator, nor a measure of unrelated skills.
- hxugufjfjf 3y agoSo the students are disallowed or discouraged from learning how to use tools like AI in the courses they take?
- watwut 3y agoStudent are neither disallowed or discouraged from learning them. The are not allowed to use them during unrelated test.
- hxugufjfjf 3y agoWhy is the test unrelated to learning methods?
- 3y ago
- marcus_holmes 3y agoBasic arithmetic and calculators. I'm old: when I went to school we weren't allowed calculators "because there'll be a situation where you need to do basic arithmetic and don't have a calculator". I occasionally laugh at this while firing up the calculator app on my phone. We'd have been better off if they'd taught us sign language rather than arithmetic. Our education system is going to have to adjust to the new reality. Setting an essay task is now the equivalent of learning the times table - utterly redundant (unless you plan on a career writing essays or doing maths).
- to11mtm 3y agoThe most 'fair' math classes (specifically, I'm thinking of algebra and calculus type courses) I have been in, allowed a good in-between. Basically, you were totally allowed to use a Calculator, even one with a built-in CAS, but you -had- to show the in-between steps if you did so on a test. This was pretty useful in it's own right, as you at least had a way to check your work, and frankly it was extremely useful for helping me solidify knowledge in those fields. [0] Of course, AI muddles this, since it can explain the steps for you.
- pclmulqdq 3y agoThat may be the strawman, but that's not the real reason why people ban calculators when you learn arithmetic. The reason is because learning to manipulate numbers in your head abstracts to other things, like learning to estimate and learning to manipulate equations. Higher-level math classes usually encourage calculators, and graduate-level math classes will often allow Matlab or Mathematica, as long as you can write a short sentence explaining what the computer algebra system is doing. Where I went to school, this was colloquially called "proof by Steven (Wolfram)."
- marcus_holmes 3y agoIt is a strawman. But I wonder how much better we'd be able to do those other things if we'd been taught to do them with calculators as a tool instead of being forced to do the basic arithmetic in our heads.
- shinryuu 3y agoSo the purpose of academia is teach pupils a subject, that they can the use in real life. When you're solving the same problems in real life you do have access to all those tools that you banned. Because in real life, if you manage to solve a problem it doesn't matter that much how you solved it. Sure, if you just Google things and always solve your problems by copy pasting. Your solutions will lack depth, and at some point this will catch up to you. My point is, cheating is primarily cheating on yourself, and it will catch up to you. I think this is the framing you should have with regards to cheating.
- rmellow 3y ago> Technically, it's possible that he just studied very very hard and his code and grammar improved. As an early procrastinator (rehabilitated perhaps?) I've had a history of doing very well on finals through cramming. In your student's case it might very well be more likely they are cheating, but those unlikely but possible students who study extra hard to recover their grades for a final deserve the benefit of the doubt. How unlikely can academic recovery get? I spent an entire semester completely lost in a second level Macroeconomics course... ... I was able to read and write chapter summaries of a semester and a half of content (about 15 chapters, corresponding to the first level and the second level courses) in a single weekend and aced the exam, with the highest marks in the class. That was the hardest I've ever crammed in my life and I had trouble speaking in the hours that followed the study session. I later got a recommendation letter from that prof, totally worth it.
- clusterhacks 3y agoSome students cheat. As an adjunct instructor at the university level, I had similar experiences. You have to both adjust your expectations and make a significant part of your grading use an in-person one-on-one oral exam or project walkthrough. It is a huge commitment of time - most professors I know seem to have quietly accepted cheating will be be rewarded because they don't have enough time to verify student performance.
- pixl97 3y agoIt seems we want to turn universities into McDonalds where we push high numbers, but also pretend they are elite institutions that we can charge premium prices for. This is not going to work well in an age of ever increasing automated intelligence... Well, lest of course the students sit under the watchful eye of AI in the near future
- kazinator 3y ago> I made it abundantly clear that they cannot use any kind of AI assistance for their work You're swimming against the riptide. If I were teaching coding today, I'd allow, even encourage, use of AI, with some ground rules. Students using AI would have to show their work (i.e. the conversation with the machine). The content of the prompts and in particular follow-up questions, can demonstrate competence. E.g. situations in which the student spotted that something was wrong in the generated code and made an intelligent hint to the machine to fix it. Students should be prepared for tomorrow's world, not yesterday's.
- balderdash 3y agoBy that logic wouldn’t it be ok for grade schoolers to use calculators when they are learning basic math? Sure you can teach someone how to get an answer, but have you actually given them an understanding of the material to be built upon?
- kazinator 3y agoKids should be using calculators in kindergarten. You can discover a lot of curious things about the numbers with calculators. Like when you multiply by 5 and keep doing it, the last two digits are always 25, and the hundreds digit bounces between 1 and 6. You can discover that without a calculator, but it's more time consuming. If we want to ensure kids can do arithmetic without calculators, we can have exams in which there are no calculators. You can give them homework problems in which they have to fill in all the work steps of multiplication and long division, e.g.: ______ 17 | 2091 -__ __ -__ __ __ If you want to ensure that the students know and follow a certain process in obtaining an answer, you can't just test the production of an answer. You have to white box it. Kids should know time tables to about 12. Beyond that, you can test whether they know the structure of long multiplication and division and such.
- balderdash 3y ago“If we want to ensure kids can do arithmetic without calculators, we can have exams in which there are no calculators.” Which is exactly what the parent was trying to do with a an exam that forbade the use of AI.