11 ms·

Giving university exams in the age of chatbots

257 points by ploum 9mo ago

diamondgeezer 9mo ago
Very interesting write up, would be curious to know more about what an Open Source Strategies course entails, as far as I can remember I never had anything like that on offer at my university.
- nkapias 9mo ago
  https://uclouvain.be/cours-2025-linfo2401 https://uclouvain.be/cours-2025-linfo2401 https://uclouvain.be/cours-2025-linfo2402 https://uclouvain.be/cours-2025-linfo2402
sublang 9mo ago
[dead]
emil-lp 9mo ago
> Most Students Don’t Want to Use Chatbots I think this is changing rapidly. I'm a university professor, and the amount of students who seem to be in need of LLM as a crutch is growing really exponentially. We are still in a place where the oldest students did their first year completely without LLMs. But younger students have used LLMs throughout their studies, and I fear that in the future, we will see full generations of students completely incapable of working without LLM assistance.
- teaearlgraycold 9mo ago
  It will be very interesting to see what will happen when LLMs start charging users for their true cost. With many people priced out how would they cope?
  - alphabetag675 9mo ago
    They would fall behind in the world just like people from developing and poor countries do today.
    - josephg 9mo ago
      Very few people fall behind at the moment due to lack of access to information. People in poor countries largely have access to the internet now. It doesn’t magically make people educated and economically prosperous.
      - alphabetag675 9mo ago
        You are arguing the converse. Access to information doesn't make people educated, but lack of access definitely puts people at a big advantage. Chatbots are not just information, they are tools and using it needs training because they hallucinate.
  - KeplerBoy 9mo ago
    It's not that expensive unless you run millions of tokens through an agent. For use cases where you actually read all the input and output by yourself (i.e. an actual conversation), it is insanely cheap.
    - tucnak 9mo ago
      Yeah in my last job, unsupervised dataset-scale transformations amounted to 97% of all spending. We were using gemini 2.5 flash in batch/prefill-caching mode in Vertex, and always the latest/brightest for ChatGPT-like conversations.
  - ben_w 9mo ago
    May happen, but I suspect not in the way implied by that question. Hardware is still improving, though not as fast as it used to; it's very plausible that even the current largest open weights models will run on affordable PCs and laptops in 5 years, and high-end smartphones in 7. I don't know how big the SOTA close-weights models are, that may come later. But: to the extent that a model that runs on your phone can do your job, your employer will ask "why are we paying you so much?" and now you can't afford the phone. Even if the SOTA is always running ahead of local models, Claude Code could cost 1500 times as much and still have the average American business asking "So why did we hire a junior? You say the juniors learn when we train them, I don't care, let some other company do that and we only hire mid-tier and up now." (Threshold is less than 1500 elsewhere, I just happened to have recently seen the average US pay for junior-grade software developers, $85k*, which is 350x cheaper, and my own observation that they're not only junior quality but also much faster to output than a junior). * but also note while looking for a citation the search results made claims varying from $55k to $97.7k
  - johndough 9mo ago
    What do you think the "true cost" is?
- kubb 9mo ago
  Please, you don’t need to counter-narrative everything. Maybe talk about what the professor did here and why students didn’t trust the output in an exam context in this particular subject.
  - nicce 9mo ago
    > Second, I learned that cheating, however lightly, is now considered a major crime. It might result in the student being banned from any university in the country for three years. Discussing exam with someone who has yet to pass it might be considered cheating. Students have very strict rules on their Discord. This has also something to do with it. Hard to make very accurate conclusions.
- raesene9 9mo ago
  Reading the article, it seemed to me that both the professor and the students were interested in the material being taught and therefore actively wanted to learn it, so using an LLM isn't the best tactic. My feeling is that for many/most students, getting a great understanding of the course material isn't the primary goal, passing the course so they can get a good job is the primary goal. For this group using LLMs makes a lot of sense. I know when I was a student doing a course I was not particularly interested in because my parents/school told me that was the right thing to do, if LLMs had been around, I absolutely would have used them :).
- deleted 9mo ago
  [deleted]
- themafia 9mo ago
  Google destroyed search and replaced it with that dippy LLM box. Are you sure student desire is the driving force here?
- jcattle 9mo ago
  I think the professor here presented them with a "special" case which can not be generalized outside of the exam context. If you're presented with the choice of "Don't use AI" and "Use AI, but live with the consequences" (consequences like mistakes being judged harsher when using AI than when not using AI), I do not think chatbots will be a desirable choice if you've properly prepared for the exam.
- Jedd 9mo ago
  > ... is growing really exponentially. Or geometrically?
- petterroea 9mo ago
  One helper here is fear. You can be failed for formal errors at university, and it means we were scared shitless of making them and payed close attention. If people know "at university you can't use LLM, you are forced to think by yourself" they will adjust, albeit by trial of fire. I think there's an argument that growing up in an educational system unable to teach you how to not rely on LLM would for all intents and purposes permanently nerf you compared to more fortunate peers. Critical thought is a skill we continue to practice until the very end
mkirsten 9mo ago
Interesting write up! I’ve thought about how university exams are done effectively nowadays. I took my degree in CS almost 20 years ago, and being a user of LLMS - I can’t really see how any of my old exams would work today if students would be allowed LLMs.
- emil-lp 9mo ago
  Spoiler: they don't. CS exercises that we can expect an average student to solve is trivially solved by LLMs. Even smaller local models.
  - bandrami 9mo ago
    Except some times they aren't solved by LLMs, but appear to be. A CS student should be able to look at it and tell the difference.
  - Ekaros 9mo ago
    This comes to not "smartness" of LLMs. But reality that we do not even want anything novel in these exercises or exams. And same areas are repeated multiple times so naturally there is lot of these in training data. This is one area where LLMs really should excel at. And that doesn't really mean that students should not also learn it and be able to solve same issues. Which is real dilemma for the school system...
- brainwad 9mo ago
  I graduated 15 years ago, and I think the exams in my degree were actually the most LLM-proof part of the student assessment. They were no-aid written exams with pencils and paper, whereas the assignments were online-submitted code only that an LLM could easily write.
- deleted 9mo ago
  [deleted]
lucb1e 9mo ago
> 3. I allow students to discuss among themselves [during an exam] if it is on topic. Makes me wonder if they should also get a diploma together then, saying "may not have the tested knowledge if not accompanied by $other_student" I know of some companies that support hiring people as a team (either all or none get hired and they're meant to then work together well), so it wouldn't necessarily be a problem if they wish to be a team like that
- witcher 9mo ago
  Yea, curious too about some more rules e.g. both parties has to contribute to the discussion (:
- kubb 9mo ago
  I think we should send all diplomas to OpenAI and end higher education. Less educated people are easier to steer via TikTok feeds anyway.
- elbci 9mo ago
  ha ha fair enough - but he does mention there's a culture of isolation and cu-throat competition at the school so, maybe it's just a reaction to that
- ploum 9mo ago
  OP here: I teach Open Source Strategies. The main strategy is collaboration. If you are smart enough to: 1. Identify your problem 2. Ask someone about it 3. Get an answer which improve your understanding Then you are doing pretty good by all standards Another trick I sometimes use. I take one student which has hard time articulate it a concept. I take student two who don’t understand that concept. I say to student 1: "You have 20 minutes to teach student 2 the concept. When I come back, you will be graded according to his answers" (I, of course, not only grade that. But it forces both of them to make an extra effort, student 2 not willing to be the cause for student 1 demise)
  - zahlman 9mo ago
    > student 2 not willing to be the cause for student 1 demise I would very much not count on that.
witcher 9mo ago
Quite a thoughtful way to adapt exams to wave of new tools for students and learn on the way. I wished other universities adapt so quickly too (and have such a mindful attitude to students e.g. try to understand them, be upfront with expectations, learning from students etc). Majority of professors are stressed and treat students as idiots... at least that was the case decade a go!
- ploum 9mo ago
  OP here: Majority of professors became professors because there were very good at passing standard exam (and, TBH, some are not good at anything else). I’m different because I was a bad student. Only managed to get my diploma with minimal grade, always rebel against everything. But some good people at my university thought that Open Source was really important and they needed someone with a good career in that field. I was that person (and I’m really thankful for offering that position)
  - malshe 9mo ago
    > Majority of professors became professors because there were very good at passing standard exam (and, TBH, some are not good at anything else). Is this a French thing? In the US we don't have standardized exams to become a college professor. Instead, we need to do original research and publish.
elbci 9mo ago
rare here: well written and insightful, I would take this course. I'm curious about why he penalized chatbot mistakes more, at first glance sounds like just discouraging their use but the hole setup indicates genuine desire to let it be a possibility. In my mind the rule should be "same penalty and extra super cookies for catching chatbot mistakes"
- qwertytyyuu 9mo ago
  Here is my guess: Usually marks are given for partially correct answers, partially to be less punishing for human error whether caused by stress or other factors, there’s a good chance the student understood the topic. If instead they are using a chat bot, but didn’t catch the mistake themselves, it’s an indication of less understanding and marked accordingly.
- jcattle 9mo ago
  I wrote this before to another comment like yours: I thought this part of penalizing mistakes made with the help of LLMs more was quite ingenious. If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. If you do not use LLMs and just misunderstood something, you will have a (flawed) justification for why you wrote this. If there's something flawed in an LLM answer, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. One shows a misunderstanding, the other doesn't necessarily show any understanding at all.
burgerone 9mo ago
I wish we could take our exams this way. It seems like a very interesting approach :)
barbegal 9mo ago
Only 2 students actually used an LLM in his exam, one well and one poorly so I'm not sure there is much you can draw from this experience. In my experience LLMs can significantly speed up the process of solving exam questions. They can surface relevant material I don't know about, they can remember how other similar problems are solved a lot better than I can and they can check for any mistakes in my answer. Yes when you get into very niche areas they start to fail (and often in a misleading way) but if you run through practise papers at all you can tell this and either avoid using the LLM or do some fine tuning on past papers.
quacked 9mo ago
Something that I think many students, indeed many people, struggle with is the question "why should I know anything?" For most of us--myself included--once you graduate from college, the answer is: "enough to not get fired". This is far less than most curriculums ask you to know, and every year, "enough to not get fired" is a lower and lower bar. With LLMs, it's practically on the floor for 90% of full-time jobs. That is why I propose exactly the opposite regimen from this course, although I admire the writer's free thinking. Return to tradition, with a twist. Closed-book exams, no note sheets, all handwritten. Add a verbal examination, even though it massively increases examination time. No homework assignments, which encourage "completionist mindset", where the turning-in of the assignment feels more real than understanding the assignment. Publish problem sets thousands of problems large with worked-out-solutions to remove the incentive to cheat. "Memorization is a prerequisite for creativity" -- paraphrase of an HN comment about a fondly remembered physics professor who made the students memorize every equation in the class. In the age of the LLM, I suspect this is triply true.
- 6LLvveMx2koXfwn 9mo ago
  This is all very well if the goal was to sift the wheat from the chaff - but modern western education is about passing as many fee paying students as possible, preferably with a passably enjoyable experience for the institutional kudos.
  - josephg 9mo ago
    I wonder if education will bifurcate back out as a result of AI. Small, bespoke institutions which insist on knowledge and difficult tests. And degree factories. It seems like students want the degree factory experience with the prestige of an elite institution. But - obviously - that can’t last long. Colleges and universities should decide what they are and commit accordingly.
    - graemep 9mo ago
      I think the UK has been heading this way for a while -- before AI. Its not been the size of the institutions that has changed, but the "elite" universities tend to give students more individual attention. A number of them (not just Oxford and Cambridge) have tutorial systems where a lot of learning is done in a small group (usually two or three students). They have always done this. At the other extreme are universities offering low quality courses that are definitely degree factories. They tend to have a strong vocational focus but nonetheless they are not effective in improving employability. In the last few decades we have expanded the university system and there are far more of these. There is no clear cutoff and a lot of variation in between so its not a bifurcation but the quality vs factory difference is there.
pautasso 9mo ago
The problem is when students just blindly copy and paste from the chatbot and submit it as their own answer without even reading it. They should be encouraged to read and review the LLM output so they can critically understand it and take ownership of it.
- dyauspitr 9mo ago
  In my experience, reading a solution and even understanding it doesn’t go very far in teaching you how to do something. I can look at calculus solutions all day but only when I actually try to solve them myself do I run into all kinds of roadblocks which is where the real learning happens.
  - 9rx 9mo ago
    You're right, but learning can take place when you need it. There is no real advantage to learning something ahead of time. The bottleneck is having awareness of what is out there to learn. You can't learn about what you don't know exists. Looking at calculus solutions all day should give a sense of what calculus can be used for, so that it is in your back pocket when the time you need it comes. Well, at least it used to be the bottleneck. Nowadays you can just ask an LLM. For all their faults, they are really good at letting you know about what tools exist in out there in the world, surfacing more than you could ever come to know about even if all you did was read about what exists all day, every day.
    - dyauspitr 9mo ago
      I believe to count as an expert on something you need to have a ready compendium of knowledge ready to go. It becomes very hard to tackle problems or gain deep insights if you don’t already have knowledgeable people that have thought deeply about a particular space. Maybe when we have supremely reliable LLMs that can replace humans we might not but we’re not there yet.
      - 9rx 9mo ago
        > I believe to count as an expert on something you need to have a ready compendium of knowledge ready to go. You are certainly headed in the right direction, but not quite. To be seen as an expert in the eyes of others you need to have had a vision for something and to have successfully executed on it. If the vision was dependent on calculus, then you will have reached a point where you had to learn something about calculus, of course... But that's different to having a taskmaster tell you to learn calculus for no apparent reason. Even if you follow through and built up a huge wealth of knowledge from it, you would still not be deemed an expert by others. You're no different than an encyclopedia, which isn't an expert either. It is being able to see things others can't and the ability to act upon it that makes an expert. Learning taking place when you need it isn't the same as never. > Maybe when we have supremely reliable LLMs that can replace humans we might not but we’re not there yet. Frankly, even Page Rank already replaced humans for this. But LLMs are even better at it. Humans are just that poorly performing. Like I said before, even someone doing nothing in life but looking for what exists in the world could not take in as much as databases that have indexed every written thing.
veltas 9mo ago
> The third chatbot-using student had a very complex setup where he would use one LLM, then ask another unrelated LLM for confirmation. He had walls of text that were barely readable. When glancing at his screen, I immediately spotted a mistake (a chatbot explaining that "Sepia Search is a compass for the whole Fediverse"). I asked if he understood the problem with that specific sentence. He did not. Then I asked him questions for which I had seen the solution printed in his LLM output. He could not answer even though he had the answer on his screen. Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe. That guy who is playing with the latest tech, and forcing it to do the job (badly), and could care less about university or the course he's on. There's a time and a place where that guy is the one you want working for you. Maybe he's not the number 1 student, but I think there should be some room for this to be the Chaotic Neutral pick.
- watwut 9mo ago
  > Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe. He might as well be the dumbest guy in the class. Playing with tech is not a proof of being smart on itself.
- cheezburger 8mo ago
  what are you smoking?
Joel_Mckay 9mo ago
"Marking Exam Done by A.I." (Sixty Symbols) https://www.youtube.com/watch?v=JcQPAZP7-sE https://www.youtube.com/watch?v=JcQPAZP7-sE LLM reasoning models are very good at searching well documented problems. =3
knallfrosch 9mo ago
I don't understand. 10 years ago, we wrote exams by hand with whatever we understood (in our heads.) No colleagues, no laptops, no internet, no LLMs. This approach still works, why do something else? Unless you're specifically testing a student's ability to Google, they don't need access to it.
- tpoacher 9mo ago
  Optics.
- zavec 9mo ago
  It was the same when I graduated 6 years ago. We had projects to test our ability to use tools and such, and I guess in that context LLMs might be a concern. But exams were pencil and paper only.
- nikitau 9mo ago
  Open book exams are not a new thing and I've often had them for STEM disciplines (maths and biology). Depending on the subject, you will often fail those unless you had a good prior understanding of the material. If you can pass an exam just by googling something, it means you're just testing rote-memorization rather, and maybe a better design is needed where synthesis and critical thinking skills are evaluated more actively.
  - HWR_14 9mo ago
    Open book, sure. But you don't even need a computer for that.
    - freehorse 9mo ago
      And even if you are allowed to use a computer, you cannot use internet (and should not be hard to prevent that).
      - croes 9mo ago
        Local LLMs
bsder 9mo ago
> We were imposed GitHub for so many exercises! I'm sympathetic to both sides here. As a professor who had to run Subversion for students (a bit before Git, et al), it's a nightmare to put the infrastructure together, keep it reliable under spiky loads (there is always a crush at the deadline), be customer support for students who manage to do something weird or lose their password, etc. You wind up spending a non-trivial amount of time being sysadmin for the class on top of your teaching duties. Being able to say "Put it on GitHub" short circuits all of that. It sucks, but it makes life a huge amount easier for the professor. From the students point of view, sure, it sucks that nobody mentioned that Git could be used independently (or jj or Mercurial or ...) However, Github is going to be better than what 99.9% of all professors will put together or be able to use. Sure, you can use Git by itself, but then it needs to go somewhere that the professor can look at it, get submitted to automated testing, etc. That's not a trivial step. My students were happy that I had the "Best Homework Submission System" (said about Subversion of all things ...) because everybody else used the dumbass university enterprise thing that was completely useless (not going to mention its name because it deserves to die in the blazing fires of the lowest circle of Hell). However, it wasn't straightforward for me to put that together. And the probability of getting a professor with my motivation and skill is pretty low.
- i_am_proteus 9mo ago
  Agree about the possibility of infra nightmare, especially in the "SVN era" -- but in 2026, it's pretty straightforward to run a gitlab instance (takes about an hour to set up, most of which is DNS and TLS stuff, ime) for a course and set up actions, or use other submission infra like CMU autolab. I do this. Agree with your comment about probability, motivation, and skill.
SwtCyber 9mo ago
If anything, this reinforces the idea that chatbots don't fundamentally change education... they just amplify whatever incentives and structures already exist
- vaylian 9mo ago
  I am not so sure about that. I think the difference this time is that AI usage is fairly mainstream. People are using it for all kinds of things, including studies. Using it to quickly get done with school/study-work, is a no-brainer (pun intended).
fexed 9mo ago
What a wonderful article, and what a wonderful way of enganging with students and adapting to the new tech. I wish all professors were like you
Zababa 9mo ago
> Mistakes made by chatbots will be considered more important than honest human mistakes, resulting in the loss of more points. >I thought this was fair. You can use chatbots, but you will be held accountable for it. So you're held more accountable for the output actually? I'd be interested in how many students would choose to use LLMs if faults weren't penalized more.
- jcattle 9mo ago
  I thought this part especially was quite ingenious. If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. If you do not use LLMs and just misunderstood something, you will have an (flawed) justification for why you wrote this. If there's something flawed in an LLM, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. One shows a misunderstanding, the other doesn't necessarily show any understanding at all.
  - Zababa 9mo ago
    >If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. You could say the same about what people find on the web, yet LLMs are penalized more than web search. >If you do not use LLMs and just misunderstood something, you will have an (flawed) justification for why you wrote this. If there's something flawed in an LLM, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. Swap "LLMs" for "websites" and you could say the exact same thing. The author has this in their conclusions: >One clear conclusion is that the vast majority of students do not trust chatbots. If they are explicitly made accountable for what a chatbot says, they immediately choose not to use it at all. This is not true. What is true is that if the students are more accountable for their use of LLMs than their use of websites, they prefer using websites. What is "more" here? We have no idea, the author didn't say so. It could be that an error from a website or your own mind is -1 point and from a LLM is -2, so LLMs have to make two times less mistakes than websites and your mind. It could be -1 and -1.25. It could be -1 and -10. The author even says themselves: >In retrospect, my instructions were probably too harsh and discouraged some students from using chatbots. But they don't note the bias they introduced against LLMs with their notation.
shevy-java 9mo ago
He describes mostly a process where the exam itself, or rather testing the knowledge of a student, is not so important. I think not all exams can occur like that. In some cases you just have to test one's knowledge about a specific topic, and knowing facts is a very, very easy way to test this. I would agree that just focusing on facts these days is overrated, but I would still reason that it is not a useless metric still. So, when the author describes "bring your own exam questions", it more means that the exam itself is not so relevant, which is fine - but saying that university exams are now useless in the age of autosolving chatbots, is simply wrong. It just means that the exam itself is not important; that in itself does not automatically mean that ALL exams or exam styles are useless. Also, it depends on what you test. For instance, testing solving math questions - yes, chatbots can solve this, but can a student solve the same without needing a chatbot? How about practical skills? Ok, 3D printing will dominate, but the ability to craft something with your own hands, that is still a skill that may be useful, at the least to some extent. I feel that the whole discussion about chatbots dumbs down a lot. Skills have not become irrelevant just because chatbots exist.
anal_reactor 9mo ago
> I was completely flabbergasted because, to me, discussing "What questions did you have?" was always part of the collaboration between students When I was a student, professors maintained a public archive of past exams. The reason was obvious: next time the questions would be different, and memorizing past answers wouldn't help you if you don't understand the core ideas being taught. Then I took part in an exchange program and went to some shit-tier uni and I realized that collaboration was explicitly forbidden because professors would usually ask questions along "what was on slide 54". My favorite part was when professor said "I can't publish the slides online because they're stolen from another professor but you can buy them in the faculcy's shop". My uni maintained a giant presence on Facebook - we'd share a lot of information, and the most popular group was "easy courses" for students who wanted to graduate but couldn't afford a difficult elective course. The exchange uni had none of that. Literally no community, no collaboration, nothing. It's astonishing. BTW regarding the stream of consciousness - I distinctly remember taking an exam and doing my best to force my brain to think about the exam questions, rather than porn I had been watching the previous day.
yazantapuz 9mo ago
Paper and pencil.
palijer 9mo ago
I'm back in school part time for a bachelor's, and have recently had a class where I had a professor who really understood how to implement LLM's into the class. Our written assignments were a lot of "have an LLM generate a business proposal, then annotate it yourself" The final exam was a 30 minute meeting where we just talked as peers, kinda like a cultural job interview. Sure there's lots of potential for bias there, but I think it's better than just blindly passing students using LLM's for the final exam.
zahlman 9mo ago
> I realized that my students are so afraid of cheating that they mostly don’t collaborate before their exams! At least not as much as what we were doing. This is radically different from the world that's been described to me. Even 20 years ago cheating was endemic and I've only heard of it getting worse.
- mnw21cam 9mo ago
  I teach at MSc level, and over the last couple of years about 15% of my students have cheated. Really obviously cheated, like two students submitting 100% byte-identical answers.
- quietbritishjim 9mo ago
  But this class is not very representative of what the same students are presumably doing for other classes.
zoobab 9mo ago
Louvain-Li-Nux forever!
sinuhe69 9mo ago
What a wonderful teacher! I wish all teachers were like him. Regarding the collaboration before the exam, it's really strange. In our generation, asking or exchanging questions was perfectly normal. I got an almost perfect score in physics thanks to that. I guess the elegant solution was still in me, but I might not have been able to come up with it in such a stressful situation. 'Almost' because the professor deducted one point from my score for being absent too often :) However, oral exams in Europe are quite different from those at US universities. In an oral exam, the professor can interact with the student to see if they truly understand the subject, regardless of the written text. Allowing a chatbot during a written exam today would be defying the very purpose of the exam.
internet_points 9mo ago
Wow. I would love to have had this teacher :-) Everything about this setup seems so thoughtful. Giving students both agency / freedom of choice and responsibility. And if they choose more power (llms), they automatically have more responsibility (having to explain the mistakes of llms). And this: > It took me 20 years after university to learn what I know today about computers. And I’ve only one reason to be there in front of you: be sure you are faster than me. Be sure that you do it better and deeper than I did. If you don’t manage to outsmart me, I will have failed.
love2read 9mo ago
I went to a university class in my last year that had the same “choice”. Any time the professor starts asking about “ai” tends to be worrisome in my experience.
- eks391 9mo ago
  Why was it worrisome?
yardie 9mo ago
> When one would finish her exam, she would come back to the room and tell all the remaining students what questions she had and how she solved them. We never considered that "cheating" and, as a professor, I always design my exams hoping that the good one (who usually choose to pass the exam early) will help the remaining crowd. You are an outlier. When I was in school any outside assistance was tantamount to cheating and, unlike an actual crime, it was on the student to prove they were not cheating. Just the suspicion was enough to get you put in front of an honor board. It was also pervasive. I would say 40% of international students were cheaters. When some were caught they fell back on cultural norms as their defense. The university never balked because those students, or their institutions, paid tuition in cash.
- SoftTalker 9mo ago
  International students in graduate programs at US institions are basically buying a degree from what I've seen. The professors know they cheat and they don't really care. The students are paying a lot of money and they will get what they paid for.
  - pavel_lishin 9mo ago
    > The professors know they cheat and they don't really care. To throw another anecdote in the bucket, I know at least one professor who does not tolerate cheating from any of his students, regardless of cultural or national background, or how they're paying for their education
    - yardie 9mo ago
      I've seen, on multiple occasions, the professor's recommendations get overruled by the dean or university administration. If the school wants them there, they stay.
      - ProllyInfamous 9mo ago
        Andés Hess (RIP) gave an examination, 2006, in his Organic Chemistry course... which ended up with 35% of the class being reported to Vanderbilt's Honor Council. He brilliantly tested students using open-ended, single-sentence questions (with half of the page blank to show your work)... which tested foundational topics and oozed with partial-credit opportunities. You then had an option to submit "test corrections" to explain why you should gain more points for your efforts (typically considered, when reasonable). ---- His first exam of the semester, there was a multi-step question which resulted in a single 1cm x 1cm box — worth 20% of the entire exam's scoring — for you to indicate whether that particular Grignard reaction resulted in a single-, double-, or triple- bond. The majority of the class answered (incorrectly) that it would be a double-bond, by writing a `=` into the blank box. In fact, that reaction resulted in a triple-bond `≡` 35% of the class ended up just adding the third parallel line (i.e. changing what they had originally answered) when handing in their test corrections. Dr. Hess had made photocopies of all the penciled exams... and reported all the cheaters. ---- I answered it correctly, originally, so was never tempted to fib a similar mistake — but this definitely opened my eyes in reinforcement of not cheating. I eventually got into medical school, and most of that 35% of branded "cheaters" did not. Ultimately I never became a physician, but remember the temptations to cheat like everybody else did. I am happier/poorer because...
doug_durham 9mo ago
I find it odd that the Professor requires the students to demonstrate why the output of the LLM is correct, while not having the same requirement for a Google search. Even pre-AI using Google required critical thinking skills because the content was driven by SEO. If someone could make money by giving you mis-information it was pushed up in the rankings.
vonneumannstan 9mo ago
90% weight to in-person exams without technology. 10% to quizzes or homework. You can't trust anything done outside of the classroom to accurately show competency. Problem solved.
petterroea 9mo ago
A similar outlook scandal happened in Norway. I believe both UiO and NTNU, biggest in Humanities and engineering respectively, used to have great internal email services. Huge protests from competent personnel ignored. Now its all outlook. Microsoft are excessively good at convincing non technical stakeholders that their own staff are idiots, and that Microsoft is the solution. I have corrected exams and graded assignments as an external party before (legal requirement). The biggest problem with LLMs I see is that the weak students copy-paste commands with unnecessary command line switches. But they would have done the same with stack overflow. Some also say they use LLM to help improve their writing but that's where the learning is so why????? I think it's the anxiety for failing, they don't seem to understand I'll not fail them as long as their incoherent text proves they understood what they were doing. Having graduated and knowing how things are ought to look, taking exams are so much less scary now because I'm confident I will be failed for being incompetent, not because I didn't write properly. Not all students have the same privilege, they gain it over time. It does help that computer science assignments and papers are pretty damn standard in form.
scotty79 9mo ago
If what you are trying to test is whether the student will be able to give good answers to those and similar questions in the future, then it's fine if they use AI to answer exam questions. Because in the future, when they are confronted with these and similar questions, they will have access to at least as good AI as they do have today. And if you want to test something else than whether the student can provide answer to your question, then why do you ask this question? Give a task that shows what you care about.
gloomyday 9mo ago
I remember my first programming class, where the exam required students to solve problems in Pascal on paper. I see no problems in this approach. I still think pen and paper is king among students.
djoldman 9mo ago
> Second, I learned that cheating, however lightly, is now considered a major crime. It might result in the student being banned from any university in the country for three years. Discussing exam with someone who has yet to pass it might be considered cheating. Students have very strict rules on their Discord. > I was completely flabbergasted because, to me, discussing "What questions did you have?" was always part of the collaboration between students. I suspect that lots of intelligent and diligent students hate our new world of AI because they probably find it more likely now that they could be accused of and disciplined for something they didn't do.
djoldman 9mo ago
This professor's capacity for empathy and compassion for his students is off the charts. You can tell he puts a lot of thought and effort into helping his students learn. Bravo. Also, the take on AI is a stark contrast to much of what we've seen by other educators.

257 points by ploum 9mo ago

Very interesting write up, would be curious to know more about what an Open Source Strategies course entails, as far as I can remember I never had anything like that on offer at my university.

nkapias 9mo ago
https://uclouvain.be/cours-2025-linfo2401 https://uclouvain.be/cours-2025-linfo2401 https://uclouvain.be/cours-2025-linfo2402 https://uclouvain.be/cours-2025-linfo2402

sublang 9mo ago

[dead]

emil-lp 9mo ago

> Most Students Don’t Want to Use Chatbots I think this is changing rapidly. I'm a university professor, and the amount of students who seem to be in need of LLM as a crutch is growing really exponentially. We are still in a place where the oldest students did their first year completely without LLMs. But younger students have used LLMs throughout their studies, and I fear that in the future, we will see full generations of students completely incapable of working without LLM assistance.

teaearlgraycold 9mo ago
It will be very interesting to see what will happen when LLMs start charging users for their true cost. With many people priced out how would they cope?
- alphabetag675 9mo ago
  They would fall behind in the world just like people from developing and poor countries do today.
  - josephg 9mo ago
    Very few people fall behind at the moment due to lack of access to information. People in poor countries largely have access to the internet now. It doesn’t magically make people educated and economically prosperous.
    - alphabetag675 9mo ago
      You are arguing the converse. Access to information doesn't make people educated, but lack of access definitely puts people at a big advantage. Chatbots are not just information, they are tools and using it needs training because they hallucinate.
- KeplerBoy 9mo ago
  It's not that expensive unless you run millions of tokens through an agent. For use cases where you actually read all the input and output by yourself (i.e. an actual conversation), it is insanely cheap.
  - tucnak 9mo ago
    Yeah in my last job, unsupervised dataset-scale transformations amounted to 97% of all spending. We were using gemini 2.5 flash in batch/prefill-caching mode in Vertex, and always the latest/brightest for ChatGPT-like conversations.
- ben_w 9mo ago
  May happen, but I suspect not in the way implied by that question. Hardware is still improving, though not as fast as it used to; it's very plausible that even the current largest open weights models will run on affordable PCs and laptops in 5 years, and high-end smartphones in 7. I don't know how big the SOTA close-weights models are, that may come later. But: to the extent that a model that runs on your phone can do your job, your employer will ask "why are we paying you so much?" and now you can't afford the phone. Even if the SOTA is always running ahead of local models, Claude Code could cost 1500 times as much and still have the average American business asking "So why did we hire a junior? You say the juniors learn when we train them, I don't care, let some other company do that and we only hire mid-tier and up now." (Threshold is less than 1500 elsewhere, I just happened to have recently seen the average US pay for junior-grade software developers, $85k*, which is 350x cheaper, and my own observation that they're not only junior quality but also much faster to output than a junior). * but also note while looking for a citation the search results made claims varying from $55k to $97.7k
- johndough 9mo ago
  What do you think the "true cost" is?
kubb 9mo ago
Please, you don’t need to counter-narrative everything. Maybe talk about what the professor did here and why students didn’t trust the output in an exam context in this particular subject.
- nicce 9mo ago
  > Second, I learned that cheating, however lightly, is now considered a major crime. It might result in the student being banned from any university in the country for three years. Discussing exam with someone who has yet to pass it might be considered cheating. Students have very strict rules on their Discord. This has also something to do with it. Hard to make very accurate conclusions.
raesene9 9mo ago
Reading the article, it seemed to me that both the professor and the students were interested in the material being taught and therefore actively wanted to learn it, so using an LLM isn't the best tactic. My feeling is that for many/most students, getting a great understanding of the course material isn't the primary goal, passing the course so they can get a good job is the primary goal. For this group using LLMs makes a lot of sense. I know when I was a student doing a course I was not particularly interested in because my parents/school told me that was the right thing to do, if LLMs had been around, I absolutely would have used them :).
deleted 9mo ago
[deleted]
themafia 9mo ago
Google destroyed search and replaced it with that dippy LLM box. Are you sure student desire is the driving force here?
jcattle 9mo ago
I think the professor here presented them with a "special" case which can not be generalized outside of the exam context. If you're presented with the choice of "Don't use AI" and "Use AI, but live with the consequences" (consequences like mistakes being judged harsher when using AI than when not using AI), I do not think chatbots will be a desirable choice if you've properly prepared for the exam.
Jedd 9mo ago
> ... is growing really exponentially. Or geometrically?
petterroea 9mo ago
One helper here is fear. You can be failed for formal errors at university, and it means we were scared shitless of making them and payed close attention. If people know "at university you can't use LLM, you are forced to think by yourself" they will adjust, albeit by trial of fire. I think there's an argument that growing up in an educational system unable to teach you how to not rely on LLM would for all intents and purposes permanently nerf you compared to more fortunate peers. Critical thought is a skill we continue to practice until the very end

mkirsten 9mo ago

Interesting write up! I’ve thought about how university exams are done effectively nowadays. I took my degree in CS almost 20 years ago, and being a user of LLMS - I can’t really see how any of my old exams would work today if students would be allowed LLMs.

emil-lp 9mo ago
Spoiler: they don't. CS exercises that we can expect an average student to solve is trivially solved by LLMs. Even smaller local models.
- bandrami 9mo ago
  Except some times they aren't solved by LLMs, but appear to be. A CS student should be able to look at it and tell the difference.
- Ekaros 9mo ago
  This comes to not "smartness" of LLMs. But reality that we do not even want anything novel in these exercises or exams. And same areas are repeated multiple times so naturally there is lot of these in training data. This is one area where LLMs really should excel at. And that doesn't really mean that students should not also learn it and be able to solve same issues. Which is real dilemma for the school system...
brainwad 9mo ago
I graduated 15 years ago, and I think the exams in my degree were actually the most LLM-proof part of the student assessment. They were no-aid written exams with pencils and paper, whereas the assignments were online-submitted code only that an LLM could easily write.
deleted 9mo ago
[deleted]

lucb1e 9mo ago

> 3. I allow students to discuss among themselves [during an exam] if it is on topic. Makes me wonder if they should also get a diploma together then, saying "may not have the tested knowledge if not accompanied by $other_student" I know of some companies that support hiring people as a team (either all or none get hired and they're meant to then work together well), so it wouldn't necessarily be a problem if they wish to be a team like that

witcher 9mo ago
Yea, curious too about some more rules e.g. both parties has to contribute to the discussion (:
kubb 9mo ago
I think we should send all diplomas to OpenAI and end higher education. Less educated people are easier to steer via TikTok feeds anyway.
elbci 9mo ago
ha ha fair enough - but he does mention there's a culture of isolation and cu-throat competition at the school so, maybe it's just a reaction to that
ploum 9mo ago
OP here: I teach Open Source Strategies. The main strategy is collaboration. If you are smart enough to: 1. Identify your problem 2. Ask someone about it 3. Get an answer which improve your understanding Then you are doing pretty good by all standards Another trick I sometimes use. I take one student which has hard time articulate it a concept. I take student two who don’t understand that concept. I say to student 1: "You have 20 minutes to teach student 2 the concept. When I come back, you will be graded according to his answers" (I, of course, not only grade that. But it forces both of them to make an extra effort, student 2 not willing to be the cause for student 1 demise)
- zahlman 9mo ago
  > student 2 not willing to be the cause for student 1 demise I would very much not count on that.

witcher 9mo ago

Quite a thoughtful way to adapt exams to wave of new tools for students and learn on the way. I wished other universities adapt so quickly too (and have such a mindful attitude to students e.g. try to understand them, be upfront with expectations, learning from students etc). Majority of professors are stressed and treat students as idiots... at least that was the case decade a go!

ploum 9mo ago
OP here: Majority of professors became professors because there were very good at passing standard exam (and, TBH, some are not good at anything else). I’m different because I was a bad student. Only managed to get my diploma with minimal grade, always rebel against everything. But some good people at my university thought that Open Source was really important and they needed someone with a good career in that field. I was that person (and I’m really thankful for offering that position)
- malshe 9mo ago
  > Majority of professors became professors because there were very good at passing standard exam (and, TBH, some are not good at anything else). Is this a French thing? In the US we don't have standardized exams to become a college professor. Instead, we need to do original research and publish.

elbci 9mo ago

rare here: well written and insightful, I would take this course. I'm curious about why he penalized chatbot mistakes more, at first glance sounds like just discouraging their use but the hole setup indicates genuine desire to let it be a possibility. In my mind the rule should be "same penalty and extra super cookies for catching chatbot mistakes"

qwertytyyuu 9mo ago
Here is my guess: Usually marks are given for partially correct answers, partially to be less punishing for human error whether caused by stress or other factors, there’s a good chance the student understood the topic. If instead they are using a chat bot, but didn’t catch the mistake themselves, it’s an indication of less understanding and marked accordingly.
jcattle 9mo ago
I wrote this before to another comment like yours: I thought this part of penalizing mistakes made with the help of LLMs more was quite ingenious. If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. If you do not use LLMs and just misunderstood something, you will have a (flawed) justification for why you wrote this. If there's something flawed in an LLM answer, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. One shows a misunderstanding, the other doesn't necessarily show any understanding at all.

burgerone 9mo ago

I wish we could take our exams this way. It seems like a very interesting approach :)

barbegal 9mo ago

Only 2 students actually used an LLM in his exam, one well and one poorly so I'm not sure there is much you can draw from this experience. In my experience LLMs can significantly speed up the process of solving exam questions. They can surface relevant material I don't know about, they can remember how other similar problems are solved a lot better than I can and they can check for any mistakes in my answer. Yes when you get into very niche areas they start to fail (and often in a misleading way) but if you run through practise papers at all you can tell this and either avoid using the LLM or do some fine tuning on past papers.

quacked 9mo ago

Something that I think many students, indeed many people, struggle with is the question "why should I know anything?" For most of us--myself included--once you graduate from college, the answer is: "enough to not get fired". This is far less than most curriculums ask you to know, and every year, "enough to not get fired" is a lower and lower bar. With LLMs, it's practically on the floor for 90% of full-time jobs. That is why I propose exactly the opposite regimen from this course, although I admire the writer's free thinking. Return to tradition, with a twist. Closed-book exams, no note sheets, all handwritten. Add a verbal examination, even though it massively increases examination time. No homework assignments, which encourage "completionist mindset", where the turning-in of the assignment feels more real than understanding the assignment. Publish problem sets thousands of problems large with worked-out-solutions to remove the incentive to cheat. "Memorization is a prerequisite for creativity" -- paraphrase of an HN comment about a fondly remembered physics professor who made the students memorize every equation in the class. In the age of the LLM, I suspect this is triply true.

6LLvveMx2koXfwn 9mo ago
This is all very well if the goal was to sift the wheat from the chaff - but modern western education is about passing as many fee paying students as possible, preferably with a passably enjoyable experience for the institutional kudos.
- josephg 9mo ago
  I wonder if education will bifurcate back out as a result of AI. Small, bespoke institutions which insist on knowledge and difficult tests. And degree factories. It seems like students want the degree factory experience with the prestige of an elite institution. But - obviously - that can’t last long. Colleges and universities should decide what they are and commit accordingly.
  - graemep 9mo ago
    I think the UK has been heading this way for a while -- before AI. Its not been the size of the institutions that has changed, but the "elite" universities tend to give students more individual attention. A number of them (not just Oxford and Cambridge) have tutorial systems where a lot of learning is done in a small group (usually two or three students). They have always done this. At the other extreme are universities offering low quality courses that are definitely degree factories. They tend to have a strong vocational focus but nonetheless they are not effective in improving employability. In the last few decades we have expanded the university system and there are far more of these. There is no clear cutoff and a lot of variation in between so its not a bifurcation but the quality vs factory difference is there.

pautasso 9mo ago

The problem is when students just blindly copy and paste from the chatbot and submit it as their own answer without even reading it. They should be encouraged to read and review the LLM output so they can critically understand it and take ownership of it.

dyauspitr 9mo ago
In my experience, reading a solution and even understanding it doesn’t go very far in teaching you how to do something. I can look at calculus solutions all day but only when I actually try to solve them myself do I run into all kinds of roadblocks which is where the real learning happens.
- 9rx 9mo ago
  You're right, but learning can take place when you need it. There is no real advantage to learning something ahead of time. The bottleneck is having awareness of what is out there to learn. You can't learn about what you don't know exists. Looking at calculus solutions all day should give a sense of what calculus can be used for, so that it is in your back pocket when the time you need it comes. Well, at least it used to be the bottleneck. Nowadays you can just ask an LLM. For all their faults, they are really good at letting you know about what tools exist in out there in the world, surfacing more than you could ever come to know about even if all you did was read about what exists all day, every day.
  - dyauspitr 9mo ago
    I believe to count as an expert on something you need to have a ready compendium of knowledge ready to go. It becomes very hard to tackle problems or gain deep insights if you don’t already have knowledgeable people that have thought deeply about a particular space. Maybe when we have supremely reliable LLMs that can replace humans we might not but we’re not there yet.
    - 9rx 9mo ago
      > I believe to count as an expert on something you need to have a ready compendium of knowledge ready to go. You are certainly headed in the right direction, but not quite. To be seen as an expert in the eyes of others you need to have had a vision for something and to have successfully executed on it. If the vision was dependent on calculus, then you will have reached a point where you had to learn something about calculus, of course... But that's different to having a taskmaster tell you to learn calculus for no apparent reason. Even if you follow through and built up a huge wealth of knowledge from it, you would still not be deemed an expert by others. You're no different than an encyclopedia, which isn't an expert either. It is being able to see things others can't and the ability to act upon it that makes an expert. Learning taking place when you need it isn't the same as never. > Maybe when we have supremely reliable LLMs that can replace humans we might not but we’re not there yet. Frankly, even Page Rank already replaced humans for this. But LLMs are even better at it. Humans are just that poorly performing. Like I said before, even someone doing nothing in life but looking for what exists in the world could not take in as much as databases that have indexed every written thing.

veltas 9mo ago

> The third chatbot-using student had a very complex setup where he would use one LLM, then ask another unrelated LLM for confirmation. He had walls of text that were barely readable. When glancing at his screen, I immediately spotted a mistake (a chatbot explaining that "Sepia Search is a compass for the whole Fediverse"). I asked if he understood the problem with that specific sentence. He did not. Then I asked him questions for which I had seen the solution printed in his LLM output. He could not answer even though he had the answer on his screen. Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe. That guy who is playing with the latest tech, and forcing it to do the job (badly), and could care less about university or the course he's on. There's a time and a place where that guy is the one you want working for you. Maybe he's not the number 1 student, but I think there should be some room for this to be the Chaotic Neutral pick.

watwut 9mo ago
> Is it possible, and this is an interesting one to me, that this is the smartest kid in the class? I think maybe. He might as well be the dumbest guy in the class. Playing with tech is not a proof of being smart on itself.
cheezburger 8mo ago
what are you smoking?

Joel_Mckay 9mo ago

"Marking Exam Done by A.I." (Sixty Symbols) https://www.youtube.com/watch?v=JcQPAZP7-sE https://www.youtube.com/watch?v=JcQPAZP7-sE LLM reasoning models are very good at searching well documented problems. =3

knallfrosch 9mo ago

I don't understand. 10 years ago, we wrote exams by hand with whatever we understood (in our heads.) No colleagues, no laptops, no internet, no LLMs. This approach still works, why do something else? Unless you're specifically testing a student's ability to Google, they don't need access to it.

tpoacher 9mo ago
Optics.
zavec 9mo ago
It was the same when I graduated 6 years ago. We had projects to test our ability to use tools and such, and I guess in that context LLMs might be a concern. But exams were pencil and paper only.
nikitau 9mo ago
Open book exams are not a new thing and I've often had them for STEM disciplines (maths and biology). Depending on the subject, you will often fail those unless you had a good prior understanding of the material. If you can pass an exam just by googling something, it means you're just testing rote-memorization rather, and maybe a better design is needed where synthesis and critical thinking skills are evaluated more actively.
- HWR_14 9mo ago
  Open book, sure. But you don't even need a computer for that.
  - freehorse 9mo ago
    And even if you are allowed to use a computer, you cannot use internet (and should not be hard to prevent that).
    - croes 9mo ago
      Local LLMs

bsder 9mo ago

> We were imposed GitHub for so many exercises! I'm sympathetic to both sides here. As a professor who had to run Subversion for students (a bit before Git, et al), it's a nightmare to put the infrastructure together, keep it reliable under spiky loads (there is always a crush at the deadline), be customer support for students who manage to do something weird or lose their password, etc. You wind up spending a non-trivial amount of time being sysadmin for the class on top of your teaching duties. Being able to say "Put it on GitHub" short circuits all of that. It sucks, but it makes life a huge amount easier for the professor. From the students point of view, sure, it sucks that nobody mentioned that Git could be used independently (or jj or Mercurial or ...) However, Github is going to be better than what 99.9% of all professors will put together or be able to use. Sure, you can use Git by itself, but then it needs to go somewhere that the professor can look at it, get submitted to automated testing, etc. That's not a trivial step. My students were happy that I had the "Best Homework Submission System" (said about Subversion of all things ...) because everybody else used the dumbass university enterprise thing that was completely useless (not going to mention its name because it deserves to die in the blazing fires of the lowest circle of Hell). However, it wasn't straightforward for me to put that together. And the probability of getting a professor with my motivation and skill is pretty low.

i_am_proteus 9mo ago
Agree about the possibility of infra nightmare, especially in the "SVN era" -- but in 2026, it's pretty straightforward to run a gitlab instance (takes about an hour to set up, most of which is DNS and TLS stuff, ime) for a course and set up actions, or use other submission infra like CMU autolab. I do this. Agree with your comment about probability, motivation, and skill.

SwtCyber 9mo ago

If anything, this reinforces the idea that chatbots don't fundamentally change education... they just amplify whatever incentives and structures already exist

vaylian 9mo ago
I am not so sure about that. I think the difference this time is that AI usage is fairly mainstream. People are using it for all kinds of things, including studies. Using it to quickly get done with school/study-work, is a no-brainer (pun intended).

fexed 9mo ago

What a wonderful article, and what a wonderful way of enganging with students and adapting to the new tech. I wish all professors were like you

Zababa 9mo ago

> Mistakes made by chatbots will be considered more important than honest human mistakes, resulting in the loss of more points. >I thought this was fair. You can use chatbots, but you will be held accountable for it. So you're held more accountable for the output actually? I'd be interested in how many students would choose to use LLMs if faults weren't penalized more.

jcattle 9mo ago
I thought this part especially was quite ingenious. If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. If you do not use LLMs and just misunderstood something, you will have an (flawed) justification for why you wrote this. If there's something flawed in an LLM, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. One shows a misunderstanding, the other doesn't necessarily show any understanding at all.
- Zababa 9mo ago
  >If you have this great resource available to you (an LLM) you better show that you read and checked its output. If there's something in the LLM output you do not understand or check to be true, you better remove it. You could say the same about what people find on the web, yet LLMs are penalized more than web search. >If you do not use LLMs and just misunderstood something, you will have an (flawed) justification for why you wrote this. If there's something flawed in an LLM, the likelihood that you do not have any justification except for "the LLM said so" is quite high and should thus be penalized higher. Swap "LLMs" for "websites" and you could say the exact same thing. The author has this in their conclusions: >One clear conclusion is that the vast majority of students do not trust chatbots. If they are explicitly made accountable for what a chatbot says, they immediately choose not to use it at all. This is not true. What is true is that if the students are more accountable for their use of LLMs than their use of websites, they prefer using websites. What is "more" here? We have no idea, the author didn't say so. It could be that an error from a website or your own mind is -1 point and from a LLM is -2, so LLMs have to make two times less mistakes than websites and your mind. It could be -1 and -1.25. It could be -1 and -10. The author even says themselves: >In retrospect, my instructions were probably too harsh and discouraged some students from using chatbots. But they don't note the bias they introduced against LLMs with their notation.

shevy-java 9mo ago

He describes mostly a process where the exam itself, or rather testing the knowledge of a student, is not so important. I think not all exams can occur like that. In some cases you just have to test one's knowledge about a specific topic, and knowing facts is a very, very easy way to test this. I would agree that just focusing on facts these days is overrated, but I would still reason that it is not a useless metric still. So, when the author describes "bring your own exam questions", it more means that the exam itself is not so relevant, which is fine - but saying that university exams are now useless in the age of autosolving chatbots, is simply wrong. It just means that the exam itself is not important; that in itself does not automatically mean that ALL exams or exam styles are useless. Also, it depends on what you test. For instance, testing solving math questions - yes, chatbots can solve this, but can a student solve the same without needing a chatbot? How about practical skills? Ok, 3D printing will dominate, but the ability to craft something with your own hands, that is still a skill that may be useful, at the least to some extent. I feel that the whole discussion about chatbots dumbs down a lot. Skills have not become irrelevant just because chatbots exist.

anal_reactor 9mo ago

> I was completely flabbergasted because, to me, discussing "What questions did you have?" was always part of the collaboration between students When I was a student, professors maintained a public archive of past exams. The reason was obvious: next time the questions would be different, and memorizing past answers wouldn't help you if you don't understand the core ideas being taught. Then I took part in an exchange program and went to some shit-tier uni and I realized that collaboration was explicitly forbidden because professors would usually ask questions along "what was on slide 54". My favorite part was when professor said "I can't publish the slides online because they're stolen from another professor but you can buy them in the faculcy's shop". My uni maintained a giant presence on Facebook - we'd share a lot of information, and the most popular group was "easy courses" for students who wanted to graduate but couldn't afford a difficult elective course. The exchange uni had none of that. Literally no community, no collaboration, nothing. It's astonishing. BTW regarding the stream of consciousness - I distinctly remember taking an exam and doing my best to force my brain to think about the exam questions, rather than porn I had been watching the previous day.

yazantapuz 9mo ago

Paper and pencil.

palijer 9mo ago

I'm back in school part time for a bachelor's, and have recently had a class where I had a professor who really understood how to implement LLM's into the class. Our written assignments were a lot of "have an LLM generate a business proposal, then annotate it yourself" The final exam was a 30 minute meeting where we just talked as peers, kinda like a cultural job interview. Sure there's lots of potential for bias there, but I think it's better than just blindly passing students using LLM's for the final exam.

zahlman 9mo ago

> I realized that my students are so afraid of cheating that they mostly don’t collaborate before their exams! At least not as much as what we were doing. This is radically different from the world that's been described to me. Even 20 years ago cheating was endemic and I've only heard of it getting worse.

mnw21cam 9mo ago
I teach at MSc level, and over the last couple of years about 15% of my students have cheated. Really obviously cheated, like two students submitting 100% byte-identical answers.
quietbritishjim 9mo ago
But this class is not very representative of what the same students are presumably doing for other classes.

zoobab 9mo ago

Louvain-Li-Nux forever!

sinuhe69 9mo ago

What a wonderful teacher! I wish all teachers were like him. Regarding the collaboration before the exam, it's really strange. In our generation, asking or exchanging questions was perfectly normal. I got an almost perfect score in physics thanks to that. I guess the elegant solution was still in me, but I might not have been able to come up with it in such a stressful situation. 'Almost' because the professor deducted one point from my score for being absent too often :) However, oral exams in Europe are quite different from those at US universities. In an oral exam, the professor can interact with the student to see if they truly understand the subject, regardless of the written text. Allowing a chatbot during a written exam today would be defying the very purpose of the exam.

internet_points 9mo ago

Wow. I would love to have had this teacher :-) Everything about this setup seems so thoughtful. Giving students both agency / freedom of choice and responsibility. And if they choose more power (llms), they automatically have more responsibility (having to explain the mistakes of llms). And this: > It took me 20 years after university to learn what I know today about computers. And I’ve only one reason to be there in front of you: be sure you are faster than me. Be sure that you do it better and deeper than I did. If you don’t manage to outsmart me, I will have failed.

love2read 9mo ago

I went to a university class in my last year that had the same “choice”. Any time the professor starts asking about “ai” tends to be worrisome in my experience.

eks391 9mo ago
Why was it worrisome?

yardie 9mo ago

> When one would finish her exam, she would come back to the room and tell all the remaining students what questions she had and how she solved them. We never considered that "cheating" and, as a professor, I always design my exams hoping that the good one (who usually choose to pass the exam early) will help the remaining crowd. You are an outlier. When I was in school any outside assistance was tantamount to cheating and, unlike an actual crime, it was on the student to prove they were not cheating. Just the suspicion was enough to get you put in front of an honor board. It was also pervasive. I would say 40% of international students were cheaters. When some were caught they fell back on cultural norms as their defense. The university never balked because those students, or their institutions, paid tuition in cash.

SoftTalker 9mo ago
International students in graduate programs at US institions are basically buying a degree from what I've seen. The professors know they cheat and they don't really care. The students are paying a lot of money and they will get what they paid for.
- pavel_lishin 9mo ago
  > The professors know they cheat and they don't really care. To throw another anecdote in the bucket, I know at least one professor who does not tolerate cheating from any of his students, regardless of cultural or national background, or how they're paying for their education
  - yardie 9mo ago
    I've seen, on multiple occasions, the professor's recommendations get overruled by the dean or university administration. If the school wants them there, they stay.
    - ProllyInfamous 9mo ago
      Andés Hess (RIP) gave an examination, 2006, in his Organic Chemistry course... which ended up with 35% of the class being reported to Vanderbilt's Honor Council. He brilliantly tested students using open-ended, single-sentence questions (with half of the page blank to show your work)... which tested foundational topics and oozed with partial-credit opportunities. You then had an option to submit "test corrections" to explain why you should gain more points for your efforts (typically considered, when reasonable). ---- His first exam of the semester, there was a multi-step question which resulted in a single 1cm x 1cm box — worth 20% of the entire exam's scoring — for you to indicate whether that particular Grignard reaction resulted in a single-, double-, or triple- bond. The majority of the class answered (incorrectly) that it would be a double-bond, by writing a `=` into the blank box. In fact, that reaction resulted in a triple-bond `≡` 35% of the class ended up just adding the third parallel line (i.e. changing what they had originally answered) when handing in their test corrections. Dr. Hess had made photocopies of all the penciled exams... and reported all the cheaters. ---- I answered it correctly, originally, so was never tempted to fib a similar mistake — but this definitely opened my eyes in reinforcement of not cheating. I eventually got into medical school, and most of that 35% of branded "cheaters" did not. Ultimately I never became a physician, but remember the temptations to cheat like everybody else did. I am happier/poorer because...

doug_durham 9mo ago

I find it odd that the Professor requires the students to demonstrate why the output of the LLM is correct, while not having the same requirement for a Google search. Even pre-AI using Google required critical thinking skills because the content was driven by SEO. If someone could make money by giving you mis-information it was pushed up in the rankings.

vonneumannstan 9mo ago

90% weight to in-person exams without technology. 10% to quizzes or homework. You can't trust anything done outside of the classroom to accurately show competency. Problem solved.

petterroea 9mo ago

A similar outlook scandal happened in Norway. I believe both UiO and NTNU, biggest in Humanities and engineering respectively, used to have great internal email services. Huge protests from competent personnel ignored. Now its all outlook. Microsoft are excessively good at convincing non technical stakeholders that their own staff are idiots, and that Microsoft is the solution. I have corrected exams and graded assignments as an external party before (legal requirement). The biggest problem with LLMs I see is that the weak students copy-paste commands with unnecessary command line switches. But they would have done the same with stack overflow. Some also say they use LLM to help improve their writing but that's where the learning is so why????? I think it's the anxiety for failing, they don't seem to understand I'll not fail them as long as their incoherent text proves they understood what they were doing. Having graduated and knowing how things are ought to look, taking exams are so much less scary now because I'm confident I will be failed for being incompetent, not because I didn't write properly. Not all students have the same privilege, they gain it over time. It does help that computer science assignments and papers are pretty damn standard in form.

scotty79 9mo ago

If what you are trying to test is whether the student will be able to give good answers to those and similar questions in the future, then it's fine if they use AI to answer exam questions. Because in the future, when they are confronted with these and similar questions, they will have access to at least as good AI as they do have today. And if you want to test something else than whether the student can provide answer to your question, then why do you ask this question? Give a task that shows what you care about.

gloomyday 9mo ago

I remember my first programming class, where the exam required students to solve problems in Pascal on paper. I see no problems in this approach. I still think pen and paper is king among students.

djoldman 9mo ago

> Second, I learned that cheating, however lightly, is now considered a major crime. It might result in the student being banned from any university in the country for three years. Discussing exam with someone who has yet to pass it might be considered cheating. Students have very strict rules on their Discord. > I was completely flabbergasted because, to me, discussing "What questions did you have?" was always part of the collaboration between students. I suspect that lots of intelligent and diligent students hate our new world of AI because they probably find it more likely now that they could be accused of and disciplined for something they didn't do.

djoldman 9mo ago

This professor's capacity for empathy and compassion for his students is off the charts. You can tell he puts a lot of thought and effort into helping his students learn. Bravo. Also, the take on AI is a stark contrast to much of what we've seen by other educators.