7 ms·

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams.

• BoiledCabbage 10mo ago
  That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote regardless of what the data was going to show.
  • knallfrosch 10mo ago
    I found "well, the LLMs converge when given each other's scores, so they agree and are correct" to be quite the jump to a conclusion.
    • pooper 10mo ago
      accuracy versus precision is something we learn in high school chemistry. https://i.imgur.com/EshEhls.png https://i.imgur.com/EshEhls.png When someone at that level pretends to not understand it, there is no way to mince words. This is malice.
    • bsenftner 10mo ago
      I've got a long standing disagreement with an AI CEO that believes LLM convergence indicates greater accuracy. How to explain basic cause and effect in these AI use cases is a real challenge. The essential basic understanding of what an LLM is is not there, and that lack of comprehension is a civilization wide issue.
  • leoc 10mo ago
    At the risk of perhaps stating the obvious, there appears to be a whiff of aggression from this article. The "fighting fire with fire" language, the "haha, we love old FakeFoster, going to have to see if we change that" response to complaints that the voice was intimidating ... if there wasn't a specific desire to punish the class for LLM use by subjecting them to a robotic NKVD interrogation then the authors should have been more careful to avoid leaving that impression.
    • Hnrobert42 10mo ago
      You can try out the voice yourself. It's not that bad. https://elevenlabs.io/app/talk-to?agent_id=agent_8101k9d1pq41f3rs8d9g9143dvvr&vars=eyJzdHVkZW50IjoiS29uc3RhbnRpbm9zIFJpemFrb3MiLCAibmV0aWQiOiJrcjg4OCIsICJwcm9qZWN0aWQiOiJCIn0= https://elevenlabs.io/app/talk-to?agent_id=agent_8101k9d1pq4...
      • yayitswei 10mo ago
        Tried it in earnest. Definitely detect some aggression, and would feel stressed if this were an exam setting. I think it was pg who said that any stress you add in an interview situation is just noise, and dilutes the signal. Also, given that there's so many ways for LLMs to go off the rails (it just gave me the student id I was supposed to say, for example), it feels a bit unprofessional to be using this to administer real exams.
      • iamthepieman 10mo ago
        Also tried it and it could have been a lot better. If I had any type of interview with that voice (press interview, mentor interview, job interview) I would think I was being scammed, sold something, or had entered the wrong room.
      • Drupon 10mo ago
        Not that bad? I gave it a random name and random net ID and it basically screamed at me to HANG UP RIGHT NOW AND FIGURE OUT THE CORRECT NET ID. Hahaha That does not resemble any good professor I've ever heard. It's very aggressive and stern, which is not generally how oral exams are conducted. Feels much more like I'm being cross examined in court.
  • skybrian 10mo ago
    I don't think they're terrible, but I'm grading on a curve because it's their first attempt and more of a trial run. It seems promising enough to fix the issues and try again.
  • bjt 10mo ago
    They did compare the automated grades to the author's own manual ones. It's in there if you read more closely.
  • wanderingbit 10mo ago
    > And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! This is not true; the professor and the TAs graded every student submission. See this paragraph from the article: (Just in case you are wondering, I graded all exams myself and I asked the TA to also grade the exams; we mostly agreed with the LLM grades, and I aligned mostly with the softie Gemini. However, when examining the cases when my grades disagreed with the council, I found that the council was more consistent across students and I often thought that the council graded more strictly but more fairly.)
  • chairmansteve 10mo ago
    As far as I can tell, there is very little empirical evidence of efficacy for most modern educational "advances". Having said that, LLMs can be good tutors if used correctly.
• NewsaHackO 10mo ago
  The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.
  • Kwpolska 10mo ago
    Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening.
    • NewsaHackO 10mo ago
      I agree with you and the other posters actually, but I think the efficiency compared with typed work is the reason it’s having such a slow adoption. Another thing to remember is that there is always a mild Jevons paradox at play; while it's true that it was possible in previous centuries, teacher expectations have also increased which strains the amount of time they would have grading handwritten work.
    • aleph_minus_one 10mo ago
      > Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening. At least in Germany, if there are only 36 students in a class, usually oral exams are used because in this case oral exams are typically more efficient. For written exams, more like 200-600 students in a class is the common situation.
    • abdullahkhalids 10mo ago
      Not the comprehensive physics exams I assigned as a prof. A well set exam takes at least 20-30 min to grade. That's 8-12 hours of work, and in practice, took several sittings over several days. If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. I wanted to do oral exams, but they are much more exhausting for the prof. Nominally, each student is with you for 30 min, but (1) you need to think of slightly different question for each student (2) you need to squeeze all the exams in only a couple of days to avoid giving later students too much extra time to prepare.
• cvoss 10mo ago
  The quote you gave is not the conclusion of the article. It's a self-evident claim that just as well could have been the first sentence of the article ("take-home exams are dead"), followed by an opinion ("reverting ... feels like a regression") which motivated the experiment. Some universities and professors have tried to move to a take-home exam format, which allows for more comprehensive evaluation with easier logistics than a too-brief in-class exam or an hours-long outside-of-class sitting where unreasonable expectations for mental and sometimes physical stamina are factors. That "take-home exams are dead" is self-evident, not a result of the experiment in the article. There used to be only a limited number of ways to cheat at a take-home exam, and most of them involved finding a second person who also lacked a moral conscience. Now, it's trivial to cheat at a take-home exam all by yourself. You also mentioned the hundreds of years of experience universities have at traditional written exams. But the type and manner of knowledge and skills that must be tested for vary dramatically by discipline, and the discipline in question (computer science / software engineering) is still new enough that we can't really say we've matured the art of examining for it. Lastly, I'll just say that student preference is hardly the way to measure the quality of an exam, or much of anything about education.
  • michaelt 10mo ago
    > The quote you gave is not the conclusion of the article. Did I say "conclusion" ? Sorry, I should have said the section just before the acknowledgements, where the conclusion would normally be, entitled "The bigger point"
    • Nifty3929 10mo ago
      I think this is the actual conclusion: "Now, AI is making them scalable again." That is, the author concluded that AI tools provide viable alternatives to the other available options, and which solve many of their problems.
• vasco 10mo ago
  One student had to talk to an AI for more than 60 minutes. These guys are creating a dystopia. Also students will just have an AI pick up the phone if this gets used for more than 2 semesters.
  • j_w 10mo ago
    It's not that the oral format should be dismissed, just that the idea of your exam being speaking to a machine to be judged on the merit of your time in a course is dystopian. Talking to another human is fine.
    • makeitdouble 10mo ago
      How different is it in essence from checking boxes to be scanned by a machine and auto-evaluated to get a one dimention numerical score ? Have exams ever been about humanity and the optics of it ?
      • sarchertech 10mo ago
        Very different. A scantron machine is deterministic and non-chaotic. In addition to being non-deterministic LLMs can product vastly different output from very slightly different input. That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common. Also if this is about avoiding in person exams, what prevents students from just letting their AI talk to test AI.
        • makeitdouble 10mo ago
          I saw this piece as the start of an experiment, and the use of a "council of AI" as they put it to average out the variability sounds like a decent path to standardization to me (prompt injecting would not be impossible, but getting something past all the steps sounds like a pretty tough challenge) They mention getting 100% agreement between the LLMs on some questions and lower rates on other, so if an exam was composed of only questions where there is near 100% convergence, we'd be pretty close to a stable state. I agree it would be reassuring to have a human somewhere in the loop, or perhaps allow the students to appeal the evaluation (at cost?) if they is evidence of a disconnect between the exam and the other criteria. But depending on how the questions and format is tweaked we could IMHO end up with something reliable for very basic assessments. PS: > Also if this is about avoiding in person exams, what prevents students from just letting their AI talk to test AI. Nothing indeed. The arms race hasn't started here, and will keep going IMO.
• Panos 10mo ago
  Not the case for the class in the blog post, but we also have many online classes. Many professionals prefer these online classes because they can attend without having to commute, and can do it from a place of their own convenience. Such classes do not have the luxury of pen-and-paper exams, and asking people to go to testing centers is a huge overkill. Take home exams for such settings (or any other form of written exam) are becoming very prone to cheating, just because the bar to cheating is very low. Oral exams like that make it a bit harder to cheat. Not impossible, but harder.
  • ninalanyon 10mo ago
    I did a C# module online run by a Norwegian University. It was worth 6 points, 180 grants you a bachelor's degree in Norway (or did, I think there have been changes since). The course ran over ten weeks and there were weekly assignments. Of course it would have been easy to cheat on those but there would be no point because there was a five hour invigilated open book exam at the end of the course. Had to go to a testing centre about 35 km away to take the exam but that really wasn't a great inconvenience. If I had wanted to pursue a whole degree then I would have had 30 such exams, roughly one a month if you do the degree over the traditional three years. That doesn't seem like overkill to me, it's a lot less effort than attending lectures and tutorials for three years as I did for my Applied Physics degree.
• xp84 10mo ago
  > they expressed a clear preference for written exams When I was a student, I would have been quite vocal with my clear preferences for all exams being open-book and/or being able to amend my answers after grading for a revised score. What I'm saying is, "the students would prefer..." isn't automatically case closed on what's best. Obviously the students would prefer a take-home because you can look up everything you can't recall / didn't show up to class to learn, and yes, because you can trivially cheat with AI (with a light rewrite step to mask the "LLM voice"). But in real life, people really will ask you to explain your decisions and to be able to reason about the problem you're supposedly working on. It seems clear from reading the revised prompts that the intent is to force the agent to be much fairer and easier to deal with than this first attempt was, so I don't think this is a bad idea. Finally, (this part came from my reading of the student feedback quotes in the article) consider that the current cohort of undergrads is accustomed to communicating mainly via texting. To throw in a further complication, they were around 13-17 when COVID hit, decreasing human contact even more. They may be exceedingly nervous about speaking to anyone who isn't a very close friend. I'm sympathetic to them, but helping them overcome this anxiety with relatively low stakes is probably better than just giving up on them being able to communicate verbally.
  • jojomodding 10mo ago
    > being able to amend my answers after grading for a revised score How do you expect that to work? After the exam, you talk to your friends (and to ChatGPT) and know the correct answers even if you could have never produced them during the exam.
    • viccis 10mo ago
      Not the person you're replying to, but I've had some courses in which you received your graded exams and had an opportunity to regain some points by choosing some number of incorrect responses and redoing the work to obtain a correct answer. This was pre-LLM, but you could cheat back then too. LLMs make it a bit easier by showing you the work to "show" on your corrections.
• InfiniteRand 10mo ago
  I feel like the arms race between student cheaters and teacher testing has been going on for hundreds of years, ever since the first answer key written on the back of a hand
• chairmansteve 10mo ago
  They are in thrall to technology and "progress".