7 ms·
I don't understand. 10 years ago, we wrote exams by hand with whatever we understood (in our heads.) No colleagues, no laptops, no internet, no LLMs. This ap
by knallfrosch 9mo ago
I don't understand.
10 years ago, we wrote exams by hand with whatever we understood (in our heads.)
No colleagues, no laptops, no internet, no LLMs.
This approach still works, why do something else? Unless you're specifically testing a student's ability to Google, they don't need access to it.
- tpoacher 9mo agoOptics.
- zavec 9mo agoIt was the same when I graduated 6 years ago. We had projects to test our ability to use tools and such, and I guess in that context LLMs might be a concern. But exams were pencil and paper only.
- nikitau 9mo agoOpen book exams are not a new thing and I've often had them for STEM disciplines (maths and biology). Depending on the subject, you will often fail those unless you had a good prior understanding of the material. If you can pass an exam just by googling something, it means you're just testing rote-memorization rather, and maybe a better design is needed where synthesis and critical thinking skills are evaluated more actively.
- HWR_14 9mo agoOpen book, sure. But you don't even need a computer for that.
- freehorse 9mo agoAnd even if you are allowed to use a computer, you cannot use internet (and should not be hard to prevent that).
- croes 9mo agoLocal LLMs
- DonHopkins 9mo agoBe sure to bring an extra power strip for all your plugs and adaptors. https://www.tomshardware.com/pc-components/gpus/tiny-corp-successfully-runs-an-nvidia-gpu-on-arm-macbook-through-usb4-using-an-external-gpu-docking-station https://www.tomshardware.com/pc-components/gpus/tiny-corp-su...
- lostmsu 9mo agoMy laptop runs gpt-oss 120B with none of that. Don't know how long though. I suspect a couple of hours continuous.
- HWR_14 9mo agoWhich laptop?
- lostmsu 9mo agoROG Flow Z13 with maxxed out RAM.
- red-iron-pine 9mo agohow is anyone going to be able to take a test with all of the noise from that fan as it cranks through tokens?
- DonHopkins 9mo agoOffer to make everyone espresso and macchiato with you GPU cooling module. They won't be able to hear the fan over the grinder and pump and milk foamer!
- vcf 9mo agoI make a point of only using references that are either available for free online or through our university’s library subscriptions. These are all electronic. My open book exam became an open computer exam when I realized students were printing hundreds of pages just for a 3-hour exam. This semester I’m switching to no-computer, bring your own printed cheat-sheet for the exam.
- johannes1234321 9mo agoExcept that the physical book isn't the way people lookup facts these days. The open book test is purposes is to not have to know all facts (formulas) but proving how to find them and how to apply them. (Finding is part of it as the more you look, the less time you got to use it, thus there is an optimisation problem which things to remember and which to look up) In modern times you wouldn't look those up in a book, thus other research techniques are required to deal with real life (which advanced certifications should prove)
- 9rx 9mo agoGoing to university isn't how people learn these days, so there is already a real-world disconnect, fundamentally. But that's okay as it isn't intended to be a reflection of the real world.
- mock-possum 9mo ago> Going to university isn't how people learn these days That’s a surpsing statement that doesn’t ring true to me, what are you basing that off of / citing?
- 9rx 9mo ago> what are you basing that off of Observation? Children show clear signs of learning before they even make it through their first year out of the womb. Man, most people don't even consider university as an option until they are around 17-18 years of age, after they have already learned the vast majority of the things they will learn in life. Data? Only 7-8% of the population have a university degree. Obviously you could learn in university without graduating, and unfortunately participation data is much harder to come by, but there is no evidence to suggest that the non-completion rate is anywhere high enough to think that even a majority of the population have step foot in a university even if for just one for day. If we go as far as to assume a 50% dropout rate, that is still no more than 16% of the population. Little more than rounding error. Nothing? It's a random comment on the internet. It is not necessarily based on anything. Fundamentally, comments are only ever written for the enjoyment of writing. One trying to derive anything more from it has a misunderstanding of the world around them. I suppose you have a point that, for those who struggle to see the obvious, a university education would teach the critical thinking necessary to recognize the same. But, the fact that we are here echoes that university isn't how people learn these days. > citing Citing...? Like, as in quoting a passage? I can find no reason why I would want to repeat what someone else has written about. Whatever gives you enjoyment, but that seems like a pointless waste of time. It is already right there. You must be trying to say something else by this? I, unfortunately, am not in tune with your pet definition.
- SwtCyber 9mo agoI think the key difference is what you're trying to measure
- willis936 9mo agoI was in university around the same time. While there I saw a concerted effort to push online courses. Professors would survey students fishing for interest. It was unpopular. To me the motivation seemed clear: charge the same or more for tuition, but reduce opex. Maybe even admit more students to just have then be remote. It watered down the value of the degree while working towards a worse product. Why would a nonprofit public university be working on maximizing profit?
- iterateoften 9mo agoOnline courses are also increases admin overhead. Universities aren’t profit maximizing. They are admin maximizing. Admin are always looking to expand admins budget. Professors, classrooms, facilities all divert money away from admin and they don’t want to pay it unless they have to. Also applies to hospitals in USA.
- cudgy 9mo agoSo they can educate more students? Many university classes are lecture only with 200+ students in the class and no direct interactions with profs. Those courses might was well be online.
- 9rx 9mo ago> Why would a nonprofit public university be working on maximizing profit? Because 'nonprofit' is only in reference to the legal entity, not the profit-seeking people working there? There is still great incentive to increase profitability.
- b40d-48b2-979e 9mo agoYou're thinking of not-for-profit. Non-profits do not seek increased profitability in the same way since it's expected (mandated?) they don't have any.
- 9rx 9mo agoI'm not thinking of either. The profit-seeking people looking to increase their profitability spoken of are neither non-profits nor not-for-profits.
- everdrive 9mo agoPeople really struggle to go back once a technology has been adopted. I think for the most part, people cannot really evaluate whether or not the technology is a net positive; the adoption is more social than it is rational, and so it'd be like asking people to change their social values or behaviors.
- atomic_reed 9mo ago[dead]
- n4r9 9mo agoOne potential answer is that this tests more heavily for the ability to memorise, as opposed to understanding. My last exams were over ten years ago and I was always good at them because I have a good medium-term memory for names and numbers. But it's not clearly useful to test for this, as most names and numbers can just be looked up. When I was studying at university there was a rumour that one of the dons had scraped through their fourth-year exams despite barely attending lectures, because he had a photographic memory and just so happened to leaf through a book containing a required proof, the night before the exam. That gave him enough points despite not necessarily understanding what he was writing. Obviously very few students have that sort of memory, but it's not necessarily fair to give advantage to those like me who can simply remember things more easily.
- technothrasher 9mo ago> because he had a photographic memory and just so happened to leaf through a book containing a required proof It makes for good rumours and TV show plots, but this sort of "photographic memory" has never been shown to actually exist.
- n4r9 9mo agoHuh, TIL [0]. Thanks. There are people who can perform extraordinary memory feats, but they're very rare and/or self-trained. [0] https://skeptoid.com/episodes/542 https://skeptoid.com/episodes/542
- bookofjoe 9mo ago>His photographic memory manifested itself early — he would amuse his parents’ friends by instantly memorizing pages of phone books on command. https://medium.com/young-spurs/the-unsung-genius-of-john-von-neumann-5abec91a6359 https://medium.com/young-spurs/the-unsung-genius-of-john-von...
- lordnacho 9mo agoI dunno, I went to a high school reunion last year, and a dude seemed to know people's phone numbers from 30 years ago. If he could remember that sort of thing, I can believe there are people who can remember steps of a proof, which is a much less random thing that you can feel your way around, given a few queues from memory. Plus, realistically, how closely does an examiner read a proof? They have a stack of dozens of almost the same thing, I bet they get pretty tired of it and use a heuristic.
- ecshafer 9mo agoI had some take home exams in Physics that you could use internet, books, anything except other people (but that was honor code based). Those were some of the hardest exams I ever took in my life. Pages and pages of mathematical derivations. An LLM with how they can do a pretty good job at constructing mathematics, would actually have solved that issue pretty well.
- bArray 9mo ago> This approach still works, why do something else? One issue is that the time provided to mark each piece of work continues to decrease. Sometimes you are only getting 15 minutes for 20 pages, and management believe that you can mark back-to-back from 9-5 with a half hour lunch. The only thing keeping people sane is the students that fail to submit, or submit something obviously sub-par. So where possible, even for designing exams, you try to limit text altogether. Multiple choice, drawing lines, a basic diagram, a calculation, etc. Some students have terrible handwriting. I wouldn't be against the use of a dumb terminal in an exam room/hall. Maybe in the background it could be syncing the text and backing it up. > Unless you're specifically testing a student's ability to Google, they don't need access to it. I've been the person testing students, and I don't always remember everything. Sometimes it is good enough for the students to demonstrate that they understand the topic enough to know where to find the correct information based on a good intuition.
- Balgair 9mo agoI want to echo this. Your blue book is being graded by a stressed out and very underpaid grad student with many better things to do. They're looking for keywords to count up, that's it. The PI gave them the list of keywords, the rubric. Any flourishes, turns of phrase, novel takes, those don't matter to your grader at 11 pm after the 20th blue book that night. Yeah sure, that's not your school, but that is the reality of ~50% of US undergrads.
- monero-xmr 9mo agoVery effective multiple choice tests can be given, that require work to be done before selecting an answer, so it can be machine graded. Not ideal in every case but a very quality test can be made multiple choice for hard science subjects
- HardwareLust 9mo agoScantron and a #2 pencil.
- footy 9mo ago
- recursivedoubts 9mo agoI am returning to this model in my classes: pen in paper quizzes, no digital devices. I also do seven equally weighted quizzes to deescalate them individually. I have reduced project/programming weight from 60-80% of my grade to 50% because it is not possible to tell if the students actually did the work.
- bsoles 9mo agoI am also doing the same. 50% for project work and 50% for individual work, including paper and pen exams with no digital devices allowed. The days of take home exams and coding lab assignments are gone...
- buckle8017 9mo agoFor a project I'm but so sure banning LLMs is actually the right approach. Industry is full of people trying to use them to become more productive. Why wouldn't you let students use the same tools? Seems like you need to make the projects much harder.
- deleted 9mo ago[deleted]
- karmakurtisaani 9mo agoBut the problem is, students need to learn to do the easy things themselves before they can do the hard things with LLMs. If you ask them to build a web browser when they can't do a hello world on their own, it's going to be a disaster. LLMs are like dumb juniors that you command, but students are less skilled than dumb juniors when they start programming classes..
- nottorp 9mo ago> 10 years ago, we wrote exams by hand with whatever we understood (in our heads.) You did, but the best exam I had was open book bring anything. 25 and some change years ago even. I've also had another professor do the "you can bring one A4 sheet with whatever notes you want to make on it."
- deleted 9mo ago[deleted]
- mnw21cam 9mo agoI teach at MSc level. My students are scattered around the country and world. This makes hand-written exams tricky. Luckily, the nature of the questions they are asked to solve in the essay I give them following their coursework are that chatbots produce appalling bad submissions.
- eks391 9mo agoThis is great. Do you have advice on making questions that LLMs are bad at answering?
- red-iron-pine 9mo agofor most of the low hanging fruit it's as easy as copy-pasting the question into multiple LLMs and logging the output do it again from a different IP or two. there will be some pretty obvious patterns in responses. the smart kids will do minor prompt engineering "explain like you're Peter Griffin from Family Guy" or whatever, but even then there will be some core similarities. or follow the example of someone here and post a question with hidden characters that will show up differently when copy-pasted.
- n4r9 9mo agoThey don't work very well with large numbers. Try asking Claude to find the prime factors of 83521.
- mnw21cam 9mo agoIn my case, I set some course-work, where they have to log in to a Linux server in the university and process a load of data, get the results, and then write the essay about the process. Because the LLM hasn't been able to log in and see the data or indeed the results, it doesn't have a clue what it's meant to talk about.
- ohstep23 9mo agoI go to school right now, and most classes actually enforce paper and pencil tests despite how annoying it is to grade and code on.
- nunez 9mo agoWe had open notebook group exams back then too.
- pks016 9mo agoGrading the students. Usually for bigger classes, universities (at least mine) don't provide adequate support for grading the tests.
- Fire-Dragon-DoL 9mo agoWe had to write C code in paper. It was horrible. It's not like writing prose And there is no syntax highlighting, no compile errors
- Yoric 9mo agoI don't know if it's the reason, but some students do need a computer for medical reasons.
- pibaker 9mo agoI dont know what you majored. But when I was a CS major maybe 50% of my grade came from projects. We wrote a compiler from scratch, wrote something that resembled a SQL engine from scratch, and wrote sizeable portions of an operating system. In my sophomore year we spent at least 20 hours a week on various projects a week. We could use any resource we coulc find as long as we didn't submit anything we didn't write ourselves. This meant stackoverflow and online documentations. There is no way you can test a student's ability to implement a large, complex system with thousands of lines of code in a three hour exam. There is just no way. I am not against closed book paper exams, I just wish the people touting them as the solution can be more realistic about what they can and cannot do.