5 ms·

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was al

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. I had three hours to do the exam. Full stop. If there was a whiff of cheating, you were expelled. Do not pass go. Do not collect $200. Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. You were expected to learn the gd material. The university thanks you for your donation. I feel like i'm taking crazy pills when I read things about trying to "adapt" to AI. We already had the solution.

acbart 10mo ago
I've had colleagues argue (prior to LLMs) that oral exams are superior to paper exams, for diagnosing understanding. I don't know how to validate that statement, but if the assumption is true than there is merit to finding a way to scale them. Not saying this is it, but I wouldn't say that it's fair to just dismiss oral exams entirely.
- NewsaHackO 10mo ago
  Yes, I hate oral exams, but they are definitely better at getting a whole picture of a person's understanding of topics. A lot of specialty boards in medicine do this. To me, the two issues are that it requires an experienced, knowledgeable, and empathetic examiner, who is able to probe the examinee about areas they seem to be struggling in, and paradoxically, its strength is in the fact that it is subjective. The examiner may have set questions, but how the examinee answers the questions and the follow-up questions are what differentiate it from a written exam. If the examiner is just the equivalent of a customer service representative and is strictly following a tree of questions, it loses its value.
  - geraldwhen 10mo ago
    Interviews have the same issues. But if you do anything more than read off templated questions like a robot, you can be accused of discrimination. It is a sad world we live in.
- jimbokun 10mo ago
  Seems like the equivalent of claiming white board coding is the best way to evaluate software development candidates. With all the same advantages and disadvantages.
- abdullahkhalids 10mo ago
  Universities are not just places for students to learn. They are also places where young faculty, grad students and teaching assistants learn to become teachers and mentors. Those are very difficult skills to learn, and slogging through a lot of hands on teaching and mentoring is necessary to learn them. You can't really become a good classroom teacher either without grading your students yourself and figuring out what they learned and didn't.
- freehorse 10mo ago
  I think oral exam where you have a student explain and ask questions on a project they did is really good for judging understanding. The ones where you are supposed to memorise the answers to 15 questions where you will have to pick one at random, not as much imo.
cryptonector 10mo ago
TFA's case involved examinations about the student's submitted project work. It's not the same thing. Even for a more traditional examination with no such context attached one might still want to rely on AI for grading. (Yeah, I know, that comes across as "the students are not allowed to use AI for cheating, but the profs are!".) Also, IMO oral examinations are quite powerful for detecting who is prepared and who isn't. On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those.
- jimbokun 10mo ago
  You could argue that for fields like law, medicine and management extroversion and confidence are important qualities.
  - cryptonector 10mo ago
    Quite.
- NewsaHackO 10mo ago
  > On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those. This is true, but it is also why it is important to get an actual expert to proctor the exam. Having confidence is good and should be a plus, but if you are confident about a point that the examiner knows is completely incorrect, you may possibly put yourself in an inescapable hole, as it will be very difficult to ascertain that you actually know the other parts you were confident (much less unconfident) in.
Wowfunhappy 10mo ago
I basically agree with the thrust of what you're saying, but also: > I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. Do you, like, really think this is the best way to assess someone's ability? Can't we find a place between the two extremes? Personally, I'd go with a school-provided computer with a development environment and access to documentation. No LLMs, except maybe (but probably not) for very high-level courses.
- mrguyorama 10mo ago
  The safe middle space still does not involve a computer Lots of my tests involved writing pseudocode, or "Just write something that looks like C or Java". Don't miss the semicolon at the end of the line, but if you write "System.print()" rather than "System.out.printLn()" you might lose a single point. Maybe. If there were specific functions you need to call, it would have a man page or similar on the test itself, or it would be the actual topic under test. I hand wrote a bunch of SQL queries. Hand wrote code for my Systems Programming class that involved pointers. I'm not even good with pointers. I hand wrote Java for job interviews. It's pretty rare that you need to actually test someone can memorize syntax, that's like the entire point of modern development environments. But if you are completely unable to function without one, you might not know as much as you would hope. The first algorithms came before the first programming languages. Sure, it means you need to be able to run the code in your head and be able to mentally "debug" it, but that's a feature If you could not manage these things, you washed out in the CS101 class that nearly every STEM student took. The remaining students were not brilliant, but most of them could write code to solve problems. Then you got classes that could actually teach and test that problem solving itself. The one class where we built larger apps more akin to actual jobs, that could have been done entirely in the lab with locked down computers if need be, but the professor really didn't care if you wanted to fake the lab work, you still needed to pass the book learning for "Programming Patterns" which people really struggled with and you still needed to be able to give a "Demo" and presentation, and you still needed to demonstrate that you understood how to read some requests from a "Customer" and turn it into features and requirements and UX Nobody cares about people sabotaging their own education except in programming because no matter how much MBAs insist that all workers are replaceable, they cannot figure out a way to actually evaluate the competency of a programmer without knowing programming. If an engineer doesn't actually understand how to evaluate static stresses on a structure, they are going to have a hard time keeping a job. Meanwhile in the world of programming, hopping around once a year is "normal" somehow, so you can make a lot of money while literally not knowing fizzbuzz. I don't think the problem is actually education. Computer Science isn't actually about using a laptop.
perching_aix 10mo ago
> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thanks you for your donation" slogan doesn't sound too hot all of a sudden. It's like some malicious compliance take on both teaching and studying. Which shouldn't even be surprising, considering the circumstances of the professors e.g. where I studied, as well as the students'. Mind you, I was (for some classes) tested the same way. People still cheated, and grading stringency varied. People still also forgot everything shortly after wrapping up their finals on the given subjects and moved on. People also memorized questions and compiled a solutions book, and then handed them down to next year's class. Because this method does jack against that on its own. You still need to keep crafting novel questions, vary them more than just by swapping key values, etc.
- jmye 10mo ago
  > And why is this a flex exactly? Almost sounds like fraud. Do you think you're just purchasing a diploma? Or do you think you're purchasing the opportunity to gain an education and potential certification that you received said education? It's entirely possible that the University stunk at teaching 99% of it's students (about as equally possible that 99% of the students stunk at learning), but "fraud" is absolute nonsense. You're not entitled to a diploma if you fail to learn the material well enough to earn it.
  - perching_aix 10mo ago
    I don't think one applies to university expecting they're purchasing themselves a diploma, nor that they should be magically absolved of putting in effort to learn the material. What I do think is that the place they describe sounds an awful lot like people being set up for failure though, and so that begged the question as to why that might be. I should probably clarify that I wasn't particularly serious about my fraud suggestion however (was just a bit of a jab rather), as that doesn't seem to have made it through. If teaching was so simple that you could just tell people to go RTFM then recite it from memory, I don't know why people are bothering with pedagogy at all. It'd seem that there's more to teaching and learning than the bare minimum, and that both parties are culpable. Doesn't sound like you disagree on that either. > you're purchasing the opportunity to We can swap out fraud for gambling if you like :) Sounds like an even closer analogy now that you mention! Jokes aside though, isn't it a gamble? You gamble with yourself that you can [grow to] endure and succeed or drop out / something worse. The stake is the tuition, the prize is the diploma. Now of course, tuition is per semester (here at least, dunno elsewhere), so it's reasonable to argue that the financial investment is not quite in such jeopardy as I painted it. Not sure about the emotional investment though. Consider the Chinese Gaokao exam, especially in its infamous historical context between the 70s and 90s. The number of available seats was way lower than the number of applications [0]. The exams grueling. What do you reckon, was it the people's fault for not winning an essentially unspoken lottery? Who do you think received the blame? According to a cursory search, the individual and their families (wasn't there, cannot know) received the blame. And no, I don't think in such a tortured scheme it is the students' fault for not making the bar. If there are fewer seats than what there is demand for, then that's overbooking, and you the test authoring / conducting authority are biased to artificially induce test failures. It is no longer a fair assessment, nor a fair dynamic. Conversely, passing is no longer an honest signal of qualification. Or rather, not passing is no longer an honest signal of unqualification. And this doesn't have to come from a single test, it can be implemented structurally too, so that you shed people along the way. Which is what I'm actually alluding to. [0] ~4.8%, so ~95% of people failed it by design: https://en.wikipedia.org/wiki/Class_of_1977%E2%80%931978_%28China%29 https://en.wikipedia.org/wiki/Class_of_1977%E2%80%931978_%28...
jimbokun 10mo ago
Admitting 1000 students to get 10 graduates means there are morons in admissions doing zero vetting to make sure the students are qualified.
- vasco 10mo ago
  Or that there's morons teaching.
- pamcake 10mo ago
  There's either a 0 missing there or something pretty weird at that uni. I think the rest of the comment is very valid if we ignore this point. My experience is the same except I think ~50% or so graduated[0]. [0]: Disclaimer that my programme was pretty competitive to get into, which is an earlier filter. Statistics looked worse for programmes at similar level with less people applying.
- baq 10mo ago
  Absolutely not morons. If the goal is to maximize collecting tuition and still have reputation of not being a diploma shop this is the obvious solution. The 20% which survives the first year is worth keeping around to hire them later in the companies which the teaching staff own or collect referral bonuses if working for a multinational.
  - jimbokun 10mo ago
    True, outright fraud is another adequate explanation.
LorenzoGood 10mo ago
I currently go to school for engineering, and it is the same way.
deleted 10mo ago
[deleted]
rfrey 10mo ago
I simply don't believe your university program had a 99% failure rate. Such a university should be shut down and sold for parts.
- jasonfarnon 10mo ago
  any private university, yes. I have seen state-supported universities in certain countries with very high failure rates for certain programs (I'm assuming 99% was an exaggeration for something more like "the vast majority failed").
  - baq 10mo ago
    In my state uni 75% was normal a couple decades ago, 50% after first year. 99% is extreme, but I can imagine that being true with uni leadership on board.
- freehorse 10mo ago
  The example above may have been a bit misleading imo. In some countries the filtering process is put inside the program itself rather than in state wide exams, entrance exams or amount of tuition fees. There is always a filtering process somewhere. Not sure where OP was though.
makeitdouble 10mo ago
What's the crazy to me is you took that as the gold standard for education evaluation. For comparison we had lengthy sessions in a jailed terminal, week after week, writing C programs covering specific algorithms, compiling and debugging them within these sessions and assistants would follow our progress and check we're getting it. Those not finishing in time get additional sessions. Last exam was extremely simple and had very little weight in the overall evaluation. That might not scale as much, but that's definitely what I'd long for, not the Chuck Norris style cram school exam you are drawing us.
TrackerFF 10mo ago
So did I, but a big difference today is the number of students, and how many of them are doing non-traditional programs. Lots and lots of online-only programs, offered through serious universities. The old ways do not scale well once you pass a certain number of students.
BalinKing 10mo ago
I'm fairly skeptical of tests that are closed-book. IMO the only reasons to do so are if 1) the goal is to test rote memorization (which is admittedly sometimes valuable, especially depending on the field) or, perhaps more commonly, 2) the test isn't actually hard enough, and the questions don't require as much "synthesis" as they should to test real understanding.
bossyTeacher 10mo ago
> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. You have a very weird idea of education if a teaching method that results in a 99% failure rate is seen as good by yourself. Do you imagine a professional turning out work that was 99% suboptimal?

acbart 10mo ago

I've had colleagues argue (prior to LLMs) that oral exams are superior to paper exams, for diagnosing understanding. I don't know how to validate that statement, but if the assumption is true than there is merit to finding a way to scale them. Not saying this is it, but I wouldn't say that it's fair to just dismiss oral exams entirely.

NewsaHackO 10mo ago
Yes, I hate oral exams, but they are definitely better at getting a whole picture of a person's understanding of topics. A lot of specialty boards in medicine do this. To me, the two issues are that it requires an experienced, knowledgeable, and empathetic examiner, who is able to probe the examinee about areas they seem to be struggling in, and paradoxically, its strength is in the fact that it is subjective. The examiner may have set questions, but how the examinee answers the questions and the follow-up questions are what differentiate it from a written exam. If the examiner is just the equivalent of a customer service representative and is strictly following a tree of questions, it loses its value.
- geraldwhen 10mo ago
  Interviews have the same issues. But if you do anything more than read off templated questions like a robot, you can be accused of discrimination. It is a sad world we live in.
jimbokun 10mo ago
Seems like the equivalent of claiming white board coding is the best way to evaluate software development candidates. With all the same advantages and disadvantages.
abdullahkhalids 10mo ago
Universities are not just places for students to learn. They are also places where young faculty, grad students and teaching assistants learn to become teachers and mentors. Those are very difficult skills to learn, and slogging through a lot of hands on teaching and mentoring is necessary to learn them. You can't really become a good classroom teacher either without grading your students yourself and figuring out what they learned and didn't.
freehorse 10mo ago
I think oral exam where you have a student explain and ask questions on a project they did is really good for judging understanding. The ones where you are supposed to memorise the answers to 15 questions where you will have to pick one at random, not as much imo.

cryptonector 10mo ago

TFA's case involved examinations about the student's submitted project work. It's not the same thing. Even for a more traditional examination with no such context attached one might still want to rely on AI for grading. (Yeah, I know, that comes across as "the students are not allowed to use AI for cheating, but the profs are!".) Also, IMO oral examinations are quite powerful for detecting who is prepared and who isn't. On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those.

jimbokun 10mo ago
You could argue that for fields like law, medicine and management extroversion and confidence are important qualities.
- cryptonector 10mo ago
  Quite.
NewsaHackO 10mo ago
> On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those. This is true, but it is also why it is important to get an actual expert to proctor the exam. Having confidence is good and should be a plus, but if you are confident about a point that the examiner knows is completely incorrect, you may possibly put yourself in an inescapable hole, as it will be very difficult to ascertain that you actually know the other parts you were confident (much less unconfident) in.

Wowfunhappy 10mo ago

I basically agree with the thrust of what you're saying, but also: > I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. Do you, like, really think this is the best way to assess someone's ability? Can't we find a place between the two extremes? Personally, I'd go with a school-provided computer with a development environment and access to documentation. No LLMs, except maybe (but probably not) for very high-level courses.

mrguyorama 10mo ago
The safe middle space still does not involve a computer Lots of my tests involved writing pseudocode, or "Just write something that looks like C or Java". Don't miss the semicolon at the end of the line, but if you write "System.print()" rather than "System.out.printLn()" you might lose a single point. Maybe. If there were specific functions you need to call, it would have a man page or similar on the test itself, or it would be the actual topic under test. I hand wrote a bunch of SQL queries. Hand wrote code for my Systems Programming class that involved pointers. I'm not even good with pointers. I hand wrote Java for job interviews. It's pretty rare that you need to actually test someone can memorize syntax, that's like the entire point of modern development environments. But if you are completely unable to function without one, you might not know as much as you would hope. The first algorithms came before the first programming languages. Sure, it means you need to be able to run the code in your head and be able to mentally "debug" it, but that's a feature If you could not manage these things, you washed out in the CS101 class that nearly every STEM student took. The remaining students were not brilliant, but most of them could write code to solve problems. Then you got classes that could actually teach and test that problem solving itself. The one class where we built larger apps more akin to actual jobs, that could have been done entirely in the lab with locked down computers if need be, but the professor really didn't care if you wanted to fake the lab work, you still needed to pass the book learning for "Programming Patterns" which people really struggled with and you still needed to be able to give a "Demo" and presentation, and you still needed to demonstrate that you understood how to read some requests from a "Customer" and turn it into features and requirements and UX Nobody cares about people sabotaging their own education except in programming because no matter how much MBAs insist that all workers are replaceable, they cannot figure out a way to actually evaluate the competency of a programmer without knowing programming. If an engineer doesn't actually understand how to evaluate static stresses on a structure, they are going to have a hard time keeping a job. Meanwhile in the world of programming, hopping around once a year is "normal" somehow, so you can make a lot of money while literally not knowing fizzbuzz. I don't think the problem is actually education. Computer Science isn't actually about using a laptop.

perching_aix 10mo ago

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thanks you for your donation" slogan doesn't sound too hot all of a sudden. It's like some malicious compliance take on both teaching and studying. Which shouldn't even be surprising, considering the circumstances of the professors e.g. where I studied, as well as the students'. Mind you, I was (for some classes) tested the same way. People still cheated, and grading stringency varied. People still also forgot everything shortly after wrapping up their finals on the given subjects and moved on. People also memorized questions and compiled a solutions book, and then handed them down to next year's class. Because this method does jack against that on its own. You still need to keep crafting novel questions, vary them more than just by swapping key values, etc.

jmye 10mo ago
> And why is this a flex exactly? Almost sounds like fraud. Do you think you're just purchasing a diploma? Or do you think you're purchasing the opportunity to gain an education and potential certification that you received said education? It's entirely possible that the University stunk at teaching 99% of it's students (about as equally possible that 99% of the students stunk at learning), but "fraud" is absolute nonsense. You're not entitled to a diploma if you fail to learn the material well enough to earn it.
- perching_aix 10mo ago
  I don't think one applies to university expecting they're purchasing themselves a diploma, nor that they should be magically absolved of putting in effort to learn the material. What I do think is that the place they describe sounds an awful lot like people being set up for failure though, and so that begged the question as to why that might be. I should probably clarify that I wasn't particularly serious about my fraud suggestion however (was just a bit of a jab rather), as that doesn't seem to have made it through. If teaching was so simple that you could just tell people to go RTFM then recite it from memory, I don't know why people are bothering with pedagogy at all. It'd seem that there's more to teaching and learning than the bare minimum, and that both parties are culpable. Doesn't sound like you disagree on that either. > you're purchasing the opportunity to We can swap out fraud for gambling if you like :) Sounds like an even closer analogy now that you mention! Jokes aside though, isn't it a gamble? You gamble with yourself that you can [grow to] endure and succeed or drop out / something worse. The stake is the tuition, the prize is the diploma. Now of course, tuition is per semester (here at least, dunno elsewhere), so it's reasonable to argue that the financial investment is not quite in such jeopardy as I painted it. Not sure about the emotional investment though. Consider the Chinese Gaokao exam, especially in its infamous historical context between the 70s and 90s. The number of available seats was way lower than the number of applications [0]. The exams grueling. What do you reckon, was it the people's fault for not winning an essentially unspoken lottery? Who do you think received the blame? According to a cursory search, the individual and their families (wasn't there, cannot know) received the blame. And no, I don't think in such a tortured scheme it is the students' fault for not making the bar. If there are fewer seats than what there is demand for, then that's overbooking, and you the test authoring / conducting authority are biased to artificially induce test failures. It is no longer a fair assessment, nor a fair dynamic. Conversely, passing is no longer an honest signal of qualification. Or rather, not passing is no longer an honest signal of unqualification. And this doesn't have to come from a single test, it can be implemented structurally too, so that you shed people along the way. Which is what I'm actually alluding to. [0] ~4.8%, so ~95% of people failed it by design: https://en.wikipedia.org/wiki/Class_of_1977%E2%80%931978_%28China%29 https://en.wikipedia.org/wiki/Class_of_1977%E2%80%931978_%28...

jimbokun 10mo ago

Admitting 1000 students to get 10 graduates means there are morons in admissions doing zero vetting to make sure the students are qualified.

vasco 10mo ago
Or that there's morons teaching.
pamcake 10mo ago
There's either a 0 missing there or something pretty weird at that uni. I think the rest of the comment is very valid if we ignore this point. My experience is the same except I think ~50% or so graduated[0]. [0]: Disclaimer that my programme was pretty competitive to get into, which is an earlier filter. Statistics looked worse for programmes at similar level with less people applying.
baq 10mo ago
Absolutely not morons. If the goal is to maximize collecting tuition and still have reputation of not being a diploma shop this is the obvious solution. The 20% which survives the first year is worth keeping around to hire them later in the companies which the teaching staff own or collect referral bonuses if working for a multinational.
- jimbokun 10mo ago
  True, outright fraud is another adequate explanation.

LorenzoGood 10mo ago

I currently go to school for engineering, and it is the same way.

deleted 10mo ago

[deleted]

rfrey 10mo ago

I simply don't believe your university program had a 99% failure rate. Such a university should be shut down and sold for parts.

jasonfarnon 10mo ago
any private university, yes. I have seen state-supported universities in certain countries with very high failure rates for certain programs (I'm assuming 99% was an exaggeration for something more like "the vast majority failed").
- baq 10mo ago
  In my state uni 75% was normal a couple decades ago, 50% after first year. 99% is extreme, but I can imagine that being true with uni leadership on board.
freehorse 10mo ago
The example above may have been a bit misleading imo. In some countries the filtering process is put inside the program itself rather than in state wide exams, entrance exams or amount of tuition fees. There is always a filtering process somewhere. Not sure where OP was though.

makeitdouble 10mo ago

What's the crazy to me is you took that as the gold standard for education evaluation. For comparison we had lengthy sessions in a jailed terminal, week after week, writing C programs covering specific algorithms, compiling and debugging them within these sessions and assistants would follow our progress and check we're getting it. Those not finishing in time get additional sessions. Last exam was extremely simple and had very little weight in the overall evaluation. That might not scale as much, but that's definitely what I'd long for, not the Chuck Norris style cram school exam you are drawing us.

TrackerFF 10mo ago

So did I, but a big difference today is the number of students, and how many of them are doing non-traditional programs. Lots and lots of online-only programs, offered through serious universities. The old ways do not scale well once you pass a certain number of students.

BalinKing 10mo ago

I'm fairly skeptical of tests that are closed-book. IMO the only reasons to do so are if 1) the goal is to test rote memorization (which is admittedly sometimes valuable, especially depending on the field) or, perhaps more commonly, 2) the test isn't actually hard enough, and the questions don't require as much "synthesis" as they should to test real understanding.

bossyTeacher 10mo ago

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. You have a very weird idea of education if a teaching method that results in a 99% failure rate is seen as good by yourself. Do you imagine a professional turning out work that was 99% suboptimal?