4 ms·

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

Kwpolska 10mo ago
Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening.
- NewsaHackO 10mo ago
  I agree with you and the other posters actually, but I think the efficiency compared with typed work is the reason it’s having such a slow adoption. Another thing to remember is that there is always a mild Jevons paradox at play; while it's true that it was possible in previous centuries, teacher expectations have also increased which strains the amount of time they would have grading handwritten work.
- aleph_minus_one 10mo ago
  > Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening. At least in Germany, if there are only 36 students in a class, usually oral exams are used because in this case oral exams are typically more efficient. For written exams, more like 200-600 students in a class is the common situation.
- abdullahkhalids 10mo ago
  Not the comprehensive physics exams I assigned as a prof. A well set exam takes at least 20-30 min to grade. That's 8-12 hours of work, and in practice, took several sittings over several days. If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. I wanted to do oral exams, but they are much more exhausting for the prof. Nominally, each student is with you for 30 min, but (1) you need to think of slightly different question for each student (2) you need to squeeze all the exams in only a couple of days to avoid giving later students too much extra time to prepare.
  - thaumasiotes 10mo ago
    > If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. That's entirely false; this is why we have multiple-choice tests.
    - fn-mote 10mo ago
      On the surface, true. Multiple choice tests are a counter example. Thinking deeper, though, multiple choice tests require SIGNIFICANTLY more preparation. I would go so far as to say almost all individual professors are completely unqualified to write valid multiple choice tests. The time investment in multiple choice comes at the start - 12 hours writing it instead of 12 hours grading it - but it’s still a lot of time and frankly there is only very general feedback on student misunderstandings.
      - cedilla 10mo ago
        Is this a new thing or do you think that most professors were always unable to do their job? Why do you think you are an exception? I don't believe that your argument is more than an ad-hoc value judgment lacking justification. And it's obvious that if you think so little of your colleagues, that they would also struggle to implement AI tests.
    - ninalanyon 10mo ago
      When I studied for my degree there were no multiple choice tests. In the final every question required a narrative answer justifying the conclusion.
    - abdullahkhalids 10mo ago
      I have never, on my own free will, assigned multiple-choice questions in a serious course. And never will. - They have a base marks of 20-25% (by random guessing) instead of 0. - You never see the working. So you can't check if students are thinking correctly. Slightly wrong thinking can get you right answers. - They don't even remotely reflect real life at all. Written worked through problems on the other hand - I still do those in my professional life as a scientist all the time. It's just that I am setting the questions for myself. - The format doesn't allow for extended thought questions. In my undergrad, I had some excellent profs who would set long work through exam question in such a way that you learned something even in the exams. Simply a joy taking those exams that gave a comprehensive walk through of the course. As a prof, I have always tried to replicate that.
pgalvin 10mo ago
University exams being marked by hand, by someone experienced enough to work outside a rigid marking scheme, has been the standard for hundreds of years and has proven scalable enough. If there are so many students that academics can’t keep up, there are likely too many students to maintain a high standard of education anyway.
- unbrice 10mo ago
  > there are likely too many students to maintain a high standard of education anyway. Right on point. I find particularly striking how little is said about whether the best students achieve the best grades. Authors are even candid that different LLMs asses differently, but seem to conclude that LLMs converging after a few rounds of cross reviews indicate they are plausible so who cares. The apparences are safe.
- aaplok 10mo ago
  A limitation of written exams is in distance education, which simply was hardly a thing for the hundreds of years exams were used. Just like WFH is a new practice employers have to learn to deal with, study from home (SFH) is a phenomenon that is going to affect education. The objections to SFH exist and are strikingly similar to objections to WFH, but the economics are different. Some universities already see value in offering that option, and they (of course) leave it to the faculty to deal with the consequences.
  - sarchertech 10mo ago
    Distance education is a tiny percentage of higher education though. Online classes at a local university are more common, but you can still bring the students in for proctored exams. Even for distance education though, proctored testing centers have been around longer than the internet.
    - aaplok 10mo ago
      > Distance education is a tiny percentage of higher education though. It is about a third of the students I teach, which amounts to several hundreds per term. It may be niche, but it is not insignificant, and definitely a problem for some of us. > Even for distance education though, proctored testing centers have been around longer than the internet. I don't know how much experience you have with those. Mine is extensive enough that I have a personal opinion that they are not scalable (which is the focus of the comment I was replying to). If you have hundreds of students disseminated around the world, organising a proctored exam is a logistical challenge. It is not a problem at many universities yet, because they haven't jumped on the bandwagon. However domestic markets are becoming saturated, visas are harder to get for international students, and there is a demand for online education. I would be surprised that it doesn't develop more in the near future.
Aurornis 10mo ago
When college degrees cost as much as they do, it's reasonable to pay people to do the transcription and/or grading. Work study and TA jobs were abundant when I was in college. It wasn't a problem in the past and shouldn't be a problem now.
vkou 10mo ago
I assure you, oral exams are completely scalable. But it does require most of a university's budget to go towards labs and faculty, and not administration and sports arenas and social services and vanity projects and three-star dorms.
- musicale 10mo ago
  One way of scaling out interactive/oral assessment (and personalized instruction in general) is to hire a group of course assistants/tutors from the previous cohort.
  - vkou 10mo ago
    So, TAs. The other half of the mission-critical staff that keeps a university running.
    - musicale 10mo ago
      I think it works differently at different schools and in different countries, but hourly (often undergraduate work-study) course assistants in the US can be very affordable since they typically still pay tuition and are paid at a lower rate than fully funded (usually graduate student) TAs.
    - ninalanyon 10mo ago
      Does any country other than the US use TAs? They certainly weren't a thing when I studied in the UK in the 1970s.
      - vkou 10mo ago
        Canada.
  - ninalanyon 10mo ago
    As a student I really would not want to be taught by someone who was simply a couple of years ahead of me. I want my tutor to be a lot more experienced in both the subject and in tutoring.
deleted 10mo ago
[deleted]
recursivecaveat 10mo ago
It is literally perfect linear scaling. For every student you must expend constant minutes of TA time grading the exam. Why is it unconscionable that the university should have an expense scale at the same rate it receives tuition revenue? $90,000 of tuition pays for a lot of grading hours. I feel that scalability is a cultural meme that has lost the plot.
- andrepd 10mo ago
  There are phrases that hn loves and "scalable" is one of them. Here, it is particularly inappropriate. Some people dream that technology (preferably duly packaged by for-profit SV concerns) can and will eventually solve each and every problem in the world; unfortunately what education boils down to is good, old-fashioned teaching. By teachers. Nothing whatsoever replaces a good, talented, and attentive teacher, all the technologies in the world, from planetariums to manim, can only augment a good teacher. Grading students with LLMs is already tone-deaf, but presenting this trainwreck of a result and framing it as any sort of success... Let's just say it reeks of 2025.
  - chii 10mo ago
    it's not so black and white. If a student is willing and desire to learn, an LLM is better than a bad teacher. If a student doesn't want to learn, and is instead being forced to (either as a minor, or via certification required to obtain work & money), then they have every incentive to cheat. An LLM is insufficient in this case - a teacher is both the enforcer and the tutor in this case. There's also nothing wrong with a teacher using an LLM to help with the grading imho.
gwern 10mo ago
To clarify the point here for people who didn't read OP: the oral exams here are customized and tailored to the student's individual unique project, that's the point and why they are not written: > In our new "AI/ML Product Management" class, the "pre-case" submissions (short assignments meant to prepare students for class discussion) were looking suspiciously good. Not "strong student" good. More like "this reads like a McKinsey memo that went through three rounds of editing," good...Many students who had submitted thoughtful, well-structured work could not explain basic choices in their own submission after two follow-up questions. Some could not participate at all...Oral exams are a natural response. They force real-time reasoning, application to novel prompts, and defense of actual decisions. The problem? Oral exams are a logistical nightmare. You cannot run them for a large class without turning the final exam period into a month-long hostage situation. Written exams do not do the same thing. You can't say 'just do a written exam'. So sure, the students may prefer them, but so what? That's apples and oranges.

Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening.

NewsaHackO 10mo ago
I agree with you and the other posters actually, but I think the efficiency compared with typed work is the reason it’s having such a slow adoption. Another thing to remember is that there is always a mild Jevons paradox at play; while it's true that it was possible in previous centuries, teacher expectations have also increased which strains the amount of time they would have grading handwritten work.
aleph_minus_one 10mo ago
> Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening. At least in Germany, if there are only 36 students in a class, usually oral exams are used because in this case oral exams are typically more efficient. For written exams, more like 200-600 students in a class is the common situation.
abdullahkhalids 10mo ago
Not the comprehensive physics exams I assigned as a prof. A well set exam takes at least 20-30 min to grade. That's 8-12 hours of work, and in practice, took several sittings over several days. If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. I wanted to do oral exams, but they are much more exhausting for the prof. Nominally, each student is with you for 30 min, but (1) you need to think of slightly different question for each student (2) you need to squeeze all the exams in only a couple of days to avoid giving later students too much extra time to prepare.
- thaumasiotes 10mo ago
  > If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. That's entirely false; this is why we have multiple-choice tests.
  - fn-mote 10mo ago
    On the surface, true. Multiple choice tests are a counter example. Thinking deeper, though, multiple choice tests require SIGNIFICANTLY more preparation. I would go so far as to say almost all individual professors are completely unqualified to write valid multiple choice tests. The time investment in multiple choice comes at the start - 12 hours writing it instead of 12 hours grading it - but it’s still a lot of time and frankly there is only very general feedback on student misunderstandings.
    - cedilla 10mo ago
      Is this a new thing or do you think that most professors were always unable to do their job? Why do you think you are an exception? I don't believe that your argument is more than an ad-hoc value judgment lacking justification. And it's obvious that if you think so little of your colleagues, that they would also struggle to implement AI tests.
  - ninalanyon 10mo ago
    When I studied for my degree there were no multiple choice tests. In the final every question required a narrative answer justifying the conclusion.
  - abdullahkhalids 10mo ago
    I have never, on my own free will, assigned multiple-choice questions in a serious course. And never will. - They have a base marks of 20-25% (by random guessing) instead of 0. - You never see the working. So you can't check if students are thinking correctly. Slightly wrong thinking can get you right answers. - They don't even remotely reflect real life at all. Written worked through problems on the other hand - I still do those in my professional life as a scientist all the time. It's just that I am setting the questions for myself. - The format doesn't allow for extended thought questions. In my undergrad, I had some excellent profs who would set long work through exam question in such a way that you learned something even in the exams. Simply a joy taking those exams that gave a comprehensive walk through of the course. As a prof, I have always tried to replicate that.

pgalvin 10mo ago

University exams being marked by hand, by someone experienced enough to work outside a rigid marking scheme, has been the standard for hundreds of years and has proven scalable enough. If there are so many students that academics can’t keep up, there are likely too many students to maintain a high standard of education anyway.

unbrice 10mo ago
> there are likely too many students to maintain a high standard of education anyway. Right on point. I find particularly striking how little is said about whether the best students achieve the best grades. Authors are even candid that different LLMs asses differently, but seem to conclude that LLMs converging after a few rounds of cross reviews indicate they are plausible so who cares. The apparences are safe.
aaplok 10mo ago
A limitation of written exams is in distance education, which simply was hardly a thing for the hundreds of years exams were used. Just like WFH is a new practice employers have to learn to deal with, study from home (SFH) is a phenomenon that is going to affect education. The objections to SFH exist and are strikingly similar to objections to WFH, but the economics are different. Some universities already see value in offering that option, and they (of course) leave it to the faculty to deal with the consequences.
- sarchertech 10mo ago
  Distance education is a tiny percentage of higher education though. Online classes at a local university are more common, but you can still bring the students in for proctored exams. Even for distance education though, proctored testing centers have been around longer than the internet.
  - aaplok 10mo ago
    > Distance education is a tiny percentage of higher education though. It is about a third of the students I teach, which amounts to several hundreds per term. It may be niche, but it is not insignificant, and definitely a problem for some of us. > Even for distance education though, proctored testing centers have been around longer than the internet. I don't know how much experience you have with those. Mine is extensive enough that I have a personal opinion that they are not scalable (which is the focus of the comment I was replying to). If you have hundreds of students disseminated around the world, organising a proctored exam is a logistical challenge. It is not a problem at many universities yet, because they haven't jumped on the bandwagon. However domestic markets are becoming saturated, visas are harder to get for international students, and there is a demand for online education. I would be surprised that it doesn't develop more in the near future.

Aurornis 10mo ago

When college degrees cost as much as they do, it's reasonable to pay people to do the transcription and/or grading. Work study and TA jobs were abundant when I was in college. It wasn't a problem in the past and shouldn't be a problem now.

vkou 10mo ago

I assure you, oral exams are completely scalable. But it does require most of a university's budget to go towards labs and faculty, and not administration and sports arenas and social services and vanity projects and three-star dorms.

musicale 10mo ago
One way of scaling out interactive/oral assessment (and personalized instruction in general) is to hire a group of course assistants/tutors from the previous cohort.
- vkou 10mo ago
  So, TAs. The other half of the mission-critical staff that keeps a university running.
  - musicale 10mo ago
    I think it works differently at different schools and in different countries, but hourly (often undergraduate work-study) course assistants in the US can be very affordable since they typically still pay tuition and are paid at a lower rate than fully funded (usually graduate student) TAs.
  - ninalanyon 10mo ago
    Does any country other than the US use TAs? They certainly weren't a thing when I studied in the UK in the 1970s.
    - vkou 10mo ago
      Canada.
- ninalanyon 10mo ago
  As a student I really would not want to be taught by someone who was simply a couple of years ahead of me. I want my tutor to be a lot more experienced in both the subject and in tutoring.

deleted 10mo ago

[deleted]

recursivecaveat 10mo ago

It is literally perfect linear scaling. For every student you must expend constant minutes of TA time grading the exam. Why is it unconscionable that the university should have an expense scale at the same rate it receives tuition revenue? $90,000 of tuition pays for a lot of grading hours. I feel that scalability is a cultural meme that has lost the plot.

andrepd 10mo ago
There are phrases that hn loves and "scalable" is one of them. Here, it is particularly inappropriate. Some people dream that technology (preferably duly packaged by for-profit SV concerns) can and will eventually solve each and every problem in the world; unfortunately what education boils down to is good, old-fashioned teaching. By teachers. Nothing whatsoever replaces a good, talented, and attentive teacher, all the technologies in the world, from planetariums to manim, can only augment a good teacher. Grading students with LLMs is already tone-deaf, but presenting this trainwreck of a result and framing it as any sort of success... Let's just say it reeks of 2025.
- chii 10mo ago
  it's not so black and white. If a student is willing and desire to learn, an LLM is better than a bad teacher. If a student doesn't want to learn, and is instead being forced to (either as a minor, or via certification required to obtain work & money), then they have every incentive to cheat. An LLM is insufficient in this case - a teacher is both the enforcer and the tutor in this case. There's also nothing wrong with a teacher using an LLM to help with the grading imho.

gwern 10mo ago

To clarify the point here for people who didn't read OP: the oral exams here are customized and tailored to the student's individual unique project, that's the point and why they are not written: > In our new "AI/ML Product Management" class, the "pre-case" submissions (short assignments meant to prepare students for class discussion) were looking suspiciously good. Not "strong student" good. More like "this reads like a McKinsey memo that went through three rounds of editing," good...Many students who had submitted thoughtful, well-structured work could not explain basic choices in their own submission after two follow-up questions. Some could not participate at all...Oral exams are a natural response. They force real-time reasoning, application to novel prompts, and defense of actual decisions. The problem? Oral exams are a logistical nightmare. You cannot run them for a large class without turning the final exam period into a month-long hostage situation. Written exams do not do the same thing. You can't say 'just do a written exam'. So sure, the students may prefer them, but so what? That's apples and oranges.