6 ms·

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with

by gklitz 2y ago

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Instead you had to waste more time on a written assignment so they could offload grading to a minimum wage assistent.

Teever 2y ago
There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think. The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience. The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators. You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay. I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.
- SketchySeaBeast 2y ago
  All excellent point, but I'd like to add that it also forces you to do your own research the correct way, by surveying the current state of academic research and then finding and incorporating scholarly sources into your own arguments. Every academic essay I ever wrote after high school started with a trip to the library and JSTOR. I had to guide my own education instead of learning from the teacher and then repeating what had been taught.
- eszed 2y ago
  You're completely correct. Learning how to write taught me how to think, and researching and writing essays taught me what I believe about nearly everything on which I have strong opinions. However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do. I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.
ordu 2y ago
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.
- lukan 2y ago
  I hope that was sarcasm?
  - ordu 2y ago
    Probably. I'm not sure myself. It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny. OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case. AI could solve this problem, but I'm not sure if AIs will be up to a task of leading the discussion. Though maybe if you try to assess students on their ability to catch AI on a hallucinated bullshit...
    - lukan 2y ago
      "OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs" Why not have the testing done externally, by really neutral persons? But AIs and especially LLMs are way too unreliable for the foreseeable future.
    - michaelmrose 2y ago
      Actually deliberately introducing confidently delivered and reasonable sounding bullshit sounds like a fantastic way to suss out who knows their topic.
ninalanyon 2y ago
When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the report of the final year project. That report had to be defended orally. Previous years exam papers were available for study as well but the variety of questions that could be asked was so vast that it was rare that any questions were repeated in the finals. It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.
- bigfudge 2y ago
  I agree. There are small question about bias (gender, race) etc in these oral systems, but I think they are resolvable and much better than written essays (which are now written by AI).
  - jack_pp 2y ago
    the teacher knows you either way so the bias would be there for the written exam as well
    - bluGill 2y ago
      In a written exam they can cover the names - give you a random number as you enter the room and you write that on the paper, and but your name and number on a different paper. You also need to type everything out on a computer with spell check. (and even then if you write bucket or pail will identify you but it is unlikely any professor knows you well enough to tell those) When you audition for a symphony you perform behind a curtain and are required to wear soft slippers (so they can't tell if you are a wearing high heals - female). We can probably use voice changers so the examiner cannot tell who you are by your voice, but those tend to be fatiguing.
      - 2y ago
mountainb 2y ago
You can still do written essay evaluations. You could just require proctored exams whether or not you use software like Examsoft. If it's a topic that benefits from writing from a store of material, you can permit students to bring either unlimited supplemental printed material or a limited body of printed material into the exam room. For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop. There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.
- cmgbhm 2y ago
  That’s 8 days of TA time. You’re going to get high variance and most likely having to boil it down to the equivalent of a multiple choice oral exam. Hiring a n TA to delegate grading that’s hard to verify seems like will cost more than you think.
  - mountainb 2y ago
    So get more TAs. They cost less than Class B CDL drivers and will drag themselves over broken glass to take these jobs. And 65 hours of work per semester for oral examinations seems entirely reasonable. A week and a half for an FTE spent observing, with another half week to full week for grading seems completely reasonable for capstone semester work.
jvanderbot 2y ago
Well, that's not necessarily true. I was perhaps the most importunate student ever, and lingered around my professor's offices whenever they were open. I had endless questions, off topic and on. I was curious sure, but I was also annoying and pushy and wouldn't take no for an answer. In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me. So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.
- selimthegrim 2y ago
  TIL a new word.
- gklitz 2y ago
  The fact that one person can easily take a cup of water from a lake does not mean the lake supports for every person to take a cup. In fact, if everyone had tried to take that cup, then there wouldn’t even be a lake for the one person to take a cup from.
CharlieDigital 2y ago
> Written assay evaluation is not and has never been an effective evaluation. I kind of disagree. I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth. I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields. The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.
tgv 2y ago
> Instead you had to waste more time I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic. Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.
WillAdams 2y ago
My best college professor (who was also an Episcopalian Priest) found the time to review one paper with each student once per semester. That strikes me as a workable bottom line.
zamadatix 2y ago
> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.
l1silver 2y ago
> Written assay evaluation is not and has never been an effective evaluation. Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.

by gklitz 2y ago

Teever 2y ago

There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think. The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past) is a critical way of developing thinking skills and refining your ability to communicate to an audience. The value of that shouldn't be overlooked just like the value of basic numeracy shouldn't be overlooked because we all carry calculators. You're right that it would be better if post secondary institutions would test people's ability to think in more ways than just what they can regurgitate onto a piece of paper, if only because that can be easily cheated but that doesn't mean that there isn't personal benefit in the experience of writing an essay. I may not be the best writer but I am a better writer because I wrote essays in university, and I may not be great at math but I can reason and estimate about a variety of things because I have taken many math courses. These things have ultimately made me a better thinker and I am grateful to have had that imparted to me.

SketchySeaBeast 2y ago
All excellent point, but I'd like to add that it also forces you to do your own research the correct way, by surveying the current state of academic research and then finding and incorporating scholarly sources into your own arguments. Every academic essay I ever wrote after high school started with a trip to the library and JSTOR. I had to guide my own education instead of learning from the teacher and then repeating what had been taught.
eszed 2y ago
You're completely correct. Learning how to write taught me how to think, and researching and writing essays taught me what I believe about nearly everything on which I have strong opinions. However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do. I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.

ordu 2y ago

> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. So the obvious solution is to make students to talk with an AI, which would grade their performance. Or, maybe the grading itself could be done by a minimum wage assistant, while AI would lead the discussion with a student.

lukan 2y ago
I hope that was sarcasm?
- ordu 2y ago
  Probably. I'm not sure myself. It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny. OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case. AI could solve this problem, but I'm not sure if AIs will be up to a task of leading the discussion. Though maybe if you try to assess students on their ability to catch AI on a hallucinated bullshit...
  - lukan 2y ago
    "OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs" Why not have the testing done externally, by really neutral persons? But AIs and especially LLMs are way too unreliable for the foreseeable future.
  - michaelmrose 2y ago
    Actually deliberately introducing confidently delivered and reasonable sounding bullshit sounds like a fantastic way to suss out who knows their topic.

ninalanyon 2y ago

When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the report of the final year project. That report had to be defended orally. Previous years exam papers were available for study as well but the variety of questions that could be asked was so vast that it was rare that any questions were repeated in the finals. It seems to me that this is pretty much immune to plagiarism as well as being much better for the student.

bigfudge 2y ago
I agree. There are small question about bias (gender, race) etc in these oral systems, but I think they are resolvable and much better than written essays (which are now written by AI).
- jack_pp 2y ago
  the teacher knows you either way so the bias would be there for the written exam as well
  - bluGill 2y ago
    In a written exam they can cover the names - give you a random number as you enter the room and you write that on the paper, and but your name and number on a different paper. You also need to type everything out on a computer with spell check. (and even then if you write bucket or pail will identify you but it is unlikely any professor knows you well enough to tell those) When you audition for a symphony you perform behind a curtain and are required to wear soft slippers (so they can't tell if you are a wearing high heals - female). We can probably use voice changers so the examiner cannot tell who you are by your voice, but those tend to be fatiguing.
    - 2y ago

mountainb 2y ago

You can still do written essay evaluations. You could just require proctored exams whether or not you use software like Examsoft. If it's a topic that benefits from writing from a store of material, you can permit students to bring either unlimited supplemental printed material or a limited body of printed material into the exam room. For longer essays, you can just build in an oral examination component. This face time requirement is just not that hard to include given that even in lecture hall style settings you can rely on graduate student TAs who do not really cost anything. The thing is that the universities don't want to change how they run things. Adjuncts in most subjects don't cost anything and graduate students don't cost anything. They earn less than e.g. backroom stocking workers. This is also why they, by and large, all perform so poorly. 30 minutes of examiner time costs maybe $11 or less. Even for a lecture class with 130 students, that's under $1,500. Big woop. There are some small changes to grading practices that would make life very hard for AI cheaters, such as even cite checking a portion of citations in an essay. The real problem is that US universities are Soviet-style institutions in which gargantuan amounts of cash are dumped upon them and they pretend to work for it while paying the actual instructors nothing.

cmgbhm 2y ago
That’s 8 days of TA time. You’re going to get high variance and most likely having to boil it down to the equivalent of a multiple choice oral exam. Hiring a n TA to delegate grading that’s hard to verify seems like will cost more than you think.
- mountainb 2y ago
  So get more TAs. They cost less than Class B CDL drivers and will drag themselves over broken glass to take these jobs. And 65 hours of work per semester for oral examinations seems entirely reasonable. A week and a half for an FTE spent observing, with another half week to full week for grading seems completely reasonable for capstone semester work.

jvanderbot 2y ago

Well, that's not necessarily true. I was perhaps the most importunate student ever, and lingered around my professor's offices whenever they were open. I had endless questions, off topic and on. I was curious sure, but I was also annoying and pushy and wouldn't take no for an answer. In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me. So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.

selimthegrim 2y ago
TIL a new word.
gklitz 2y ago
The fact that one person can easily take a cup of water from a lake does not mean the lake supports for every person to take a cup. In fact, if everyone had tried to take that cup, then there wouldn’t even be a lake for the one person to take a cup from.

CharlieDigital 2y ago

> Written assay evaluation is not and has never been an effective evaluation. I kind of disagree. I've kept a blog for almost 20 years now and one thing is for sure: well-structured writing is very different from an oral exam the writing allows for restructuring your thoughts and ideas as you go and allows for far more depth. I don't think, for most folks, that they could have as much depth in an F2F as they could in their writing with the exception of true experts in their fields. The written essay has a cohesiveness and a structure to it that provides a better framework for evaluation and conveyance of information.

tgv 2y ago

> Instead you had to waste more time I'm not so sure that writing takes more time than studying. For starters, you don't have to memorize anything, and you can limit yourself to the assigned topic. Of course, it can be that students don't take studying for an oral exam seriously, and trust the teacher to only ask superficial questions.

WillAdams 2y ago

My best college professor (who was also an Episcopalian Priest) found the time to review one paper with each student once per semester. That strikes me as a workable bottom line.

zamadatix 2y ago

> It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher to evaluate your performance. Average student debt after a 4 year degree is ~$35,000 after ~45 courses. Before even running the math it should be obvious the gigantic cost of higher ed over 4 years is entirely unrelated to what an instructor would be making for ~23 hours of work (barring a secret society of multi millionaires). I.e. the problem you're identifying is the vast majority of $ spent in higher ed is not going to time with your professors, not that doing so is itself expensive.

l1silver 2y ago

> Written assay evaluation is not and has never been an effective evaluation. Could not disagree more. Researching, formulating arguments, can give a student a complete view of the subject that studying for tests misses. But, similarly to tests, it probably depends on the skill of the teacher in creating the right kind of written assignments.