4 ms·

My strategy was to, like you said, grade problem by problem. Then for each problem, first find all those who got full marks. Then group the others into piles

My strategy was to, like you said, grade problem by problem. Then for each problem, first find all those who got full marks. Then group the others into piles based on what mistakes they made. This ensures that everyone who made the same mistake(s) gets the same grade. It also tends to shuffle the order of the exams after every problem. Obviously you don’t need this strategy for simple multiple choice questions, and it’s probably also not a great fit for long-form essays. But it worked great for technical short answer problems in CS and security.

jcla1 2y ago
This sounds like an organisational nightmare to be honest. You'd be going through the pile of exams multiple times (at least twice) and what do you do if there are multiple mistakes that are common in a single exam question? Also: if you're sorting into "mistakes piles" for single exercises, how can you parallelise marking of separate and independent questions?
- cvwright 2y ago
  Teach at a broke public university, and you never have to juggle huge teams of TAs.
  - kkylin 2y ago
    Even at top-notch universities (public or private), when I talk to retired faculty, grading almost always comes up as a reason they don't want to teach anymore. [Edit: not disagreeing with your point.]
    - bobthepanda 2y ago
      Not only is it generally time intensive, you are also subject to lots of tiring back and forth with some students about their grades. No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade.
      - mschuster91 2y ago
        > No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade. Given that students have taken on hundreds of thousands of dollars in debt that they'll have to repay no matter what and on top of that a lot of jobs being completely out of reach these days without an academic degree (that for fucks sake isn't remotely required by virtually all jobs requiring it!), that's completely understandable. Want to fix higher education? Bring the hammer down on companies abusing it as a proxy for legally discriminating against classes of society that are closely correlated with poor academic outcomes. Academic education should be reserved for the best of the best of our youth, and it should be fully paid for by the government, not simply another hurdle to pass to get a job that pays barely more than flipping burgers.
        bobthepanda 2y ago
        I think it is rational that students can feel entitled to that. I also think that the vast majority of poorly paid, non-tenured professors and other teaching staff don't love being the targets of this harassment, since it's not their fault and largely out of their control, and it's not like they're getting the bulk of the tuition money. (That mostly goes to administrative expenses and sports programs.) Heck, most adjunct faculty are often paid below minimum wage and qualify for food stamps.
        bsder 2y ago
        > Given that students have taken on hundreds of thousands of dollars in debt that they'll have to repay no matter what and on top of that a lot of jobs being completely out of reach these days without an academic degree (that for fucks sake isn't remotely required by virtually all jobs requiring it!), that's completely understandable. Would that my students were this engaged before the exam. Guess which students show up the most often for office hours? ... yeah, the ones that are getting the best grades. If my students spent half as much time learning the subject as arguing with me about grades, they would be getting a higher grade than the one they are arguing for.
  - jcla1 2y ago
    I do (I'm a mathematican). We are usually between 4 and 10 people marking an exam with anywhere between 50 and 600 participants.
- kkylin 2y ago
  Online tools like Gradescope make this a little less painful (but still painful), but sometimes it's what's needed, especially on problems that are a little open-ended.
nextos 2y ago
My CS school implemented OCR test sheets, with some exceptions, and equivalent strategies, such as test suites and benchmarks for programming assignments. This was done to avoid subjective grading, as it was a big issue even in well-intentioned cases. Often, you still get big problems, but the set of solutions is small. It's always three options plus a fourth option (none / all). If you make a mistake you score negative points. It's not perfect, sometimes wording is ambiguous and it's unclear whether you need to tick the fourth catch-all option, but I found it better than the alternatives as it removes most arbitrariness from the process, but obviously has other issues. Regular exams often had wildly different grading standards for the same course depending on the class, and thus on the professor who was correcting exams. This was really annoying.
bobbiechen 2y ago
When I was a TA at CMU, we used Gradescope https://www.gradescope.com/ https://www.gradescope.com/ for this. Every exam would be scanned and divided into problems (based on a predefined template - fixed page space for answers). Then, each problem was assigned to a TA. Either there's a predefined rubric, or you create it as you go (-1 point for mistake X, half credit for mistake Y, etc.). There's a pretty slick interface where you just read the answer, and use keyboard shortcuts to apply the relevant deductions. It still has the issue that every time you change the rubric, you'd need to go back and re-do previously-graded instances of that problem. But it was way faster and (equally important) less tiring.
- Tijdreiziger 2y ago
  There’s also open-source software that does the same job at TU Delft: https://zesje.tudelft.nl/ https://zesje.tudelft.nl/ (disclaimer: I briefly worked on the software for my bachelor’s thesis)
underdeserver 2y ago
Sibling comment already said so, but I want to emphasize - this requires two run-throughs (at least). When I was grading homework, it took about 5 hours a week per class per run through. They didn't pay me enough to make sense for it to be 10 hours.
- raydev 2y ago
  A second pass wouldn't necessarily take the same amount of time, especially if you note the issues/concerns on your first pass.
  - underdeserver 2y ago
    True, but the overhead is large. I graded into linear algebra and intro calculus, so there were a lot of students - I think 150 or so - and most of them were wrong. Graders know that wrong homework takes much longer than correct homework to grade. It's correct? Full marks, move on. Is it wrong? Well, how wrong is it? Did they make a bad assumption, but followed it through to its conclusion? Did they forget a minus sign? Or is it complete hogwash? So it might not be 10 hours, but still would be around 8 hours. And that's still too much.
ska 2y ago
For final exams, we use to mark across all sections of a course (so for 101 type courses, this can be hundreds to 1000s of papers). Get all the profs and TA's together, break in to groups taking one problem or set of problems. Then you random sample (each group takes a stack) to get a feel for the 'typical' errors, once that's done - you are a machine going through the stacks. Every once in a while (not that often) you run into a novel error or approach, and the group discusses.
anticensor 2y ago
An even better strategy is to have the papers scanned by a double-sided scanner and graded by an AI grader.

jcla1 2y ago

This sounds like an organisational nightmare to be honest. You'd be going through the pile of exams multiple times (at least twice) and what do you do if there are multiple mistakes that are common in a single exam question? Also: if you're sorting into "mistakes piles" for single exercises, how can you parallelise marking of separate and independent questions?

cvwright 2y ago
Teach at a broke public university, and you never have to juggle huge teams of TAs.
- kkylin 2y ago
  Even at top-notch universities (public or private), when I talk to retired faculty, grading almost always comes up as a reason they don't want to teach anymore. [Edit: not disagreeing with your point.]
  - bobthepanda 2y ago
    Not only is it generally time intensive, you are also subject to lots of tiring back and forth with some students about their grades. No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade.
    - mschuster91 2y ago
      > No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade. Given that students have taken on hundreds of thousands of dollars in debt that they'll have to repay no matter what and on top of that a lot of jobs being completely out of reach these days without an academic degree (that for fucks sake isn't remotely required by virtually all jobs requiring it!), that's completely understandable. Want to fix higher education? Bring the hammer down on companies abusing it as a proxy for legally discriminating against classes of society that are closely correlated with poor academic outcomes. Academic education should be reserved for the best of the best of our youth, and it should be fully paid for by the government, not simply another hurdle to pass to get a job that pays barely more than flipping burgers.
      - bobthepanda 2y ago
        I think it is rational that students can feel entitled to that. I also think that the vast majority of poorly paid, non-tenured professors and other teaching staff don't love being the targets of this harassment, since it's not their fault and largely out of their control, and it's not like they're getting the bulk of the tuition money. (That mostly goes to administrative expenses and sports programs.) Heck, most adjunct faculty are often paid below minimum wage and qualify for food stamps.
      - bsder 2y ago
        > Given that students have taken on hundreds of thousands of dollars in debt that they'll have to repay no matter what and on top of that a lot of jobs being completely out of reach these days without an academic degree (that for fucks sake isn't remotely required by virtually all jobs requiring it!), that's completely understandable. Would that my students were this engaged before the exam. Guess which students show up the most often for office hours? ... yeah, the ones that are getting the best grades. If my students spent half as much time learning the subject as arguing with me about grades, they would be getting a higher grade than the one they are arguing for.
- jcla1 2y ago
  I do (I'm a mathematican). We are usually between 4 and 10 people marking an exam with anywhere between 50 and 600 participants.
kkylin 2y ago
Online tools like Gradescope make this a little less painful (but still painful), but sometimes it's what's needed, especially on problems that are a little open-ended.

nextos 2y ago

My CS school implemented OCR test sheets, with some exceptions, and equivalent strategies, such as test suites and benchmarks for programming assignments. This was done to avoid subjective grading, as it was a big issue even in well-intentioned cases. Often, you still get big problems, but the set of solutions is small. It's always three options plus a fourth option (none / all). If you make a mistake you score negative points. It's not perfect, sometimes wording is ambiguous and it's unclear whether you need to tick the fourth catch-all option, but I found it better than the alternatives as it removes most arbitrariness from the process, but obviously has other issues. Regular exams often had wildly different grading standards for the same course depending on the class, and thus on the professor who was correcting exams. This was really annoying.

bobbiechen 2y ago

When I was a TA at CMU, we used Gradescope https://www.gradescope.com/ https://www.gradescope.com/ for this. Every exam would be scanned and divided into problems (based on a predefined template - fixed page space for answers). Then, each problem was assigned to a TA. Either there's a predefined rubric, or you create it as you go (-1 point for mistake X, half credit for mistake Y, etc.). There's a pretty slick interface where you just read the answer, and use keyboard shortcuts to apply the relevant deductions. It still has the issue that every time you change the rubric, you'd need to go back and re-do previously-graded instances of that problem. But it was way faster and (equally important) less tiring.

Tijdreiziger 2y ago
There’s also open-source software that does the same job at TU Delft: https://zesje.tudelft.nl/ https://zesje.tudelft.nl/ (disclaimer: I briefly worked on the software for my bachelor’s thesis)

underdeserver 2y ago

Sibling comment already said so, but I want to emphasize - this requires two run-throughs (at least). When I was grading homework, it took about 5 hours a week per class per run through. They didn't pay me enough to make sense for it to be 10 hours.

raydev 2y ago
A second pass wouldn't necessarily take the same amount of time, especially if you note the issues/concerns on your first pass.
- underdeserver 2y ago
  True, but the overhead is large. I graded into linear algebra and intro calculus, so there were a lot of students - I think 150 or so - and most of them were wrong. Graders know that wrong homework takes much longer than correct homework to grade. It's correct? Full marks, move on. Is it wrong? Well, how wrong is it? Did they make a bad assumption, but followed it through to its conclusion? Did they forget a minus sign? Or is it complete hogwash? So it might not be 10 hours, but still would be around 8 hours. And that's still too much.

ska 2y ago

For final exams, we use to mark across all sections of a course (so for 101 type courses, this can be hundreds to 1000s of papers). Get all the profs and TA's together, break in to groups taking one problem or set of problems. Then you random sample (each group takes a stack) to get a feel for the 'typical' errors, once that's done - you are a machine going through the stacks. Every once in a while (not that often) you run into a novel error or approach, and the group discusses.

anticensor 2y ago

An even better strategy is to have the papers scanned by a double-sided scanner and graded by an AI grader.