7 ms·

I work in academia. When we grade exams, the order of the exams on the stack is the order in which they were collected in the room (people can sit wherever they

by lqet 3y ago

I work in academia. When we grade exams, the order of the exams on the stack is the order in which they were collected in the room (people can sit wherever they like). For grading, we are usually 5 people in a single room, and everyone grades a specific exercise for consistency. The exams are getting shuffled heavily, with everyone just grabbing stacks, looking for exams where "their" exercise was not yet graded, and taking them out. So basically, the order in which we grade exams can be considered random. However, I also grade weekly exercise sheets during the semester, and these are committed into a repository, where each student has a folder that... begins with the first letter of their first name. Everyone I have ever worked with acknowledges that you have to shuffle the order in which you grade these submissions each week, for fairness. Several effects come into play: (1) your are usually less tired at the beginning, (2) your mood gets better during the last 2 sheets because you know you are done soon, (3, and crucially) at the beginning, you have not yet seen all the common errors / developed a "feeling" for them, and you might thus miss them in early submissions, but spot them immediately in later submissions. Another alphabetic effect: In elementary school, my name was on top of the list of students in my class. I remember that I often had to do some special job simply because I was the first name on this list (for example, carry a group ticket when we visited some museum, keep track of something, be the first at something where nobody wanted to be the first, with everyone watching, be the first to be graded in PE, again with everyone watching, etc.). As a fairly shy kid, this already annoyed me in first grade.

V__ 3y ago
A teacher friend of mine always goes through his stack twice. Once to correct all mistakes and a second time to write down points. As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is.
- smogcutter 3y ago
  > As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is. Crucially, this is not quite what the poster said. It’s not about stack ranking students against each other. Say every paper makes the same subtle mistake, and you only notice it halfway through the pile. Unless you go back through them all, you’ll unfairly grade the later entries more harshly.
  - Zancarius 3y ago
    > It’s not about stack ranking students against each other. It's not, but it sort of has that effect, albeit indirectly, and definitely unfairly.
    - smogcutter 3y ago
      I think we’re talking about the same thing, but to clarify my meaning: If you weigh the severity of students mistakes (or successes for that matter) in relation to each other rather than to an objective rubric, you’re effectively stack ranking them whether you mean to or not.
  - kkylin 3y ago
    I'm not a big fan of putting everything in the cloud, but one of the advantages of online grading systems is that it is easier to make this kind of adjustment. The workflow goes like this: make a rubric item for a specific kind of mistake (it takes a little experience to know which mistakes are likely one-off and which ones are likely to be repeated by other students), assign X points, and later if you decide there are worse mistakes, adjust the points and that gets applied to everyone.
cvwright 3y ago
My strategy was to, like you said, grade problem by problem. Then for each problem, first find all those who got full marks. Then group the others into piles based on what mistakes they made. This ensures that everyone who made the same mistake(s) gets the same grade. It also tends to shuffle the order of the exams after every problem. Obviously you don’t need this strategy for simple multiple choice questions, and it’s probably also not a great fit for long-form essays. But it worked great for technical short answer problems in CS and security.
- jcla1 3y ago
  This sounds like an organisational nightmare to be honest. You'd be going through the pile of exams multiple times (at least twice) and what do you do if there are multiple mistakes that are common in a single exam question? Also: if you're sorting into "mistakes piles" for single exercises, how can you parallelise marking of separate and independent questions?
  - cvwright 3y ago
    Teach at a broke public university, and you never have to juggle huge teams of TAs.
    - kkylin 3y ago
      Even at top-notch universities (public or private), when I talk to retired faculty, grading almost always comes up as a reason they don't want to teach anymore. [Edit: not disagreeing with your point.]
      - bobthepanda 3y ago
        Not only is it generally time intensive, you are also subject to lots of tiring back and forth with some students about their grades. No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade.
pjdesno 3y ago
There are all sorts of good ways to avoid these biases. I use the same practice described above for paper exams, and grading order for eg question 2 may be affected by score on question 1, but it won’t be affected by name or ID number. If you use Canvas or Gradescope with the default settings, it’s almost impossible to avoid this sort of bias. Worse yet, in Gradescooe you’re strongly steered towards grading with a fixed “rubric” with specific points off for each of N pre-defined errors, allowing grading to be done by TAs with little more knowledge than the students themselves, resulting in scores which have little relationship to the quality of the student answer.
spullara 3y ago
Everything in this thread just randomizes who doesn't get graded fairly.
- jibe 3y ago
  For a single assignment, yes. But at least randomization might mitigate the effect across a term.
- karaterobot 3y ago
  Is there a better solution? It's not for teachers to be perfect. Since that's not possible, it's not a solution.
  - thaumasiotes 3y ago
    Yes, you can grade objectively.
  - jtriangle 3y ago
    No, the solution is for the scoring to be handled by software that doesn't exist yet. Some things have easy, objective measures of correctness. STEM is mostly this way. Others, your humanities et al, are fairly subjective. You could probably cover most of this with an LLM, and access to a large body of graded material for a given course, provided said material was graded fairly. Generating that data would be time consuming, as, any given assignment would need to be graded by as many people as possible in order to find a fair average. From there, it's simple comparison between your sample work and the presented work. We're probably a decade from this really being viable en masse, but, it no doubt will happen, and for better or worse we'll likely end up with EDUAAS (education as a service).
    - CuriousCosmic 3y ago
      LLMs are not going to be a solution. LLMs have absolutely no concept of truth. And not everything has an objective solution. Even those that do often have a process associated with them and factoring in that work/process is an important part of grading. Reducing that subjective grading process to only objective solutions being right is grossly reductive and disproportionately punishes students who have the process right and understand the material but make small errors. That's exactly what you don't want to do. --- Instead the solution is to make sure each assignment gets multiple eyes on it and in a random order. Then to document biases and trends in biases so that the TAs and professors can be aware of them and mitigate them. It's a process problem that can only be solved by a process solution. Replacing the graders with technology or reducing problems to a binary right/wrong will never ever solve this and in many cases will end up being more harmful than the biases they claim to solve would be.
dheera 3y ago
When I saw the title I would have thought that the higher concentrations of Asian names starting with V, W, X, Y, Z would have led to higher grades at that end of the alphabet, and thought that effect would have eclipsed anything else.
- pks016 3y ago
  Anecdotally, the course I grade has this effect (just looking at the average score). I have been grading this course from last 5 years(9-10 times). Last names with L-Z score slightly more than A-L.
- lupire 3y ago
  Indian names start with A,B, N. Chinese names also start with, C, F, L.
donatj 3y ago
In around the year 2000 I had an essay due that day I had forgotten, and about ten minutes of computer lab time before home room in the morning. I wrote an introduction and conclusion; then filled the remainder with copy pasted chunks of the introduction and conclusion. The thought being at least I’d get a laugh. If anyone had read the thing it would have been clear it was nonsense. I received an 80% with no notes or markup. I have been left wondering for the last 25 years how much student work is actually even reviewed. I work in EdTech and every time we add a feature that requires manual teacher review of student work you will see that some teachers are VERY diligent while others never touch it.
- filipezf 3y ago
  There was this numerical calculus class at Uni where the teacher forbid us to use the calculator. So I just programmed the integral on it, got the partial steps, and just wrote random numbers to fill the the substeps. Got full grade :D The other case everybody got to pass the class, but after vacation we found the stack of exams completely untouched under a desk. The teacher had a side business to run...
- jtriangle 3y ago
  I know a guy who copy/pasted a wikipedia article, in line citations and all, and submitted it for a sociology class and got an A, no notes, nothing.
  - mixmastamyk 3y ago
    He “only cheated himself.” :-D
    - wolverine876 3y ago
      The point is to develop skills and knowledge, so I would agree. Do you disagree?
      - mixmastamyk 3y ago
yeahwhatever10 3y ago
When I was a TA I always did a second pass to make sure everything was even. It’s not that hard.
- eks391 3y ago
  It's hard when you are the only TA for 260 students who get 3 assignments per week, you must also hold free hours and you aren't allowed to go over 27 hrs each week so the school isnt breaking federal laws.
madeofpalk 3y ago
> you have to shuffle the order in which you grade these submissions each week, for fairness I don't think this is fair. It's just a more randomly distributed unfairness, rather than by a deterministic factor (like the student's name) 'Fair' would be each student is assessed independently for the work they did, rather than their mark being impacted by how early or late they were marked.
- luplex 3y ago
  There are many notions of "fairness", many of which are logically incompatible with each other. In this example, I think it's kind of fair to give everyone an equal chance of being advantaged. You're not hurting anyone specifically.
- shepherdjerred 3y ago
  It would be essentially impossible to have something "truly" fair for open-ended questions since humans are stateful. Maybe this is a case that AI could actually do quite well. Manually grade the answers and identify the classes of mistakes. Then hand the classes of mistakes to the AI and ask for it to determine which answers have which types of mistakes. Once you've done that, you just need to associate a deduction for each type of mistake and do some simple math.
  - vagrantJin 3y ago
    what do you mean AI? you must be joking.
    - shepherdjerred 3y ago
      Imagine a question: compare bubble sort and quick sort algorithm. Some students might mix up the algorithms, some might give the incorrect computation complexity, some might describe them incorrectly in some way. Manually grade some (or all of) the answers by noting the kinds of things students got wrong (e.g. the above criteria). Then, feed in to ChatGPT (or your favorite alternative) the answer + the categories of mistakes to expect. Here's a simplified example: https://chat.openai.com/share/bf801e12-51d5-4255-9968-bbf91bb29196 https://chat.openai.com/share/bf801e12-51d5-4255-9968-bbf91b...
gonzo41 3y ago
Have you ever thought about just passing out a set of grades on random to random individuals and see how that shakes out. Like totally random and unjustified grades. D minus for an A+ student. A+ for fails etc. Just random chaos. Then just score the final correctly and see the effect? Or just having a Kafkaesque pass fail grade with no feedback for each student relative to their own performance over time with an expected growth rate applied?
bandrami 3y ago
We tried a lot of things. What eventually worked was ending grades. You mastered the material or you did not; perhaps a couple of students mastered it with high marks. Obvs this takes an administration that is OK with that, which most aren't.
- dev_tty01 3y ago
  Having hired a lot of engineers, I can tell you that mastery of material is nothing close to a bimodal distribution.
Fnoord 3y ago
> [..] As a fairly shy kid, this already annoyed me in first grade. (I suppose the cons outweighted the cons.) Did you perceive any pros? I suppose one way to do grades is first read through all papers to get an idea of the levels of the students. Though you still have bias/nepotism and such then. Perhaps a teamwork or commitee would work, or teachers swapping classes/schools? I had a French teacher on high school who dropped a pen on list of students and then where it landed that person would get rehearsal. People in mid (waves) were fried. Plus, there is also the issue of certain last names being common in certain cultures, leading to skewed statistics.
ripjaygn 3y ago
While this helps the students with names lower down the order, people who are graded later still suffer.
xorvoid 3y ago
We graded similarly, incidentally, when I was at U-of-M (lol). I don’t think we ever sorted by name so I don’t know if we’d have a bias effect by name unless it’s an implicit bias towards lexicographical esthetics. I won’t deny that grading fatigue can have subjective effects. I always thought we did a pretty fair and objective job. I taught Computer Architecture and we we developed answer keys and grading scales before grading a single test. Of course assigning partial credit always ended up being pretty subjective. Typically though people would error in the same ways and so those would be subjectively identical. I never thought names factored into this much but, to be fair, no one ever collected data… Finally, I guess I’ll admit that I’m probably very biased because my initials as A.B. and I’ve always gotten excellent grades, so… maybe maybe maybe
starttoaster 3y ago
This might come off rude on accident, but I mean genuinely without malice. When I'm writing an essay to submit to my professor/teacher, I am asked to make multiple drafts to get a proper end result that is ready to submit. Understanding that educational staff is already often overworked, should I expect _less_ from the person I receive my education from? If you acknowledge that many of the grades I receive are actually not fair to me, and there's an attempt to randomize the order that papers are graded, many of the grades that I received (whether high or low) were done partially (that is to say, the opposite of "impartially".) And there's a real concern that in your example where the submissions are committed to a repository that you need to shuffle, that my submission ends up in a similar position in the stack week after week, unless you're actually doing something to ensure my position in the stack is different between submissions. It's probably sufficient in many cases but doesn't guarantee randomness unless the algorithm to randomize submissions takes previous stack orderings into account.
- jamiek88 3y ago
  It’s simply human nature. Teachers can either lie to themselves and you about it or mitigate it. What more could you possible want from them as humans?
  - starttoaster 3y ago
    I somewhat assumed there would be commenters suggesting the human angle as a retort. That's why I prefaced with both "this is what the teacher expects of me" and "understanding that educational staff is already often overworked." It just seems to me that the current systems aren't sufficient, and acknowledging that is what leads people to improving those systems. The above commentor suggested what they do in academia as workarounds to what the study showed, and I'm saying even that is not sufficient. It seems like you're agreeing with me, but jumping to their defense with "people are fallible." People are fallible, that's why we build systems to take human elements out of it. Recognizing where humanity has soured something is key to that.
- advael 3y ago
  I know it's not the point of your post, but I think it's worth pointing out that you're misunderstanding randomness (albeit in a very typical way). Although randomness is likely eventually (over a lot of instances) going to be the most "fair" way to distribute where your submission is in the order, it does not guarantee that it will always be different, and in fact a "random" algorithm that took previous orderings into account would be provably less random than one that didn't It's also worth noting that randomization in a context like this is inherently an imperfect solution to a problem that generally can't be solved perfectly. If we find out that weird ordering biases exist, I think randomization is done on the assumption that many we don't know about could also exist, that there's no clear way to mitigate them completely, and then randomizing the order per-instance is just the best we can do to ensure it's fair (Which, again, won't be perfect. Perfect isn't available)
euroderf 2y ago
For grading essay assignments, and possibly also essay-style exams: It is important to get a feel for the collective level of writing before grading essays individually, and it is important to avoid over-grading or under-grading essays at the beginning or end of a stream of papers. Therefore I did a three-stage grading process, with three colors of pens: The first pass, with a red pen, is marking up single-point problems like misspellings and glaring usage errors. Also of course the general level of writing begins to seep in. This pass of course includes all papers, and it can pass fairly quickly. The second pass, with a green pen, mostly just marks in the margin where a (good) point is made or a conclusion is reached. This is to prepare for the next pass. Again, all papers are done in this pass. The third pass, in blue pen, is where the quality of the writing is assessed and critiqued. Maybe some short notes in the margins, maybe just comments at the end of the essay. When students get their papers back, there are some chuckles (or whatevs) when students see all the pretty colors. But after I explain the method and its rationale, the method is clear and understood (and also appreciated?).
Jsebast24 2y ago
Only people who's last name start with an A can relate. The rest, I have found, just don't care. It's a curse you have to carry from 1st grade and all through school and college.

by lqet 3y ago

V__ 3y ago

A teacher friend of mine always goes through his stack twice. Once to correct all mistakes and a second time to write down points. As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is.

smogcutter 3y ago
> As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is. Crucially, this is not quite what the poster said. It’s not about stack ranking students against each other. Say every paper makes the same subtle mistake, and you only notice it halfway through the pile. Unless you go back through them all, you’ll unfairly grade the later entries more harshly.
- Zancarius 3y ago
  > It’s not about stack ranking students against each other. It's not, but it sort of has that effect, albeit indirectly, and definitely unfairly.
  - smogcutter 3y ago
    I think we’re talking about the same thing, but to clarify my meaning: If you weigh the severity of students mistakes (or successes for that matter) in relation to each other rather than to an objective rubric, you’re effectively stack ranking them whether you mean to or not.
- kkylin 3y ago
  I'm not a big fan of putting everything in the cloud, but one of the advantages of online grading systems is that it is easier to make this kind of adjustment. The workflow goes like this: make a rubric item for a specific kind of mistake (it takes a little experience to know which mistakes are likely one-off and which ones are likely to be repeated by other students), assign X points, and later if you decide there are worse mistakes, adjust the points and that gets applied to everyone.

cvwright 3y ago

My strategy was to, like you said, grade problem by problem. Then for each problem, first find all those who got full marks. Then group the others into piles based on what mistakes they made. This ensures that everyone who made the same mistake(s) gets the same grade. It also tends to shuffle the order of the exams after every problem. Obviously you don’t need this strategy for simple multiple choice questions, and it’s probably also not a great fit for long-form essays. But it worked great for technical short answer problems in CS and security.

jcla1 3y ago
This sounds like an organisational nightmare to be honest. You'd be going through the pile of exams multiple times (at least twice) and what do you do if there are multiple mistakes that are common in a single exam question? Also: if you're sorting into "mistakes piles" for single exercises, how can you parallelise marking of separate and independent questions?
- cvwright 3y ago
  Teach at a broke public university, and you never have to juggle huge teams of TAs.
  - kkylin 3y ago
    Even at top-notch universities (public or private), when I talk to retired faculty, grading almost always comes up as a reason they don't want to teach anymore. [Edit: not disagreeing with your point.]
    - bobthepanda 3y ago
      Not only is it generally time intensive, you are also subject to lots of tiring back and forth with some students about their grades. No grading is perfect, but there’s also some undercurrent of an attitude that students have paid to be there and are entitled to a certain grade.

pjdesno 3y ago

There are all sorts of good ways to avoid these biases. I use the same practice described above for paper exams, and grading order for eg question 2 may be affected by score on question 1, but it won’t be affected by name or ID number. If you use Canvas or Gradescope with the default settings, it’s almost impossible to avoid this sort of bias. Worse yet, in Gradescooe you’re strongly steered towards grading with a fixed “rubric” with specific points off for each of N pre-defined errors, allowing grading to be done by TAs with little more knowledge than the students themselves, resulting in scores which have little relationship to the quality of the student answer.

spullara 3y ago

Everything in this thread just randomizes who doesn't get graded fairly.

jibe 3y ago
For a single assignment, yes. But at least randomization might mitigate the effect across a term.
karaterobot 3y ago
Is there a better solution? It's not for teachers to be perfect. Since that's not possible, it's not a solution.
- thaumasiotes 3y ago
  Yes, you can grade objectively.
- jtriangle 3y ago
  No, the solution is for the scoring to be handled by software that doesn't exist yet. Some things have easy, objective measures of correctness. STEM is mostly this way. Others, your humanities et al, are fairly subjective. You could probably cover most of this with an LLM, and access to a large body of graded material for a given course, provided said material was graded fairly. Generating that data would be time consuming, as, any given assignment would need to be graded by as many people as possible in order to find a fair average. From there, it's simple comparison between your sample work and the presented work. We're probably a decade from this really being viable en masse, but, it no doubt will happen, and for better or worse we'll likely end up with EDUAAS (education as a service).
  - CuriousCosmic 3y ago
    LLMs are not going to be a solution. LLMs have absolutely no concept of truth. And not everything has an objective solution. Even those that do often have a process associated with them and factoring in that work/process is an important part of grading. Reducing that subjective grading process to only objective solutions being right is grossly reductive and disproportionately punishes students who have the process right and understand the material but make small errors. That's exactly what you don't want to do. --- Instead the solution is to make sure each assignment gets multiple eyes on it and in a random order. Then to document biases and trends in biases so that the TAs and professors can be aware of them and mitigate them. It's a process problem that can only be solved by a process solution. Replacing the graders with technology or reducing problems to a binary right/wrong will never ever solve this and in many cases will end up being more harmful than the biases they claim to solve would be.

dheera 3y ago

When I saw the title I would have thought that the higher concentrations of Asian names starting with V, W, X, Y, Z would have led to higher grades at that end of the alphabet, and thought that effect would have eclipsed anything else.

pks016 3y ago
Anecdotally, the course I grade has this effect (just looking at the average score). I have been grading this course from last 5 years(9-10 times). Last names with L-Z score slightly more than A-L.
lupire 3y ago
Indian names start with A,B, N. Chinese names also start with, C, F, L.

donatj 3y ago

In around the year 2000 I had an essay due that day I had forgotten, and about ten minutes of computer lab time before home room in the morning. I wrote an introduction and conclusion; then filled the remainder with copy pasted chunks of the introduction and conclusion. The thought being at least I’d get a laugh. If anyone had read the thing it would have been clear it was nonsense. I received an 80% with no notes or markup. I have been left wondering for the last 25 years how much student work is actually even reviewed. I work in EdTech and every time we add a feature that requires manual teacher review of student work you will see that some teachers are VERY diligent while others never touch it.

filipezf 3y ago
There was this numerical calculus class at Uni where the teacher forbid us to use the calculator. So I just programmed the integral on it, got the partial steps, and just wrote random numbers to fill the the substeps. Got full grade :D The other case everybody got to pass the class, but after vacation we found the stack of exams completely untouched under a desk. The teacher had a side business to run...
jtriangle 3y ago
I know a guy who copy/pasted a wikipedia article, in line citations and all, and submitted it for a sociology class and got an A, no notes, nothing.
- mixmastamyk 3y ago
  He “only cheated himself.” :-D
  - wolverine876 3y ago
    The point is to develop skills and knowledge, so I would agree. Do you disagree?
    - mixmastamyk 3y ago

yeahwhatever10 3y ago

When I was a TA I always did a second pass to make sure everything was even. It’s not that hard.

eks391 3y ago
It's hard when you are the only TA for 260 students who get 3 assignments per week, you must also hold free hours and you aren't allowed to go over 27 hrs each week so the school isnt breaking federal laws.

madeofpalk 3y ago

> you have to shuffle the order in which you grade these submissions each week, for fairness I don't think this is fair. It's just a more randomly distributed unfairness, rather than by a deterministic factor (like the student's name) 'Fair' would be each student is assessed independently for the work they did, rather than their mark being impacted by how early or late they were marked.

luplex 3y ago
There are many notions of "fairness", many of which are logically incompatible with each other. In this example, I think it's kind of fair to give everyone an equal chance of being advantaged. You're not hurting anyone specifically.
shepherdjerred 3y ago
It would be essentially impossible to have something "truly" fair for open-ended questions since humans are stateful. Maybe this is a case that AI could actually do quite well. Manually grade the answers and identify the classes of mistakes. Then hand the classes of mistakes to the AI and ask for it to determine which answers have which types of mistakes. Once you've done that, you just need to associate a deduction for each type of mistake and do some simple math.
- vagrantJin 3y ago
  what do you mean AI? you must be joking.
  - shepherdjerred 3y ago
    Imagine a question: compare bubble sort and quick sort algorithm. Some students might mix up the algorithms, some might give the incorrect computation complexity, some might describe them incorrectly in some way. Manually grade some (or all of) the answers by noting the kinds of things students got wrong (e.g. the above criteria). Then, feed in to ChatGPT (or your favorite alternative) the answer + the categories of mistakes to expect. Here's a simplified example: https://chat.openai.com/share/bf801e12-51d5-4255-9968-bbf91bb29196 https://chat.openai.com/share/bf801e12-51d5-4255-9968-bbf91b...

gonzo41 3y ago

Have you ever thought about just passing out a set of grades on random to random individuals and see how that shakes out. Like totally random and unjustified grades. D minus for an A+ student. A+ for fails etc. Just random chaos. Then just score the final correctly and see the effect? Or just having a Kafkaesque pass fail grade with no feedback for each student relative to their own performance over time with an expected growth rate applied?

bandrami 3y ago

We tried a lot of things. What eventually worked was ending grades. You mastered the material or you did not; perhaps a couple of students mastered it with high marks. Obvs this takes an administration that is OK with that, which most aren't.

dev_tty01 3y ago
Having hired a lot of engineers, I can tell you that mastery of material is nothing close to a bimodal distribution.

Fnoord 3y ago

> [..] As a fairly shy kid, this already annoyed me in first grade. (I suppose the cons outweighted the cons.) Did you perceive any pros? I suppose one way to do grades is first read through all papers to get an idea of the levels of the students. Though you still have bias/nepotism and such then. Perhaps a teamwork or commitee would work, or teachers swapping classes/schools? I had a French teacher on high school who dropped a pen on list of students and then where it landed that person would get rehearsal. People in mid (waves) were fried. Plus, there is also the issue of certain last names being common in certain cultures, leading to skewed statistics.

ripjaygn 3y ago

While this helps the students with names lower down the order, people who are graded later still suffer.

xorvoid 3y ago

We graded similarly, incidentally, when I was at U-of-M (lol). I don’t think we ever sorted by name so I don’t know if we’d have a bias effect by name unless it’s an implicit bias towards lexicographical esthetics. I won’t deny that grading fatigue can have subjective effects. I always thought we did a pretty fair and objective job. I taught Computer Architecture and we we developed answer keys and grading scales before grading a single test. Of course assigning partial credit always ended up being pretty subjective. Typically though people would error in the same ways and so those would be subjectively identical. I never thought names factored into this much but, to be fair, no one ever collected data… Finally, I guess I’ll admit that I’m probably very biased because my initials as A.B. and I’ve always gotten excellent grades, so… maybe maybe maybe

starttoaster 3y ago

This might come off rude on accident, but I mean genuinely without malice. When I'm writing an essay to submit to my professor/teacher, I am asked to make multiple drafts to get a proper end result that is ready to submit. Understanding that educational staff is already often overworked, should I expect _less_ from the person I receive my education from? If you acknowledge that many of the grades I receive are actually not fair to me, and there's an attempt to randomize the order that papers are graded, many of the grades that I received (whether high or low) were done partially (that is to say, the opposite of "impartially".) And there's a real concern that in your example where the submissions are committed to a repository that you need to shuffle, that my submission ends up in a similar position in the stack week after week, unless you're actually doing something to ensure my position in the stack is different between submissions. It's probably sufficient in many cases but doesn't guarantee randomness unless the algorithm to randomize submissions takes previous stack orderings into account.

jamiek88 3y ago
It’s simply human nature. Teachers can either lie to themselves and you about it or mitigate it. What more could you possible want from them as humans?
- starttoaster 3y ago
  I somewhat assumed there would be commenters suggesting the human angle as a retort. That's why I prefaced with both "this is what the teacher expects of me" and "understanding that educational staff is already often overworked." It just seems to me that the current systems aren't sufficient, and acknowledging that is what leads people to improving those systems. The above commentor suggested what they do in academia as workarounds to what the study showed, and I'm saying even that is not sufficient. It seems like you're agreeing with me, but jumping to their defense with "people are fallible." People are fallible, that's why we build systems to take human elements out of it. Recognizing where humanity has soured something is key to that.
advael 3y ago
I know it's not the point of your post, but I think it's worth pointing out that you're misunderstanding randomness (albeit in a very typical way). Although randomness is likely eventually (over a lot of instances) going to be the most "fair" way to distribute where your submission is in the order, it does not guarantee that it will always be different, and in fact a "random" algorithm that took previous orderings into account would be provably less random than one that didn't It's also worth noting that randomization in a context like this is inherently an imperfect solution to a problem that generally can't be solved perfectly. If we find out that weird ordering biases exist, I think randomization is done on the assumption that many we don't know about could also exist, that there's no clear way to mitigate them completely, and then randomizing the order per-instance is just the best we can do to ensure it's fair (Which, again, won't be perfect. Perfect isn't available)

euroderf 2y ago

For grading essay assignments, and possibly also essay-style exams: It is important to get a feel for the collective level of writing before grading essays individually, and it is important to avoid over-grading or under-grading essays at the beginning or end of a stream of papers. Therefore I did a three-stage grading process, with three colors of pens: The first pass, with a red pen, is marking up single-point problems like misspellings and glaring usage errors. Also of course the general level of writing begins to seep in. This pass of course includes all papers, and it can pass fairly quickly. The second pass, with a green pen, mostly just marks in the margin where a (good) point is made or a conclusion is reached. This is to prepare for the next pass. Again, all papers are done in this pass. The third pass, in blue pen, is where the quality of the writing is assessed and critiqued. Maybe some short notes in the margins, maybe just comments at the end of the essay. When students get their papers back, there are some chuckles (or whatevs) when students see all the pretty colors. But after I explain the method and its rationale, the method is clear and understood (and also appreciated?).

Jsebast24 2y ago

Only people who's last name start with an A can relate. The rest, I have found, just don't care. It's a curse you have to carry from 1st grade and all through school and college.