4 ms·

Yep, I noticed this with myself too when I first did some grading a few months ago. There was also the factor that the ones I graded initially did not make cer

Yep, I noticed this with myself too when I first did some grading a few months ago. There was also the factor that the ones I graded initially did not make certain mistakes or answered in expected ways, such that when I did encounter unexpected answers/mistakes, I had to go back and rethink the grading on the papers I had graded previously. Eg if someone answered in a way that made me think an answer I considered incorrect was actually less wrong. I only had to deal with a small class, so backtracking was doable and I graded the papers in whatever shuffled up order they were turned in, otherwise there would have definitely been a bias.

bee_rider 3y ago
I especially noticed this when grading programming projects, because it is slightly complicated. I’d either find that: A bug was really common, got to re-evaluate after the first couple times I see it, apparently it is an easy mistake to make. Or, I’d find a new bug that was pretty common, but which I didn’t know about at first. Got to update my tests and re-run everybody. I tended to be really thorough and re-do the whole stack eventually, but it was a real pain. Could have half-assed it of course, but they spend weeks on these things, feel like I owe them honest feedback. It would tend to lead me to “softer” grading as well, if you are lazy and only check for a couple bugs, you might take a large number of points off for each problem. Finding some problems and punishing them harshly is not very fair for those students that randomly hit the bugs you expect. If you find every bug, you can only take a couple points off per bug without tanking everybody’s score.
JadeNB 3y ago
> I only had to deal with a small class, so backtracking was doable and I graded the papers in whatever shuffled up order they were turned in, otherwise there would have definitely been a bias. Grading papers in submission order just introduces a different bias, though. (For what it's worth, I'm in the same boat and I do the same, because I don't trust my ability to give the papers any true random sorting by hand, so I take the very weak randomization that the submission order gives me.)
- dotnet00 3y ago
  Introducing a slight bias factor that is randomized each time results in a lower average bias compared to a bias factor that is the same every time. Plus, as these weren't take-home assignments, I think someone finishing earlier is more likely to be either someone who was already going to score well, or someone who was already going to make the most common errors.
  - withinboredom 3y ago
    I take tests extremely quickly, I either know the answer or guess it from what I know. I don't think about it. I was usually one of the first people to turn in tests. I was usually (almost always) the last person to turn in assignments, I like to be one of the last people out of a door or the last person in a line (I don't like crowds). Grading by order-turned-in would almost always mean my assignment would be one of the first or last one's graded. If I were to guess that if you did a frequency analysis of people to order, you'd find there were always a certain group who turned it in first, and another group that turned it in last.
    - brewdad 3y ago
      You need to find a classmate to be a chaos gremlin that randomly mixes up the pile when they drop off their assignments.
  - JadeNB 3y ago
    > Introducing a slight bias factor that is randomized each time results in a lower average bias compared to a bias factor that is the same every time. That's what I'm saying—it's reasonable to believe that the submission time is correlated with other factors, such as ability or confidence (though the effect can cut both ways, with extremely able students submitting early because they finish early or late because they are extra careful, and similarly for other factors). Thus, this isn't really randomization, just correlation with another factor than the name.

bee_rider 3y ago

I especially noticed this when grading programming projects, because it is slightly complicated. I’d either find that: A bug was really common, got to re-evaluate after the first couple times I see it, apparently it is an easy mistake to make. Or, I’d find a new bug that was pretty common, but which I didn’t know about at first. Got to update my tests and re-run everybody. I tended to be really thorough and re-do the whole stack eventually, but it was a real pain. Could have half-assed it of course, but they spend weeks on these things, feel like I owe them honest feedback. It would tend to lead me to “softer” grading as well, if you are lazy and only check for a couple bugs, you might take a large number of points off for each problem. Finding some problems and punishing them harshly is not very fair for those students that randomly hit the bugs you expect. If you find every bug, you can only take a couple points off per bug without tanking everybody’s score.

JadeNB 3y ago

> I only had to deal with a small class, so backtracking was doable and I graded the papers in whatever shuffled up order they were turned in, otherwise there would have definitely been a bias. Grading papers in submission order just introduces a different bias, though. (For what it's worth, I'm in the same boat and I do the same, because I don't trust my ability to give the papers any true random sorting by hand, so I take the very weak randomization that the submission order gives me.)

dotnet00 3y ago
Introducing a slight bias factor that is randomized each time results in a lower average bias compared to a bias factor that is the same every time. Plus, as these weren't take-home assignments, I think someone finishing earlier is more likely to be either someone who was already going to score well, or someone who was already going to make the most common errors.
- withinboredom 3y ago
  I take tests extremely quickly, I either know the answer or guess it from what I know. I don't think about it. I was usually one of the first people to turn in tests. I was usually (almost always) the last person to turn in assignments, I like to be one of the last people out of a door or the last person in a line (I don't like crowds). Grading by order-turned-in would almost always mean my assignment would be one of the first or last one's graded. If I were to guess that if you did a frequency analysis of people to order, you'd find there were always a certain group who turned it in first, and another group that turned it in last.
  - brewdad 3y ago
    You need to find a classmate to be a chaos gremlin that randomly mixes up the pile when they drop off their assignments.
- JadeNB 3y ago
  > Introducing a slight bias factor that is randomized each time results in a lower average bias compared to a bias factor that is the same every time. That's what I'm saying—it's reasonable to believe that the submission time is correlated with other factors, such as ability or confidence (though the effect can cut both ways, with extremely able students submitting early because they finish early or late because they are extra careful, and similarly for other factors). Thus, this isn't really randomization, just correlation with another factor than the name.