3 ms·

>One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still

by tokai 3y ago

>One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still receive lower grades.

kibwen 3y ago
It would be less than ideal, but still an improvement over the current situation as long as the order is re-randomized for every assignment, because at least then you'd only be occasionally disadvantaged rather than consistently disadvantaged.
exe34 3y ago
It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.
- zeroonetwothree 3y ago
  It won’t average out perfectly. There will still be lucky and unlucky students. Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.
  - furyofantares 3y ago
    Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.
    - zeroonetwothree 3y ago
      Some courses are only graded based on a small number of tests. I actually went to UM and a grade might be something like 30% midterm 60% final 10% homework (obviously different professors have different systems). In that case if you get unlucky just twice on the two tests you basically get the full penalty.
      - furyofantares 3y ago
        I'm not sure how much a +/- 0.3 (out of 100) deviation from average on a single course matters even if you end up dead first/last for both midterm and final in that example. I mean, it will matter sometimes. But it's (by far) not as big a deal as if it happens for all your courses. Still, yes, you could flip the order from midterm to final instead of randomizing both and the effect goes to more like +/- 0.1 out of 100 for the luckiest and unluckiest.
        gwern 3y ago
        Yes, that sort of mirror-sampling would reduce variance. The problem is, though, you need to know all the uses of randomness in order to properly counterbalance them, and these systems are already enough of a pain to use. (For example, if you have two, you can simply swap: but what about other biases? like if it's broken in half to assign to 2 grades. Or what about if there are three exams? And what about balance across other courses? if you want to do variance-reduction and tricks like antithetic sampling, you need to know all this in order to structure it properly - get it wrong, and you may make things worse.) So that's why simple random shuffling would be preferred. It allows total ignorance of all other uses (past present and future), handles all ordering biases, and can be done independent in parallel across arbitrary sets of courses/exams/grades/students.
tetha 3y ago
There are however other factors involved in the grade, which have a higher impact on the grade. Like, understanding of the material and ability to present a solution. - E - I'm mostly saying that because a bunch of comments are jumping on this as a significant bias against some students. From my experience as a tutor, yes, this bias exists. But it won't turn a horribly wrong or an excellently correct solution into anything else. I eventually knew my strugglers and my excellers. I'd skim the excellers first, because if they messed up, something bad was going on. Then I'd go through the strugglers to see problems. And then I'd grade the rest first in whatever order I got the sheets, then the strugglers and then the excellers. I needed the baseline to see how bad the worst ones actually do. Some exercise sheets were an accidental adventure, I can tell you. And writing it like that, it sounds totally callous and cold. But focusing on the lower third in the exercises and communicating their struggles to the TA and prof was very appreciated by everyone, especially those students. It makes sure to get the important fundamentals right.

by tokai 3y ago

>One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still receive lower grades.

kibwen 3y ago

It would be less than ideal, but still an improvement over the current situation as long as the order is re-randomized for every assignment, because at least then you'd only be occasionally disadvantaged rather than consistently disadvantaged.

exe34 3y ago

It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.

zeroonetwothree 3y ago
It won’t average out perfectly. There will still be lucky and unlucky students. Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.
- furyofantares 3y ago
  Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.
  - zeroonetwothree 3y ago
    Some courses are only graded based on a small number of tests. I actually went to UM and a grade might be something like 30% midterm 60% final 10% homework (obviously different professors have different systems). In that case if you get unlucky just twice on the two tests you basically get the full penalty.
    - furyofantares 3y ago
      I'm not sure how much a +/- 0.3 (out of 100) deviation from average on a single course matters even if you end up dead first/last for both midterm and final in that example. I mean, it will matter sometimes. But it's (by far) not as big a deal as if it happens for all your courses. Still, yes, you could flip the order from midterm to final instead of randomizing both and the effect goes to more like +/- 0.1 out of 100 for the luckiest and unluckiest.
      - gwern 3y ago
        Yes, that sort of mirror-sampling would reduce variance. The problem is, though, you need to know all the uses of randomness in order to properly counterbalance them, and these systems are already enough of a pain to use. (For example, if you have two, you can simply swap: but what about other biases? like if it's broken in half to assign to 2 grades. Or what about if there are three exams? And what about balance across other courses? if you want to do variance-reduction and tricks like antithetic sampling, you need to know all this in order to structure it properly - get it wrong, and you may make things worse.) So that's why simple random shuffling would be preferred. It allows total ignorance of all other uses (past present and future), handles all ordering biases, and can be done independent in parallel across arbitrary sets of courses/exams/grades/students.

tetha 3y ago

There are however other factors involved in the grade, which have a higher impact on the grade. Like, understanding of the material and ability to present a solution. - E - I'm mostly saying that because a bunch of comments are jumping on this as a significant bias against some students. From my experience as a tutor, yes, this bias exists. But it won't turn a horribly wrong or an excellently correct solution into anything else. I eventually knew my strugglers and my excellers. I'd skim the excellers first, because if they messed up, something bad was going on. Then I'd go through the strugglers to see problems. And then I'd grade the rest first in whatever order I got the sheets, then the strugglers and then the excellers. I needed the baseline to see how bad the worst ones actually do. Some exercise sheets were an accidental adventure, I can tell you. And writing it like that, it sounds totally callous and cold. But focusing on the lower third in the exercises and communicating their struggles to the TA and prof was very appreciated by everyone, especially those students. It makes sure to get the important fundamentals right.