11 ms·

U-M finds students with alphabetically lower-ranked names receive lower grades

• hilux 3y ago
  > Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably through enforced randomizing.
  • buggy6257 3y ago
    Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately affected by this phenomenon.
    • hilux 3y ago
      In the real world, universities are never going to fix the problem of overworked and underpaid grad students getting tired.
    • bluGill 3y ago
      > it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.
      • bigfudge 3y ago
        In my case, I have to make a conscious effort to remain consistently (in)tolerant of lazy writing. It’s hard to keep on reading between the lines and giving the benefit of the doubt.
      • rjzzleep 3y ago
        I had the same conclusion. You learn things as you go, including things you don't like.
      • 13of40 3y ago
        I've participated in day-long and multi-day interview events for job candidates, and I see the same effect. At the beginning you don't have a frame of reference and you're more likely to question your own decision or give someone the benefit of the doubt, but by the end you're far more systematic, plus a little bit numb to the effect your decision is having.
        • cyanydeez 3y ago
          For grading, you could probably just add a mediating factor and throw in test cases that calibrate the factor and then you curve everyone on that factor. It'd seemingly be more work but would result in averages that are more reasonable to the changes in stress.
        • throwaway35777 3y ago
          > by the end you're far more systematic, plus a little bit numb to the effect your decision is having Maybe decision fatigue is supposed to bias humans toward the optimal solution for the fiancee problem [1]. [1] https://en.m.wikipedia.org/wiki/Secretary_problem https://en.m.wikipedia.org/wiki/Secretary_problem
      • deleted 3y ago
        [deleted]
      • labcomputer 3y ago
        Yes, and: Additionally, universities (and, by extension departments) want grades to approximately follow a normal distribution (and yes, you in the back, their actions show they do actually want that, even if they say otherwise). When you start grading a problem you have some idea what a "good" solution looks like, what an "ok" solution looks like, and same for "bad" solutions... If you award points based on that, the result will be a normal-ish distribution. But your idea of a good/ok/bad solution evolves as you see more papers. There's two reasons for that: First, you can't (ahead of time) imagine all the ways that students will invent to fuck up a problem set, and find edge cases in your grading rubric that result in unfairly-high or -low scores. As you gain experience teaching, you will anticipate more of the ways, but you will never anticipate every way. Second, the TA/grader wants to be able to stack-rank the papers and have the scores be monotonic. The grader wants this because non-monotonic scoring triggers far more complaining than harsh scoring or picky scoring. When you come across papers that are worse than ones you've already recently graded, you assign even lower scores. This results in a ratcheting effect with more extreme scores as you get closer to the bottom of the pile. But, since the mean score is usually a B/B-/C+ (~75-85), and since scores are usually limited to the range 0-100, this means that papers closer to the bottom will receive statistically lower scores. Now, you could go back a re-grade ones you've already done, but: 1. The university is officially only paying you for 20hrs/week (and requires a signed end-of-semester statement attesting to the same). 2. The assigned workload of teaching and grading doesn't permit a two-pass grading scheme while keeping within 20 hours. 3. If you complain to the graduate ombudsman about the workload needing more than 20 hours, you won't have funding next semester (so you have a prisoner's dilemma among TAs who might want to grade more fairly). 4. If you're grading (say) a final exam for a frosh/soph class, you're probably in a room with 4-8 other graders late into the night. One effective way to make your coworkers hate you is to be that guy who always finishes grading his stack last, when everyone is worried about catching the last train/bus. Basically, all the incentives are aligned to make this happen.
    • WaitWaitWha 3y ago
      > graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. I will admit to this. Initially, my patience and tolerance for errors is significantly higher than towards the end of the grading. By the second hour grading, I am not only mentally exhausted my tolerance is significantly lower. I try to prevent this by creating very explicit grading rubric and I stick to it as much as possible.
      • ghaff 3y ago
        Clear rubrics are the thing where possible. They aren't everywhere though. I've been on conference committees and so many different factors come into play--including how late in the day it is. But, in that case, a bunch of people are rating and commenting and there's no strict order so it probably evens out to a reasonable degree.
    • andix 3y ago
      Even distribution would fix the problem. If grading has a subjective component, there will always be deviations from the "correct" grade. If those patterns are randomly distributed over all students, their grade averages will be comparable again.
    • BugsJustFindMe 3y ago
      > Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.
      • zeroonetwothree 3y ago
        Random order would still mean a few students in the class get unlucky and near the end the majority of the time. Although over the course of all classes it would tend to even out somewhat. It’s certainly better than fixed order.
        • deleted 3y ago
          [deleted]
        • BugsJustFindMe 3y ago
          "randomization" is not the important part here. "evenly distributing" is. It is absolutely possible to reorder the sequence fairly such that your scenario doesn't occur. It could even to a human observer look randomized if you want. In a trivial example case where the effect were linear you could just switch the order back and forth, and on average every student would receive the same middle-of-group impact.
      • whiterknight 3y ago
        The mistake is assuming grades are an objective measurement, and not gamification to try to help you learn.
        • BugsJustFindMe 3y ago
          It's a common mistake. So common, in fact, that it has real practical impact on students at the edge who might not otherwise have failed or passed.
    • throwaway35777 3y ago
      I was a grader once. I guarantee if someone gives a good answer they'll get full marks even near the bottom of the stack. For BS answers I'll admit I got less generous as the hours went on. No one's getting hurt by this system if it's randomized. It's a matter of graders giving out partial credit for wrong answers which is discretionary. Rarely students are granted a small mercy. Seems OK.
      • bumby 3y ago
        >For BS answers I'll admit I got less generous as the hours went on. What do you think is the cause of this? Do you become more cynical (and less generous) because you’ve seen so many BS answers previously? Is it just that getting fatigued makes you less generous?
        • ihaveajob 3y ago
          When I was a TA in grad school, I noticed the same. Early on I thought some BS answers were at least kind of funny, and I gave them the benefit of the doubt, maybe giving more attention to the parts that were correct. After I saw similar answers later on, the novelty wore off and I was probably less amused, so the inclination to be lenient disappeared. Sometimes I went back to previous decisions if I remembered them, to be fair, but I don't think I always remembered since the volume could be high (grading 80 exams in a row is TEDIOUS).
      • dunham 3y ago
        I was one of many TAs for a large math class in college (pre-calc - think high school math for college students). For uniformity, the prof had the partial credit down to a science - specifying points for getting certain aspects of the problem. For the finals, a few TAs would be assigned to a given page, for uniformity. The fascinating thing was that the distribution of grades was about the same every year. And I had a math prof for analysis who would give negative points for BS answers. You could say “I need X but don’t know how to prove it” in the middle of a proof, but if you made up something that was incorrect, you’d get negative points.
        • 3y ago
    • bumby 3y ago
      As the number of assignments grows, wouldn’t randomization help converge on the more accurate grades (in aggregate)?
      • falseprofit 3y ago
        It would help, but with only a couple dozen courses and most determined by a couple exams it’s not quite a large number.
    • skhunted 3y ago
      For me I grade tests as follows. The stack is created as students turn in the test. I grade the first page in that order. The stack reverses for the second page. So on and so forth. I teach college math. I just cant imagine a system of grading done in alphabetical order.
      • falseprofit 3y ago
        Scanning and grading on a computer can alphabetize them.
        • skhunted 3y ago
          That makes sense. I haven't had people upload assignments for a long time. I'd forgotten that this was a thing.
      • kurthr 3y ago
        I also came here to say this. My only guess is that the alphabetization (by the "learning management system") to make filling the grades into a table "easier" for the computer or for the person handing out the results? Why is it "easier" if the system doesn't have to order them at all, or it could do so by student number (same issue as alphabetical order) or something random, which is the other (non default) option for the "learning management system". I feel like only the most obsessive compulsive humans would have this issue (without computer "help"), as the last thing I wanted to do as a TA was to add another step of ordering all the papers before grading them. I also always reviewed the first few papers I graded after grading the rest to make sure I was being fair, because it was obvious to me that until I saw a representative distribution of answers I couldn't do fair grading/marking.
    • furyofantares 3y ago
      It's a 0.6 gap from top to bottom out of a score of 100. Plus or minus a third of a percent from average. Pretty small effect. But it would add up (or, well, persist - it wouldn't get bigger) if it happens to you for every assignment for every class and that sucks. If there's more than one assignment you can basically erase it by randomizing each separately. If you really care beyond that then randomize for one assignment, flip it for the next, then randomize again for the next etc.
    • davrosthedalek 3y ago
      In my experience, it's not tired/lazy/inattentive, but resignation. You normally have some expectation what students will be able to solve. Typically, these expectations are set too high. That's very common, not only for me, but for pretty much anyone I know. So over the time of grading, one adjusts down the expectations and gives partial credit earlier, for example.
      • deleted 3y ago
        [deleted]
    • cyanydeez 3y ago
      Unfortunately, it's gonna be AI to the "rescue" and the problem is obfuscated.
  • deleted 3y ago
    [deleted]
  • freeopinion 3y ago
    My first thought was, "Who takes the time to sort before grading?" Computers change the world in such incredibly subtle ways. Of course, such subtleties exist without computers. This is just one case where computers make the subtleties more detectable.
• boesboes 3y ago
  This seems related: https://news.ycombinator.com/item?id=39672111 https://news.ycombinator.com/item?id=39672111 As in, order matters
  • m12k 3y ago
    Also https://www.theguardian.com/law/2011/apr/11/judges-lenient-break https://www.theguardian.com/law/2011/apr/11/judges-lenient-b...
  • zeroonetwothree 3y ago
    This looks like one of the classic studies that won’t reproduce. For one thing, the effect size is unreasonably large. 50% more positive words just because of sequence order would be so huge we should be able to notice it anecdotally.
• tokai 3y ago
  >One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still receive lower grades.
  • kibwen 3y ago
    It would be less than ideal, but still an improvement over the current situation as long as the order is re-randomized for every assignment, because at least then you'd only be occasionally disadvantaged rather than consistently disadvantaged.
  • exe34 3y ago
    It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.
    • zeroonetwothree 3y ago
      It won’t average out perfectly. There will still be lucky and unlucky students. Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.
      • furyofantares 3y ago
        Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.
        • zeroonetwothree 3y ago
          Some courses are only graded based on a small number of tests. I actually went to UM and a grade might be something like 30% midterm 60% final 10% homework (obviously different professors have different systems). In that case if you get unlucky just twice on the two tests you basically get the full penalty.
• noodlesUK 3y ago
  At my university, almost all of our marking was pseudonymised. We were assigned a random candidate number at the beginning of each year, and that is what went on our important papers/exams. The less important coursework often didn’t bother with this, and used our student numbers instead, but the general idea was the same. We didn’t put our names on any of our work other than our dissertation (and a few trivial assignments that didn’t impact overall marks). It wasn’t that hard to de-anonymise, but it meant that the system had a bit more integrity. It’s a really straightforward system to implement and I don’t know why it isn’t done more frequently. I also think that our VLE sorted assignments by time of submission rather than any identifier.
  • deleted 3y ago
    [deleted]
  • ghaff 3y ago
    University exams, this probably makes a lot of sense. After all, the exam is the exam and whether a student is well-spoken and actively participates in class shouldn't matter for an exam grade. I'm less convinced that blinded conference proposals are a good idea--an argument I've had with various people. If you know based on past experience that someone will almost certainly hit a home run, I'm less inclined to pick a random person without obvious qualifications for the same topic--although just picking friends of the committee can obviously go too far.
    • wongarsu 3y ago
      You could try to work around that by first grading all anonymized proposals, then grading all potential speakers without knowing their proposal. In the third round you deanonymize and look at the weighted average of the two grades. You probably still need some judgment calls because the combination of speaker and topic can be important. But the score would give you a good base to work of. Maybe you could make it even more impartial by allowing conditional scores in the first two rounds. Like "Jim is a 6, but a 8 if his talk is about molecular biology" or "this Lessons Learnt talk is a 5, but if it's by X, Y or Z it's a 9"
• dotnet00 3y ago
  Yep, I noticed this with myself too when I first did some grading a few months ago. There was also the factor that the ones I graded initially did not make certain mistakes or answered in expected ways, such that when I did encounter unexpected answers/mistakes, I had to go back and rethink the grading on the papers I had graded previously. Eg if someone answered in a way that made me think an answer I considered incorrect was actually less wrong. I only had to deal with a small class, so backtracking was doable and I graded the papers in whatever shuffled up order they were turned in, otherwise there would have definitely been a bias.
  • bee_rider 3y ago
    I especially noticed this when grading programming projects, because it is slightly complicated. I’d either find that: A bug was really common, got to re-evaluate after the first couple times I see it, apparently it is an easy mistake to make. Or, I’d find a new bug that was pretty common, but which I didn’t know about at first. Got to update my tests and re-run everybody. I tended to be really thorough and re-do the whole stack eventually, but it was a real pain. Could have half-assed it of course, but they spend weeks on these things, feel like I owe them honest feedback. It would tend to lead me to “softer” grading as well, if you are lazy and only check for a couple bugs, you might take a large number of points off for each problem. Finding some problems and punishing them harshly is not very fair for those students that randomly hit the bugs you expect. If you find every bug, you can only take a couple points off per bug without tanking everybody’s score.
  • JadeNB 3y ago
    > I only had to deal with a small class, so backtracking was doable and I graded the papers in whatever shuffled up order they were turned in, otherwise there would have definitely been a bias. Grading papers in submission order just introduces a different bias, though. (For what it's worth, I'm in the same boat and I do the same, because I don't trust my ability to give the papers any true random sorting by hand, so I take the very weak randomization that the submission order gives me.)
    • dotnet00 3y ago
• deleted 3y ago
  [deleted]
• underseacables 3y ago
  Anchoring?
• jimmar 3y ago
  Order effects are real. I'm a prof. I notice that the longer I grade, the less motivated I am to take off points and then justify why I took off those points. It's easier just to give points and move on. (And if anybody wants to criticize this, I'll be happy to launch into a diatribe on the psychometric dumpster fire that most assignments and their associated grading scales really are.)
  • dgacmu 3y ago
    Also prof: me too. I'm much more likely to provide comments on the first couple of exams I grade than on the later ones. I've found that gradescope is helpful in this regard, because it at least forces every point assignment to be matched to a rubric item. I don't have data, but I believe it makes our grading a lot more uniform compared to the pre-gradescope days. (This might be easier in grading computer science exams than in more subjective areas, though.)
  • zeroonetwothree 3y ago
    This is the opppsite of the effect they found. I do wonder if there is a big difference depending on grader and the study found some kind of average.
    • jimmar 3y ago
      The article mentions that the paper is under review, but I'm guessing the effect size is small and that individual differences between graders is very substantial. The article states: > The researchers collected available historical data of all programs, students and assignments on Canvas from the fall 2014 semester to the summer 2022 semester. Thousands of students X 8 years X lots of assignments per year and you get a sample size so big that it would be hard not to find statistically significant effects.
• Aldo_MX 3y ago
  Maybe the answer is smaller groups?
• xyst 3y ago
  That 0.6 pt gap over multiple semesters is the difference between graduating with “summa cum laude” or “magma cum laude”
  • zeroonetwothree 3y ago
    It’s 0.6% so it would only be if you happened to drop a letter grade as a result. Like 90.5 -> 89.9. And that would have to happen multiple times to significantly affect your GPA.
• zdw 3y ago
  As someone whose initials are Z and W, I tend to notice alpha sort a lot. Asking a friend whose initials are A and B about this, it's not something they ever noticed. I haven't noticed a grading/ranking difference, but far more frequently I'll hear that "oh, we ran out of item/time/etc. before we got to you", which has made me much more sensitive to issues of planning/organization.
  • StevenXC 3y ago
    Like most inequities, those who are in the benefiting group frequently don't realize that privilege.
    • dustingetz 3y ago
      I for one am glad that I was not born a mosquito, the odds are not in our favor!
    • sdwr 3y ago
      They realize (bring form to, make real) them, but don't realize (understand) them
      • wryoak 3y ago
        I hate how much I love this worthlessly picky comment
        • sdwr 3y ago
          Love me or hate me, you gotta love me! http://sdwr.ca/link/26 http://sdwr.ca/link/26
      • godelski 3y ago
        This reminds me of cliques. I give them the definition: insight everyone can recite but nobody can act upon.
        • 3y ago
• princeb 3y ago
  >“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” Pei said. there is a similar effect found here https://en.wikipedia.org/wiki/Hungry_judge_effect https://en.wikipedia.org/wiki/Hungry_judge_effect
  • tokai 3y ago
    I believe the hungry judge effect has generally been accepted as false.
  • zeroonetwothree 3y ago
    The thing is, it’s unclear why that effect would make you give people lower grades. surely an equally reasonable guess is that less cognitive abilities could make you give higher grades because you don’t notice errors?
    • bee_rider 3y ago
      It depends on what you are doing and how you are grading. I’d try to not take many points off if an error is somehow “really easy to make,” but that depends on my ability to evaluate the difficulty of mistakes.
    • janci 3y ago
      Sometimes you see the result is wrong so you do not give any points initially and then look on the steps and try to find something that looks correct to give at least some points. The willingness to track through every step diminishes with increasing fatigue.
• bigbacaloa 3y ago
  [dead]
• jncfhnb 3y ago
  Most exam grading is not viewing the writing as a whole but rather looking for incidences of specific points to assign credit for. One could imagine an LLM be quite effective at labeling sentences as pertaining to a predefined idea at scale.
• klysm 3y ago
  Job interviews have similar effects
  • 1-6 3y ago
    Order matters a lot but recruiters typically present the highest flyers first and the lower candidates last.
    • ghaff 3y ago
      In my experience, it varies. I've been on interview panels where we just weren't feeling it for a number of candidates and basically told the recruiter to try harder and eventually hit someone who we were "That's who we want. Find a way to make it happen."
• dboreham 3y ago
  Again, if this kind of thing surprises you, read this book: https://www.amazon.com/Knowledge-Illusion-Never-Think-Alone/dp/039918435X https://www.amazon.com/Knowledge-Illusion-Never-Think-Alone/... The human brain is just a fancy ChatGPT with an internal UI that fools itself into believing it is more logical/smart than it actually is.
  • zeroonetwothree 3y ago
    If anything the difference only being 0.6% seems pretty impressive for the brain.
• llm_trw 3y ago
  I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.
  • verdverm 3y ago
    Unlikely. If you talk with anyone who's done grading, this will likely jive with our experience and make us data aware of the outcomes. Like anything, with grading you can get into a flow, and the more you process an assignment, the more answers you've seen and those can change how you grade future answers
    • zeroonetwothree 3y ago
      I really doubt you can notice a 0.6% discrepancy anecdotally. They only detected it in the study because of the massive amount of data they used. Classic confirmation bias.
      • verdverm 3y ago
        Anecdotally, I would go back and adjust grades on individual problems from earlier in the stack. I can very easily notice my own over strictness from early in the stack.
        • 2cynykyl 3y ago
          For sure. I also find I have to update my rubric to give more/less part marks, which also requires going back. It takes about 10-15 papers grades before things settle down.
    • somenameforme 3y ago
      Not really taking a position on this one way or the other, but I would say that "this jives with my experience" is near to being a prerequisite for junk science. Somebody saying something controversial is going to be challenged -- confirming biases is precisely how you peddle junk. For instance the Journal of Personality and Social Psychology [1] is a terrible journal, with a replication success rate in the 20% range. Yet it's ironically well regarded. Both can probably be explained by the exact same phenomena - go read their articles and reads like a stream of bias confirmations for those of a certain ideological orientation -- the same orientation that's clearly widely shared amongst social science researchers. [1] - https://psycnet.apa.org/PsycARTICLES/journal/psp/126/2 https://psycnet.apa.org/PsycARTICLES/journal/psp/126/2
• lqet 3y ago
  I work in academia. When we grade exams, the order of the exams on the stack is the order in which they were collected in the room (people can sit wherever they like). For grading, we are usually 5 people in a single room, and everyone grades a specific exercise for consistency. The exams are getting shuffled heavily, with everyone just grabbing stacks, looking for exams where "their" exercise was not yet graded, and taking them out. So basically, the order in which we grade exams can be considered random. However, I also grade weekly exercise sheets during the semester, and these are committed into a repository, where each student has a folder that... begins with the first letter of their first name. Everyone I have ever worked with acknowledges that you have to shuffle the order in which you grade these submissions each week, for fairness. Several effects come into play: (1) your are usually less tired at the beginning, (2) your mood gets better during the last 2 sheets because you know you are done soon, (3, and crucially) at the beginning, you have not yet seen all the common errors / developed a "feeling" for them, and you might thus miss them in early submissions, but spot them immediately in later submissions. Another alphabetic effect: In elementary school, my name was on top of the list of students in my class. I remember that I often had to do some special job simply because I was the first name on this list (for example, carry a group ticket when we visited some museum, keep track of something, be the first at something where nobody wanted to be the first, with everyone watching, be the first to be graded in PE, again with everyone watching, etc.). As a fairly shy kid, this already annoyed me in first grade.
  • V__ 3y ago
    A teacher friend of mine always goes through his stack twice. Once to correct all mistakes and a second time to write down points. As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is.
    • smogcutter 3y ago
      > As you said, once you have seen all mistakes you know how "bad" of a mistake it actually is. Crucially, this is not quite what the poster said. It’s not about stack ranking students against each other. Say every paper makes the same subtle mistake, and you only notice it halfway through the pile. Unless you go back through them all, you’ll unfairly grade the later entries more harshly.
• 1-6 3y ago
  Let’s just hope parents don’t try to game the system by starting to name their kids AAAi Aung.
  • jen20 3y ago
    Fortunately Bobby is near the front of the alphabet anyway!
  • nsenifty 3y ago
    I'm Indian (in the US) and I've noticed a vast majority of my Indian friends name their kids Aanav, Aanir or Aanvi etc. some of which aren't even words in any Indian language. Now I probably know why.
• largbae 3y ago
  What other popular systems might lead to different outcomes based on sort order? Dating site matches? Your own contact list? Interesting category of problems...
• COGlory 3y ago
  Multiple factors at play here. 1) Rubrics are often defined, but the application of the rubric is by a human. Application will shift as the grader gets a sense of the classes understanding. 2) As you get fatigued while grading, you'll make mistakes, and be less tolerant of others. Especially if you're an overworked adjunct or graduate student. 3) There are probably a lot more last names early in the alphabet so weighting is important. My policy on this when I was a grad student was to publish the rubric, and ask all students to check their grades too.
• analog31 3y ago
  I propose one of the following: 1. Keep the present system of grading by alphabetical order 2. Record the order in which the papers are actually graded When the grading is done, the teacher assigns a point scale (A = 90, B = 80 or whatever) but the computer does a regression fit and removes the bias.
  • 2cynykyl 3y ago
    This is a great idea! Next time I mark a stack of exams I will also note the time of day that the mark was entered. I can then cross-reference this with how long I have been sitting between breaks, since my last meal, etc, etc. Unfortunately I will not have this opportunity until mid-fall 2024.
• nebulous1 3y ago
  I wonder why Helen Wang chose this as a research topic
  • jeegsy 3y ago
    Well spotted!
• 1shooner 3y ago
  This reminds me of an experience I had of just the opposite: tightly-controlled consistency in writing assessments: Almost 20 years ago I worked for a standardized test essay grading service. We graded against all sorts of secondary-level rubrics (not AP, who do their own). These would usually be from 9 - 12 grade, from every US state, and evaluating everything from reading comprehension to subject matter-specific assessment. We'd do weeks long jobs of a single test (e.g. Alabama 9th grade reading proficiency). These usually had at least 3 dimensions, and at least 4 points per dimension. We would go through a week or more of training on a rubric, then another week of 'leveling', where a manager would occasionally bring you aside and talk through why that '3' you gave on a dimension should have been a '2'. By the end of the training, we usually had had enough discussions and encountered enough edge cases to understand the weaknesses/inconsistencies in the rubric (which we had to abide by anyway). Once we were running at full-speed, everything was still double-graded and inconsistent scores were reviewed. Sometimes graders were pulled if they still didn't get the rubric. It was a simultaneously stimulating and very boring job, and most readers were educators themselves. I wonder how long before it disappears completely.
• ryandrake 3y ago
  Maybe related, or maybe not, but I remember when I was in K-12 school back in the 80s and early 90s, they would always seat us physically in the class front-to-back by last name. So the kids with last names starting with A-D or so would always be in front, and the kids with last names starting with U-Z would be in the back. For every class. I remember this because many of my friends had last names "near" my last name since we were always in close proximity to each other. I vaguely remember, by the time we were in high school, there were definitely more high-achieving kids with A-D last names and definitely more of the troublemakers were U-Z. Was it caused by sitting in closer proximity to the teacher and getting more teacher attention? We'll never know because this wasn't an experiment and there wasn't a control group.
  • nsriv 3y ago
    I'm a teacher now, and this made me wince. It's exactly how I've been told by my parents that seating worked for them in school (India, 60s-80s) but their grading was done by semi-anonymous roll numbers.
    • user_7832 3y ago
      Today I'm 99% sure all CBSE board exams (I think equivalent to A-levels?) are randomized heavily. However I did notice the name's alphabetical order effect in school, albeit in a minor way (folks with later letters were less involved in anything a teacher might need a volunteer for).
  • wongarsu 3y ago
    "students who sit closer are more likely to be high achievers" might also be the source of most of the stereotypes of people with glasses. It took me years to realize I'm mildly shortsighted, so the first half of school I chose seats in the front half of the classroom to make reading the blackboard easier. Many of my friends had glasses and preferred to sit up front because their glasses didn't fully correct their vision.
    • Ekaros 3y ago
• m3kw9 3y ago
  But all the wangs and Xiang and Zhu’s still getting high grades
• xmddmx 3y ago
  Is anyone confused by "lower-ranked names"? To me this means A, B, C, but the article says "Wang said students whose surnames start with A, B, C, D or E received a 0.3-point higher grade out of 100 possible points than compared with when they were graded randomly." So I guess "alphabetically lower ranked" means the last letters of the alphabet, not first? Confused.
  • samatman 3y ago
    This is an important observation! The programmer's perspective and the user's perspective aren't always the same, and both need consideration. A user is going to see a list: it starts at the top, and it ends at the bottom. The first fields are higher, the later fields are lower. Of course, if this is a sorted list, the first field will be the "lowest" value, for whatever comparison is used to sort it.
  • ghaff 3y ago
    Yeah, I misunderstood this at first and then was somewhat confused by the comments until I actually clicked through and looked that the post. :-) I can actually believe the effect going in either direction and it's small.
  • pks016 3y ago
    Yes, while grading we divide the students by their last names.
• danilor 3y ago
  Has anybody found this link to this study? Or even the title? I searched the authors in google scholar but I couldn't find it.
  • bmacho 3y ago
    https://ssrn.com/abstract=4603146 https://ssrn.com/abstract=4603146 Not on sci-hub, but downloadable in pdf for me without any issues
    • deleted 3y ago
      [deleted]
• samatman 3y ago
  A computer-based system like this is an opportunity to remove all personal details from an assignment while grading it, it baffles me that this isn't the default. The database could tag every assignment with a UUID4, and present them for grading top-to-bottom in UUID lexical order, without exposing who is being graded in any way. You can't fix fatigue bias, but this would distribute it randomly. It also removes the opportunity for favoritism and hostility, subconscious or otherwise, which is probably more important. Once grading is completed, the assignments are reconnected with students. Give the profs a way to mark assignments with metadata, sometimes they need to talk to a student personally about something, this should be made easy. Grades can't be immutable, professors need discretion in that, but it would leave an audit trail if professors maliciously modified grades (or the opposite). That should be uncommon to begin with, but both professors and students benefit from an audit trail here. A system like this should be used whenever it's practical, and always for high-stakes tests like midterms and finals. Not making a case against oral exams here, just that when it's possible to blind the grading process, it should be.
• stikit 3y ago
  A .3 point difference isn’t going to make a real difference to anyone’s life and is likely a wash when other yet undiscovered biases are in the mix. Unfairness and bias is a critical factor in driving people to extraordinary achievements.
  • wolverine876 3y ago
    > Unfairness and bias is a critical factor in driving people to extraordinary achievements. The evidence is a strong negative correlation between bias and achievement: Extraordinary achievements so disproportionately achieved by people in groups that are not the target of bias. Look at top government officials, SV leaders, Nobel Prize winners, etc etc - mostly white males. The biggest targets of bias in the US, for example - probably women and black people - genrerally get the worst results (in areas where there is discrimination). By contrast, as an example wherever black people aren't subject to bias, such as certain forms of music and certain sports, achievement is extraordinary. Imagine all that talent and drive in other fields.
  • inemesitaffia 3y ago
    It stacks over time
• faitswulff 3y ago
  I wonder if these biases are replicable in LLMs.
• mistrial9 3y ago
  current curricular trends in California include "algebra removed from 8th grade as unfair" (or more extreme rhetoric given) and this week "equity grading for K-8" where there is no D or F given in any subject. These real-life changes combined with something so arbitrary as this one as "news" really give an impression of a collapse of some kind in public education discourse.
• p0w3n3d 3y ago
  Just do name coding. I doubt this happens everywhere on the world
• cm2187 3y ago
  We know there are big disparities of academic success by ethnic group (cf the whole harvard discrimination against asians controversy), and there are also big concentrations of patronyms by ethnic groups (or at the minimum first letters that are more common in one part of the world than another). And on top of that if the university itself discriminates against certain ethnic groups in its recruitment it will reinforce this bias (like if asians students require better grades to get in, it is unsurprising those students that get in perform better than the rest). That would be my best guess for a rationale behind that result.
• prof-dr-ir 3y ago
  Randomizing the grading order just hides the problem at the level of an individual course, but at least it helps in the average. More worrying is when e.g. job candidates are discussed (often in alphabetical order) and people simply tire out near the end of the meeting. When this happens, be sure to suggest taking a break!
• StefanBatory 3y ago
  I have an surname that's alphabetically low. Even at uni amount time I went to class and came out empty-handed as my teacher didn't score my assignement on time (at my uni 90% we have oral discussion about it) and I have to come next week while others don't are way too high.
• redandblack 3y ago
  When I studied engineering in India, we never put our names in the finals at college. Every one gets a exam id and that goes in the answer sheets. Also, it is never your professor who grades you - the answer sheets are collected and lecturers/professors will correct them at the state level across all the engineering colleges in my state. I do not know how it is now as there has been an explosion of colleges in the state. But expect the standardized tests are similarly conducted.
  • user_7832 3y ago
    As far as I know even now it's the same for government universities (eg Delhi/Mumbai Uni). But private unis may just have a few/one profs grade everything.
  • kwhitefoot 3y ago
    A lot of bachelor's degrees these days are awarded on the basis of modules with no finals. For instance when I did a course on C# a few years ago in Norway that was worth 6 points (I got full marks :-) ). If I had done another 29 modules of similar difficulty I would have got 180 points and been awarded a BSc in Computer Science. It's quite different from the way it was when I studied physics in the 1970s when only the final counted. Annual exams only determined whether one was allowed to continue but had no effect on the class of degree that was awarded.
  • deleted 3y ago
    [deleted]
• redandblack 3y ago
  The other benefit for being higher in the alpha order is you get the snow day calls first - 4:30 am, and get to call your friends before school calls them. We were always woken up by my daughter screaming as here friends called her. No such luck for the post-pandemic kids.
• jedberg 3y ago
  This is basically the reason my kids have the last name that they do. My last name starts with E and my wife's with Y. Bucking tradition, she didn't change her name when we got married, so when we had kids we had to decide what name to give them. We opted to hyphenate. Historically, hyphenated last names were [Woman's last name]-[Man's last name]. However, my wife hated that her last name was near the end of the alphabet growing up. We bucked tradition once again and put my name first, so that when sorted alphabetically they would be at the front of the list. Incidentally their first names start with A and B so that they show up at the front when sorted by first name too.
  • throw_pm23 3y ago
    Wow, you really gave your children a headstart there :)
  • alephknoll 3y ago
    [flagged]
    • jedberg 3y ago
      You seem to be irrationally upset about my light-hearted anecdote. I sincerely hope your weekend gets better.
      • alephknoll 3y ago
        It was just a simple observation. You are reading into things too much. There's no need to be so defensive. Your comment hasn't upset me in the slightest ( rationally or irrationally ) and I sincerely hope you weren't offended by mine.
        • macintux 3y ago
          > It must be exhausting being married to a woman who wants to 'buck tradition'. Why didn't she buck tradition and just name your kids 'Aa, Aa', 'Aaa, Aaa', etc and be done with it? Heck why not go all the way and let them go nameless. You managed to combine snarky reductio ad absurdum and a gratuitous attack on his wife in three sentences. Why wouldn't someone be annoyed by that?
• flawsofar 3y ago
  what’s weird is just how long it took to find a statistic like this one
• huffmsa 3y ago
  I had a theory in school that this was the case for presentations too so I always forced myself to go first. No one else to compare me against, and no sitting around getting jittery.
• beryilma 3y ago
  With huge grade inflation in US universities, all students are already getting better grades than they really deserve. The amount of gymnastics that professors do to pass all students is insane. So, no student is really receiving a lower grade.
• levocardia 3y ago
  > Wang said students whose surnames start with A, B, C, D or E received a 0.3-point higher grade out of 100 possible points than compared with when they were graded randomly. Likewise, students with later-in-the-alphabet surnames received a 0.3-point lower grade — creating a 0.6-point gap. The hand-wringing over such a small effect size seems unwarranted. I suspect you would find similar effect sizes for other small interventions, like whether the grading took place during the week or the weekend, or in the morning vs. the evening.
• pavlov 3y ago
  Clearly evidence of anti-Polish bias when all the Zbigniews and Zygmunts and Wojteks get lower grades. (Or just another example of correlation vs. causation in action)
• TrianguloY 3y ago
  As a different but similar situation: I have a first name that is usually at the top when sorted alphabetically. Nowadays it's not a problem anymore, but as a kid I usually received a lot of calls from people that either misclicked or didn't know how to use a phone properly. It turned out it was because I was the first on the phonebook list.
• TrianguloY 3y ago
  I also have the theory that having an app/software starting with A, B, or an "alphabetically first" letter was noticeable in the past. Nowadays things are usually sorted "algorithmically", but it was common for stores to list searches with some alphabetical score, which meant that those apps were usually shown first. Even now, for example, if you go to Play Store and want to know the apps that you had but are not installed, the default sorting is by name.
• shipmaster 3y ago
  My last name starts with a letter at the bottom of the alphabet. I notice this all the time. Anecdote from this year: My son is in a high school class that requires constant input from the teacher on long running projects they have. The teacher reviews the projects alphabetically by surname, about 40% of the time, the teacher never gets to the bottom of the class, and asks the students to find her after school if they have issues. But the nature of the projects definitely requires proactive comments from the teacher. I ask my son to go find the teacher regardless and get a pro-active review, but not all the kids do that, and hence the potential for a lower grade.
• RecycledEle 3y ago
  I can explain why the kids with A names outperform the kids with Z names. As someone whose first and last names are both very early in the alphabet, I was always called on first or second when I was in elementary school and middle school. I always had to be there early. My friend whose name was very late in the alphabet learned he did not have to be ready for the first minute or two of class. He would be standing near the door talking as I was quickly pulling out last night's homework, and I would be marked down for not being ready while he would later be commended for being ready when the teacher called his name. As a teacher, I see that the kids who stand outside the door talking do not do as well as the kids who are there early.
  • deleted 3y ago
    [deleted]
• candrewlee14 3y ago
  Serious unintended consequences of ordering… Reminds me of the hungry judge effect [1] - judges tend to be more harsh before a break and more lenient after. [1] https://en.m.wikipedia.org/wiki/Hungry_judge_effect https://en.m.wikipedia.org/wiki/Hungry_judge_effect
  • thaumasiotes 3y ago
    https://nautil.us/impossibly-hungry-judges-236688/ https://nautil.us/impossibly-hungry-judges-236688/ > we should dismiss this finding, simply because it is impossible. When we interpret how impossibly large the effect size is, anyone with even a modest understanding of psychology should be able to conclude that it is impossible that this data pattern is caused by a psychological mechanism. As psychologists, we shouldn’t teach or cite this finding, nor use it in policy decisions as an example of psychological bias in decision making.
    • SamBam 3y ago
      Odd article. It simply states that the effect size is too big to be believable (it calls it repeatedly "impossible," but it doesn't seem like it can possibly mean "literally impossible" or "mathematically impossible.") It doesn't give any alternative explanations or specific ways the study is wrong. And it links to a rebuttal by the original authors where the responded to a bunch of the suggestions for data error or confounding factors and found that their results remain.
      • thaumasiotes 3y ago
        That is explained in pretty much the section I quoted. The explanation of the effect is given in the article's links. But the article is written specifically to make the point that it should be enough to observe that it isn't possible for the effect to be real. You aren't making a good point when you cite an effect that is obviously nonsense.
• markusde 3y ago
  I noticed this in myself last time I was as a TA. I'd go back and re-grade the first 15 assignments or so to make sure the rules were being applied consistently.
• corimaith 3y ago
  If we changed our policy of exams from discriminative to evaluative, grading bias would be a trivial issue but here we are since we just NEED ways to fit everyone into numbers that we can easily use.
• retrac 3y ago
  Electoral ballots have often listed the candidates in alphabetic order, but some studies have suggested that it gives a small benefit, to the first person listed. [1] Many election authorities, in Canada at least, have shifted to randomizing the order in some way [2]. Some people have even played with alphabetic sort for novelty purposes; a man in Ontario changed his legal name to "Above Znoneofthe" so he would appear last on the ballot as "Znoneofthe, Above". [1] https://electionlab.mit.edu/research/ballot-order-effects https://electionlab.mit.edu/research/ballot-order-effects [2] https://www.cbc.ca/news/canada/british-columbia/vancouver-does-away-with-alphabetical-order-on-election-ballots-1.4696092 https://www.cbc.ca/news/canada/british-columbia/vancouver-do...
  • zeroonetwothree 3y ago
    In the US it’s usually randomised as well
• yencabulator 3y ago
  It seems it would take less time for Instructure, Inc. (makers of the mentioned software) to fix this than it took do this research. Anyone know whether this is happening, and if not why not?
• stevage 3y ago
  Would it be possible to simply accept that this exists and automatically unskew the grades after marking?
• ghghgfdfgh 3y ago
  There's a section of one of the Diary of a Wimpy Kid books that talks about this exact thing. I was reminded of it as soon as I saw the headline. The justification is comes up with is that kids with names at the front of the alphabet sit in the front of the classroom, so they get called on and learn more. It definitely turned some gears in my brain when I first read it as a teen. Here's the relevant page: https://imgur.com/a/6wIx6qg https://imgur.com/a/6wIx6qg
• searealist 3y ago
  Entirely explained by https://en.wikipedia.org/wiki/List_of_common_Chinese_surnames https://en.wikipedia.org/wiki/List_of_common_Chinese_surname...
• diogenescynic 3y ago
  It's the same with applying to jobs. The first applicants have a greater likelihood to get the job. If you're given a list of names... you're just generally more likely to pick something from the top of the list than the bottom.
• dcposch 3y ago
  I bet this correlation goes away if you separate the data by ethnicity.
  • carabiner 3y ago
    Yeah Chen, Cho, and Cohen are up there and would bias results.
    • justrealist 3y ago
      Wang, Zhao, Xi.
• Jovita111 3y ago
  [dead]