5 ms·

“In most machine scoring states, any of the randomly selected essays with wide discrepancies between human and machine scores are referred to another human for

by jakear 7y ago

“In most machine scoring states, any of the randomly selected essays with wide discrepancies between human and machine scores are referred to another human for review”. And “between 5 to 20 percent” of essays are randomly selected for human review. So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower, but are not selected for human review, your education future is systematically fucked and you have no knowledge of why or how to change it. Absolutely reprehensible. Anyone involved in the creation or adoption of these systems should be ashamed.

kazinator 7y ago
The thing is, you could be similarly screwed by a biased human whose grading is not checked by a less biased human. At least the machines offer the following hope: even if unbiased humans are rare among paper-grading teachers, those humans can be used to train the machines, so then bias-free or lower-bias grading becomes more ubiquitous. Basically, the system has the potential for systematically identifying and reducing systematic bias. A computer program can be retrained much more readily than nation-wide army of humans. Humans can be given a lecture on bias, and then they will just return to their ways.
- gibolt 7y ago
  AI has a lot more potential for bias than humans. It depends on the input data which is likely heavily biased based on other data set results like face detection. It will only amplify any small bias present in the data.
  - solveit 7y ago
    So will humans. With AI, the model and training data are auditable and, if necessary, modifiable. You can't audit a person's life history and you can only really modify a grader by replacing them, and it's much less socially damaging to retrain a model than to fire an employee. AI certainly has a lot of potential for bias, but claims that AI bias is somehow worse than good old human bias always seem shoddily supported (Note I'm not claiming it's untrue. Just that it's never been shown to my satisfaction, which is not surprising given how quickly AI is changing. It may well be true.)
    - kazinator 7y ago
      > but claims that AI bias is somehow worse than good old human bias always seem shoddily supported Well, AI bias can combine sampling bias with human bias. Like say we train the AI with the output of only 10 human paper graders, all chosen from the same school district. Due to the sampling bias, that data could create markedly more (or less!) bias than the entire population of human paper-graders. The resulting AI will ideally mimic those 10 humans, though; it shouldn't show more bias than that group. If those 10 are flaring racists, and grade accordingly, the AI will be the same. (In fact, we hope that it will be the same, if the algorithm actually works in mimicking human grading.)
  - Spivak 7y ago
    It's amazing to see how the general opinion of CS people has completely shifted in the last few years from "algorithmic scoring is important in removing the bias from human graders" to the exact opposite.
    - Larrikin 7y ago
      That was the hope, but all the most effective methods suffer from data collection bias and the studies show that makes them worst than implicitly biased humans.
    - kazinator 7y ago
      If we can quantify the bias in the machine, that gives us an opportunity to close the feedback loop and control the bias. The bias comes from the human-generated training data in the first place; the machine isn't introducing its own. For instance, the machine has no inherent concept of disparaging someone's language because it's from an identifiable inner city dialect. If it picks up that bias, at least it will apply it consistently. When we investigate the machine, the machine will not know that it's being investigated and will not try to conceal its bias from us. On the other hand, eliminating bias from humans basically means this: producing a new litter of small humans and teaching them better than their predecessors.
      - gibolt 7y ago
        If...
    - peteradio 7y ago
      It is important but its not ready.
    - deleted 7y ago
      [deleted]
sjg007 7y ago
Weapons of Math Destruction talks about this.
Hitton 7y ago
>So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true[1][2]. [1] https://www.studyinternational.com/news/record-high-numbers-women-outnumbering-men-university-globally/ https://www.studyinternational.com/news/record-high-numbers-... [2] https://www.independent.co.uk/news/world/americas/black-women-become-most-educated-group-in-us-a7063361.html https://www.independent.co.uk/news/world/americas/black-wome...
- mintplant 7y ago
  Did you read the article this thread is discussing? It cites multiple specific results wherein minority and female writers were underscored by the machine graders compared to human ones.
- topologistics 7y ago
  The idea that black folks and females score dramatically lower on anything is a commonly held racist opinion that is foundational for the continuation of institutional oppression. When you assume the worst about people it ends up hindering their personal progress.
  - shkkmo 7y ago
    They do when graded by a computer program as described in the article. Are you disputing the article's veracity? What is your point?
    - deleted 7y ago
      [deleted]
  - garmaine 7y ago
    That's not what the article is saying, at all. There ARE general differences in style and word choice between minority groups or women and the average white male in writing. The corpus of training examples used in making this AI grader are at least biased towards the average white male. When the AI grades an essay harshly, it is not saying "this essay by a black woman is bad writing", it is saying "this essay differes from my training set by an aggregate score of ____", then sorts those results and (maybe) applies a curve. An essay could be different from the reference standard because the standard is an example of good writing and the essay is not. Or it can be different because the author has a cultural, regional, gender, or developed background that imparts a different style than anything in the training corpus. Mistaking the two is very, very bad.
deleted 7y ago
[deleted]
IanSanders 7y ago
>Anyone involved in the creation or adoption of these systems should be ashamed That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.

by jakear 7y ago

kazinator 7y ago

The thing is, you could be similarly screwed by a biased human whose grading is not checked by a less biased human. At least the machines offer the following hope: even if unbiased humans are rare among paper-grading teachers, those humans can be used to train the machines, so then bias-free or lower-bias grading becomes more ubiquitous. Basically, the system has the potential for systematically identifying and reducing systematic bias. A computer program can be retrained much more readily than nation-wide army of humans. Humans can be given a lecture on bias, and then they will just return to their ways.

gibolt 7y ago
AI has a lot more potential for bias than humans. It depends on the input data which is likely heavily biased based on other data set results like face detection. It will only amplify any small bias present in the data.
- solveit 7y ago
  So will humans. With AI, the model and training data are auditable and, if necessary, modifiable. You can't audit a person's life history and you can only really modify a grader by replacing them, and it's much less socially damaging to retrain a model than to fire an employee. AI certainly has a lot of potential for bias, but claims that AI bias is somehow worse than good old human bias always seem shoddily supported (Note I'm not claiming it's untrue. Just that it's never been shown to my satisfaction, which is not surprising given how quickly AI is changing. It may well be true.)
  - kazinator 7y ago
    > but claims that AI bias is somehow worse than good old human bias always seem shoddily supported Well, AI bias can combine sampling bias with human bias. Like say we train the AI with the output of only 10 human paper graders, all chosen from the same school district. Due to the sampling bias, that data could create markedly more (or less!) bias than the entire population of human paper-graders. The resulting AI will ideally mimic those 10 humans, though; it shouldn't show more bias than that group. If those 10 are flaring racists, and grade accordingly, the AI will be the same. (In fact, we hope that it will be the same, if the algorithm actually works in mimicking human grading.)
- Spivak 7y ago
  It's amazing to see how the general opinion of CS people has completely shifted in the last few years from "algorithmic scoring is important in removing the bias from human graders" to the exact opposite.
  - Larrikin 7y ago
    That was the hope, but all the most effective methods suffer from data collection bias and the studies show that makes them worst than implicitly biased humans.
  - kazinator 7y ago
    If we can quantify the bias in the machine, that gives us an opportunity to close the feedback loop and control the bias. The bias comes from the human-generated training data in the first place; the machine isn't introducing its own. For instance, the machine has no inherent concept of disparaging someone's language because it's from an identifiable inner city dialect. If it picks up that bias, at least it will apply it consistently. When we investigate the machine, the machine will not know that it's being investigated and will not try to conceal its bias from us. On the other hand, eliminating bias from humans basically means this: producing a new litter of small humans and teaching them better than their predecessors.
    - gibolt 7y ago
      If...
  - peteradio 7y ago
    It is important but its not ready.
  - deleted 7y ago
    [deleted]

sjg007 7y ago

Weapons of Math Destruction talks about this.

Hitton 7y ago

>So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true[1][2]. [1] https://www.studyinternational.com/news/record-high-numbers-women-outnumbering-men-university-globally/ https://www.studyinternational.com/news/record-high-numbers-... [2] https://www.independent.co.uk/news/world/americas/black-women-become-most-educated-group-in-us-a7063361.html https://www.independent.co.uk/news/world/americas/black-wome...

mintplant 7y ago
Did you read the article this thread is discussing? It cites multiple specific results wherein minority and female writers were underscored by the machine graders compared to human ones.
topologistics 7y ago
The idea that black folks and females score dramatically lower on anything is a commonly held racist opinion that is foundational for the continuation of institutional oppression. When you assume the worst about people it ends up hindering their personal progress.
- shkkmo 7y ago
  They do when graded by a computer program as described in the article. Are you disputing the article's veracity? What is your point?
  - deleted 7y ago
    [deleted]
- garmaine 7y ago
  That's not what the article is saying, at all. There ARE general differences in style and word choice between minority groups or women and the average white male in writing. The corpus of training examples used in making this AI grader are at least biased towards the average white male. When the AI grades an essay harshly, it is not saying "this essay by a black woman is bad writing", it is saying "this essay differes from my training set by an aggregate score of ____", then sorts those results and (maybe) applies a curve. An essay could be different from the reference standard because the standard is an example of good writing and the essay is not. Or it can be different because the author has a cultural, regional, gender, or developed background that imparts a different style than anything in the training corpus. Mistaking the two is very, very bad.

deleted 7y ago

[deleted]

IanSanders 7y ago

>Anyone involved in the creation or adoption of these systems should be ashamed That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.