4 ms·

What is the racial bias encoded in the SAT? Is it possible for a world to exist where an objective test showed different performance for different racial subgro

What is the racial bias encoded in the SAT? Is it possible for a world to exist where an objective test showed different performance for different racial subgroups? I'm not saying we are in this world. But it looks to me that if we were in this world people would just complain that all tests were biased.

joshuamorton 7y ago
> Is it possible for a world to exist where an objective test showed different performance for different racial subgroups? Maybe. But I think that world would be very different than ours today. > What is the racial bias encoded in the SAT? I can't find a source for this now, but I recall reading that if you give versions of reading tests to black students that use AAVE instead of common English, the racial gap in reading performance mostly disappears. In other words, something like the SAT measures, at best a sort of confirmative intelligence and not an objective one (if that's even a thing. I'd suggest that intelligence is always relative to society). So institutions looking to influence society can't simply use a test that measures in part how well one conforms.
- DuskStar 7y ago
  How does that get reconciled with the consistently reproduced [0] ~15 point racial IQ gap in the US, since I was under the impression that the SAT was mostly a shitty IQ test only valid for HS graduates? I find it hard to believe that a test like Raven's Progressive Matrices [1] would discriminate against people more familiar with AAVE, even if the others would. 0: https://en.wikipedia.org/wiki/Race_and_intelligence#United_States_test_scores https://en.wikipedia.org/wiki/Race_and_intelligence#United_S... 1: https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices
  - joshuamorton 7y ago
    Indeed, AAVE is just sort of an egregious example, for the pattern matching tests, men score better than women. Is this because men are smarter than women? Or is it because performance on Raven's progressive matrix tests are essentially spacial reasoning tests[1], and thus. Similarly, western women have better reasoning about color than western men (I can't remember exactly where I found this, but it was probably an article about the World Color Survey controversy), and western cultured-people score better in pattern recognition tests compared to people from cultures that have fewer color words. Such motifs are similar: people from cultures whose languages don't express numbers (1, 2, many) or express time[2] differently have more or less trouble reasoning in those contexts. Even formal logic can be culturally tied[3] (and that article also mentions certain forms of spacial reasoning being culturally tied too), which is weird (you can think of this as an extension to the Sapir-Whorf hypothesis[4]: language and culture influence how we think, and thus trying to judge intelligence across-culture will run into...issues). tl;dr: Fairly judging "general intelligence" in any way is difficult. We continually find that we're really bad at it. Something like IQ is useful in certain contexts: an individual with a much higher IQ is probably going to perform better in US life than someone with a comparatively lower IQ, but when you try to make strong conclusions about intelligence, especially about group-levels of intelligence, from something like IQ scores, you're veering off into pseudoscience. [1]: (pdf) https://s3.amazonaws.com/academia.edu.documents/45855297/Sex_differential_item_functioning_in_the20160522-30824-bpwk8l.pdf?response-content-disposition=inline%3B%20filename%3DSex_differential_item_functioning_in_the.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200112T233026Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2e26c804d400158360ffd5ac919c9dba1978e68aa7d8ab0601024b2b73c9f751 https://s3.amazonaws.com/academia.edu.documents/45855297/Sex... [2]: https://www.ncbi.nlm.nih.gov/pubmed/21564254 https://www.ncbi.nlm.nih.gov/pubmed/21564254 [3]: https://www.nytimes.com/2000/08/08/science/how-culture-molds-habits-of-thought.html https://www.nytimes.com/2000/08/08/science/how-culture-molds... [4]: https://www.youtube.com/watch?v=u6eXw0AAKZ8 https://www.youtube.com/watch?v=u6eXw0AAKZ8
    - DuskStar 7y ago
      If I'm understanding your objection correctly, it'd be summarized as "objective cross-cultural tests are impossible", right? And at least to me... Spatial reasoning, color comparison and differentiation, and formal logic would all be parts of general intelligence. Penalizing someone for being innumerate might not capture a person's theoretical intelligence if raised in a different culture, but that doesn't make it inaccurate as a test of how they are now.
      - joshuamorton 7y ago
        I'm not sure I'd go as far as saying impossible (perhaps for true objectivity, yes?) but certainly really, really, really difficult. And as a corollary, for things that aren't designed explicitly as fair-ish cross cultural tests, we really shouldn't go about trying to draw cross-cultural conclusions. I'll be fully transparent, this does make the question of "well how can we judge people without insisting on equality of outcome" rather tricky. I resolve this personally by just concluding that the precise judgement of people such that we'd need a good test like that is rarely actually important in day to day life. In other words, I'm okay with wider error bars for the reduction of implicit bias.
        DuskStar 7y ago
        > I'll be fully transparent, this does make the question of "well how can we judge people without insisting on equality of outcome" rather tricky. I resolve this personally by just concluding that the precise judgement of people such that we'd need a good test like that is rarely actually important in day to day life. In other words, I'm okay with wider error bars for the reduction of implicit bias. Isn't this just "I'm OK with error bars in the direction I favor", which happens to be equity? Unless you're saying you're OK with accepting the tests we have, with error bars that might err too far on the side of "other racial/cultural groups are less intelligent".
        joshuamorton 7y ago
        > Isn't this just "I'm OK with error bars in the direction I favor", which happens to be equity? Unless you're saying you're OK with accepting the tests we have, with error bars that might err too far on the side of "other racial/cultural groups are less intelligent". Not exactly: Precisely, I'd prefer error bars that are equally sized for everyone, instead of error bars that differ based on cultural factors, especially if we aren't capable of acknowledging the cultural disparity in distribution of those error bars (and IME, people generally aren't). In other words, if you have a test that has an error of X when comparing 2 white kids, and an error of X when comparing two black kids, but an error of 3X when comparing a white kid to a black kid, you don't have a test with a precision of X, you have a test with an error of 3X (or worse: you have a test with a precision of X, but that consistently biases in favor of one group). If on the other hand we have a test that has errors of 2X, but has that across cultural boundaries, that's a better test and we should use it instead. This is because the errors are randomly distributed across the population (so anyone has the same chance of being positively or negatively impacted by the test's imprecision) instead of culturally tied. If we're willing to acknowledge the different distributions of error, then maybe we can make do, but consistent problems with things like affirmative action suggest that, at a macro level, we aren't.

joshuamorton 7y ago

> Is it possible for a world to exist where an objective test showed different performance for different racial subgroups? Maybe. But I think that world would be very different than ours today. > What is the racial bias encoded in the SAT? I can't find a source for this now, but I recall reading that if you give versions of reading tests to black students that use AAVE instead of common English, the racial gap in reading performance mostly disappears. In other words, something like the SAT measures, at best a sort of confirmative intelligence and not an objective one (if that's even a thing. I'd suggest that intelligence is always relative to society). So institutions looking to influence society can't simply use a test that measures in part how well one conforms.

DuskStar 7y ago
How does that get reconciled with the consistently reproduced [0] ~15 point racial IQ gap in the US, since I was under the impression that the SAT was mostly a shitty IQ test only valid for HS graduates? I find it hard to believe that a test like Raven's Progressive Matrices [1] would discriminate against people more familiar with AAVE, even if the others would. 0: https://en.wikipedia.org/wiki/Race_and_intelligence#United_States_test_scores https://en.wikipedia.org/wiki/Race_and_intelligence#United_S... 1: https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices
- joshuamorton 7y ago
  Indeed, AAVE is just sort of an egregious example, for the pattern matching tests, men score better than women. Is this because men are smarter than women? Or is it because performance on Raven's progressive matrix tests are essentially spacial reasoning tests[1], and thus. Similarly, western women have better reasoning about color than western men (I can't remember exactly where I found this, but it was probably an article about the World Color Survey controversy), and western cultured-people score better in pattern recognition tests compared to people from cultures that have fewer color words. Such motifs are similar: people from cultures whose languages don't express numbers (1, 2, many) or express time[2] differently have more or less trouble reasoning in those contexts. Even formal logic can be culturally tied[3] (and that article also mentions certain forms of spacial reasoning being culturally tied too), which is weird (you can think of this as an extension to the Sapir-Whorf hypothesis[4]: language and culture influence how we think, and thus trying to judge intelligence across-culture will run into...issues). tl;dr: Fairly judging "general intelligence" in any way is difficult. We continually find that we're really bad at it. Something like IQ is useful in certain contexts: an individual with a much higher IQ is probably going to perform better in US life than someone with a comparatively lower IQ, but when you try to make strong conclusions about intelligence, especially about group-levels of intelligence, from something like IQ scores, you're veering off into pseudoscience. [1]: (pdf) https://s3.amazonaws.com/academia.edu.documents/45855297/Sex_differential_item_functioning_in_the20160522-30824-bpwk8l.pdf?response-content-disposition=inline%3B%20filename%3DSex_differential_item_functioning_in_the.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200112T233026Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2e26c804d400158360ffd5ac919c9dba1978e68aa7d8ab0601024b2b73c9f751 https://s3.amazonaws.com/academia.edu.documents/45855297/Sex... [2]: https://www.ncbi.nlm.nih.gov/pubmed/21564254 https://www.ncbi.nlm.nih.gov/pubmed/21564254 [3]: https://www.nytimes.com/2000/08/08/science/how-culture-molds-habits-of-thought.html https://www.nytimes.com/2000/08/08/science/how-culture-molds... [4]: https://www.youtube.com/watch?v=u6eXw0AAKZ8 https://www.youtube.com/watch?v=u6eXw0AAKZ8
  - DuskStar 7y ago
    If I'm understanding your objection correctly, it'd be summarized as "objective cross-cultural tests are impossible", right? And at least to me... Spatial reasoning, color comparison and differentiation, and formal logic would all be parts of general intelligence. Penalizing someone for being innumerate might not capture a person's theoretical intelligence if raised in a different culture, but that doesn't make it inaccurate as a test of how they are now.
    - joshuamorton 7y ago
      I'm not sure I'd go as far as saying impossible (perhaps for true objectivity, yes?) but certainly really, really, really difficult. And as a corollary, for things that aren't designed explicitly as fair-ish cross cultural tests, we really shouldn't go about trying to draw cross-cultural conclusions. I'll be fully transparent, this does make the question of "well how can we judge people without insisting on equality of outcome" rather tricky. I resolve this personally by just concluding that the precise judgement of people such that we'd need a good test like that is rarely actually important in day to day life. In other words, I'm okay with wider error bars for the reduction of implicit bias.
      - DuskStar 7y ago
        > I'll be fully transparent, this does make the question of "well how can we judge people without insisting on equality of outcome" rather tricky. I resolve this personally by just concluding that the precise judgement of people such that we'd need a good test like that is rarely actually important in day to day life. In other words, I'm okay with wider error bars for the reduction of implicit bias. Isn't this just "I'm OK with error bars in the direction I favor", which happens to be equity? Unless you're saying you're OK with accepting the tests we have, with error bars that might err too far on the side of "other racial/cultural groups are less intelligent".
        joshuamorton 7y ago
        > Isn't this just "I'm OK with error bars in the direction I favor", which happens to be equity? Unless you're saying you're OK with accepting the tests we have, with error bars that might err too far on the side of "other racial/cultural groups are less intelligent". Not exactly: Precisely, I'd prefer error bars that are equally sized for everyone, instead of error bars that differ based on cultural factors, especially if we aren't capable of acknowledging the cultural disparity in distribution of those error bars (and IME, people generally aren't). In other words, if you have a test that has an error of X when comparing 2 white kids, and an error of X when comparing two black kids, but an error of 3X when comparing a white kid to a black kid, you don't have a test with a precision of X, you have a test with an error of 3X (or worse: you have a test with a precision of X, but that consistently biases in favor of one group). If on the other hand we have a test that has errors of 2X, but has that across cultural boundaries, that's a better test and we should use it instead. This is because the errors are randomly distributed across the population (so anyone has the same chance of being positively or negatively impacted by the test's imprecision) instead of culturally tied. If we're willing to acknowledge the different distributions of error, then maybe we can make do, but consistent problems with things like affirmative action suggest that, at a macro level, we aren't.