7 ms·

Many here are in the throws of this "data mythology"... It is trivial for algorithms to introduce novel racism: consider processing text analysing minority cau

Many here are in the throws of this "data mythology"... It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. A machine if it should have learned anything at all from this text, should learn to associate minority terms with virtuous political projects; it should have acquired an understanding of the many types of associative relation: needing, wanting, opposing, supporting, trusting, advocating... Yet a machine performing an action based on statistical associations will not learn these; and so, will act prejudicially. It will merely expect terms to co-occur, wholly unable to determine why. We expect charity workers to be greedy? No, we expect them to morally oppose greed. We expect movies to smell like candy? No, we expect them to be sold together. (And so on.) It is somewhat alarming that statistical associations in terms in historical texts are being seen as characteristic of human communication, meaning, belief, association, ethics, action. I think it has led here to a great moral oversight: that absent the ability to understand text, machines here are introducing novel prejudice where none existed prior.

davidgerard 5y ago
> they are not racist "having racist effects" counts as a perfectly valid usage of the word. This is a cheap sophistry: 1. It's only racist if it was intended to be. 2. You can't read minds. 3. Therefore, you can't say "this is racist." This isn't a useful usage of the term, except for attempting to deflect people from calling out racist effects.
- concordDance 5y ago
  I really wish the word "racist" had a single meaning.
  - user-the-name 5y ago
    It is the norm, not the exception, for words to have multiple meanings.
  - MichaelGroves 5y ago
    Any word with powerful meaning in one context will inevitably be repurposed in other contexts by people who want to borrow some of that power for their own uses. Here is an example: https://hn.algolia.com/?q=democratize https://hn.algolia.com/?q=democratize
  - a9h74j 5y ago
    In general that won't happen, but one can do close readings of presumed valid arguments, and spot cases where equivocation (e.g. bringing in different meanings of a word within different premises) can invalidate a superficially good argument.
- nine_k 5y ago
  Would you call a virus that predominantly infects a particular ethnic group "racist"? Would you call alcohol or milk "racist" because Europeans, North Siberian and Alaskan peoples, and East-Asian peoples react differently to them? I'd say something like "race-biased" or "race-sensitive" would be a more proper term. You can't read minds, but you can read laws. Laws state the intent expressly, and also state the policy expressly. This is why "racist" can be very exactly applied to some laws (like those mandating segregation, different rights, etc, depending on the race). So, to my mind, using the word "racist" to denote intentional action makes enough sense.
  - davidgerard 5y ago
    I would call the thing we're actually talking about, a machine set up and administered by humans and applied by humans, something that involves human agency. "Racist" is a perfectly applicable word for its effects when applied by the humans applying it. Some unrelated hypothetical about things not involving human agency is irrelevant to the question, and is functionally just another attempted deflection.
    - deleted 5y ago
      [deleted]
    - bsanr2 5y ago
      I'm confused and slightly alarmed by the insistence on doing everything possible to scrub the prospect of human agency from matters that affected by and that effect humans. If we see some social ill, shouldn't be combing the systems that are related to thei existence for the flaws that let those ills happen, instead of just pointing at the system and saying, "Well, a human didn't make that specific decision, so it must be objective and fair." It's high-tech ventriloquism.
      - SpicyLemonZest 5y ago
        I think you're misunderstanding the concern. As seen in stories like the x-ray race detection thing (https://www.wired.com/story/these-algorithms-look-x-rays-detect-your-race/ https://www.wired.com/story/these-algorithms-look-x-rays-det...), there's a lot of people in the AI field who believe that any correlation with race in a model is a "bias" which should be corrected. If a radiology AI model stumbles across some unknown condition like sickle cell anemia with a strong racial correlation, I think we should learn about it, and I worry that under the kinds of practices the source article suggests it would just be bias adjusted away.
- dabbledash 5y ago
  I think when the person you’re responding to says “they are not racist” he is referring to the texts being analyzed, not the model. I.e, “your model can take texts about or against racism as inputs and from these texts form an association between negative sentiment and certain races.”
- deleted 5y ago
  [deleted]
- jhgb 5y ago
  Computers are not sapient, and therefore can't be prejudiced (which would be a necessity for them to be racist) because prejudice is a feature of sapient entities. That would be like calling an elevator that had a malfunction and fell to the ground floor killing everyone inside "a murderer".
toxik 5y ago
I don't think you've fully understood how far "statistical associations", as your derisively call it, can get you in understanding text. Modern language models absolutely make both sentiment and semantic distinctions, and they would not complete a sentence like "The movie was like" with "candy" simply because the word candy exists in texts simultaneously with the word "movie". That model would be completely useless.
- sokoloff 5y ago
  Then again, we have things like: https://www.google.com/search?q=when+did+neil+armstrong+set+foot+on+mars https://www.google.com/search?q=when+did+neil+armstrong+set+...
  - TchoBeer 5y ago
    This isn't because Google's language model is stupid, but because it's trying to give you what it thinks you meant rather than give you precisely what you put in the search bar.
    - kbenson 5y ago
      The problem here is that unlike the "did you mean to search for" text that makes it obvious that they're showing you what they thought you were looking for in regular search results, they're not doing something like that here, and it's unclear if that's because their model is so loose that they don't actually have knowledge that this is what they're doing (i.e. it's baked in), or if it's just an oversight. The former is a problem because it promulgates incorrect information and provides veracity to mistakes and misconceptions, and if it's the latter, why wasn't it fixed long ago?
- rsfern 5y ago
  I agree with your general point, but I think “the movie was like candy“ could be a perfectly reasonable simile to make. Maybe the movie was flashy but had no substance? I’m not sure if I expect modern language models to work at that level of abstraction though? I guess you need a larger generated fragment to assess if the hypothetical sentence was statistical nonsense or part of some larger conceptual sequence
mattmcknight 5y ago
Sequence models go beyond co-occurrence.
darawk 5y ago
I think your example is obscuring the issue here a little bit, because you didn't really describe a targeted learning problem: > It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. Yes, negative words will associate with black people in such literature. They will also likely associate with white people. That is, emotionally intense language will associate with race in literature related to racial justice issues. That is a perfectly valid inference. What you seem to be implying is that the model will learn the propositional statement "black people are bad", but your example is just about associations, not propositional assertions. Associations are not assignments of quality, they are just that: associations. Such a model would correctly learn such associations and not be in any sense biased, morally or otherwise, because the model is not making decisions or evaluations of people or moral objects. The notion of bias usually talked about in ML is in the context of either: 1. Making statistical decisions (e.g. granting a loan, or predicting criminal recidivism) 2. Providing a service of some kind (e.g. facial recognition in a camera/video, ad targeting) Talking about bias in these cases brings the issue into focus, because there is a morally relevant objective function in these cases. And my point is that in these cases, standard untrained ML models are morally neutral. They come to the table with no preconceptions. In a society without racism, they will not learn racism. In a society without sexism, they will not learn sexism. They only reflect what we feed them. Is it reasonable for me to describe my mirror as biased if I don't like how I look? In theory, I could build another "mirror" that reflects me the way I want. It just wouldn't comport with how I actually look, and we would no longer call that object a mirror. I want to step back for a moment and say that I think we probably agree on the object-level facts here. I believe you can correct the morally biased ML-output problem at the model level or the data level equally well. I'm mostly objecting to the linguistic utility of locating the moral bias equally in each of them. I think that kind of relativism is just not a very useful way to describe things. I think for some reason this idea has formed that saying the bias is in X means we have to correct it in X, but I think it's that view that's mistaken. We can and should correct it wherever we feel is most efficient and effective.
lvxferre 5y ago
>that absent the ability to understand text, machines here are introducing novel prejudice where none existed prior. Kinda off-topic, but what haunts me is that this is equally true for people. People who don't understand what they read are always introducing new prejudices, where none existed prior.

davidgerard 5y ago

> they are not racist "having racist effects" counts as a perfectly valid usage of the word. This is a cheap sophistry: 1. It's only racist if it was intended to be. 2. You can't read minds. 3. Therefore, you can't say "this is racist." This isn't a useful usage of the term, except for attempting to deflect people from calling out racist effects.

concordDance 5y ago
I really wish the word "racist" had a single meaning.
- user-the-name 5y ago
  It is the norm, not the exception, for words to have multiple meanings.
- MichaelGroves 5y ago
  Any word with powerful meaning in one context will inevitably be repurposed in other contexts by people who want to borrow some of that power for their own uses. Here is an example: https://hn.algolia.com/?q=democratize https://hn.algolia.com/?q=democratize
- a9h74j 5y ago
  In general that won't happen, but one can do close readings of presumed valid arguments, and spot cases where equivocation (e.g. bringing in different meanings of a word within different premises) can invalidate a superficially good argument.
nine_k 5y ago
Would you call a virus that predominantly infects a particular ethnic group "racist"? Would you call alcohol or milk "racist" because Europeans, North Siberian and Alaskan peoples, and East-Asian peoples react differently to them? I'd say something like "race-biased" or "race-sensitive" would be a more proper term. You can't read minds, but you can read laws. Laws state the intent expressly, and also state the policy expressly. This is why "racist" can be very exactly applied to some laws (like those mandating segregation, different rights, etc, depending on the race). So, to my mind, using the word "racist" to denote intentional action makes enough sense.
- davidgerard 5y ago
  I would call the thing we're actually talking about, a machine set up and administered by humans and applied by humans, something that involves human agency. "Racist" is a perfectly applicable word for its effects when applied by the humans applying it. Some unrelated hypothetical about things not involving human agency is irrelevant to the question, and is functionally just another attempted deflection.
  - deleted 5y ago
    [deleted]
  - bsanr2 5y ago
    I'm confused and slightly alarmed by the insistence on doing everything possible to scrub the prospect of human agency from matters that affected by and that effect humans. If we see some social ill, shouldn't be combing the systems that are related to thei existence for the flaws that let those ills happen, instead of just pointing at the system and saying, "Well, a human didn't make that specific decision, so it must be objective and fair." It's high-tech ventriloquism.
    - SpicyLemonZest 5y ago
      I think you're misunderstanding the concern. As seen in stories like the x-ray race detection thing (https://www.wired.com/story/these-algorithms-look-x-rays-detect-your-race/ https://www.wired.com/story/these-algorithms-look-x-rays-det...), there's a lot of people in the AI field who believe that any correlation with race in a model is a "bias" which should be corrected. If a radiology AI model stumbles across some unknown condition like sickle cell anemia with a strong racial correlation, I think we should learn about it, and I worry that under the kinds of practices the source article suggests it would just be bias adjusted away.
dabbledash 5y ago
I think when the person you’re responding to says “they are not racist” he is referring to the texts being analyzed, not the model. I.e, “your model can take texts about or against racism as inputs and from these texts form an association between negative sentiment and certain races.”
deleted 5y ago
[deleted]
jhgb 5y ago
Computers are not sapient, and therefore can't be prejudiced (which would be a necessity for them to be racist) because prejudice is a feature of sapient entities. That would be like calling an elevator that had a malfunction and fell to the ground floor killing everyone inside "a murderer".

toxik 5y ago

I don't think you've fully understood how far "statistical associations", as your derisively call it, can get you in understanding text. Modern language models absolutely make both sentiment and semantic distinctions, and they would not complete a sentence like "The movie was like" with "candy" simply because the word candy exists in texts simultaneously with the word "movie". That model would be completely useless.

sokoloff 5y ago
Then again, we have things like: https://www.google.com/search?q=when+did+neil+armstrong+set+foot+on+mars https://www.google.com/search?q=when+did+neil+armstrong+set+...
- TchoBeer 5y ago
  This isn't because Google's language model is stupid, but because it's trying to give you what it thinks you meant rather than give you precisely what you put in the search bar.
  - kbenson 5y ago
    The problem here is that unlike the "did you mean to search for" text that makes it obvious that they're showing you what they thought you were looking for in regular search results, they're not doing something like that here, and it's unclear if that's because their model is so loose that they don't actually have knowledge that this is what they're doing (i.e. it's baked in), or if it's just an oversight. The former is a problem because it promulgates incorrect information and provides veracity to mistakes and misconceptions, and if it's the latter, why wasn't it fixed long ago?
rsfern 5y ago
I agree with your general point, but I think “the movie was like candy“ could be a perfectly reasonable simile to make. Maybe the movie was flashy but had no substance? I’m not sure if I expect modern language models to work at that level of abstraction though? I guess you need a larger generated fragment to assess if the hypothetical sentence was statistical nonsense or part of some larger conceptual sequence

mattmcknight 5y ago

Sequence models go beyond co-occurrence.

darawk 5y ago

I think your example is obscuring the issue here a little bit, because you didn't really describe a targeted learning problem: > It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. Yes, negative words will associate with black people in such literature. They will also likely associate with white people. That is, emotionally intense language will associate with race in literature related to racial justice issues. That is a perfectly valid inference. What you seem to be implying is that the model will learn the propositional statement "black people are bad", but your example is just about associations, not propositional assertions. Associations are not assignments of quality, they are just that: associations. Such a model would correctly learn such associations and not be in any sense biased, morally or otherwise, because the model is not making decisions or evaluations of people or moral objects. The notion of bias usually talked about in ML is in the context of either: 1. Making statistical decisions (e.g. granting a loan, or predicting criminal recidivism) 2. Providing a service of some kind (e.g. facial recognition in a camera/video, ad targeting) Talking about bias in these cases brings the issue into focus, because there is a morally relevant objective function in these cases. And my point is that in these cases, standard untrained ML models are morally neutral. They come to the table with no preconceptions. In a society without racism, they will not learn racism. In a society without sexism, they will not learn sexism. They only reflect what we feed them. Is it reasonable for me to describe my mirror as biased if I don't like how I look? In theory, I could build another "mirror" that reflects me the way I want. It just wouldn't comport with how I actually look, and we would no longer call that object a mirror. I want to step back for a moment and say that I think we probably agree on the object-level facts here. I believe you can correct the morally biased ML-output problem at the model level or the data level equally well. I'm mostly objecting to the linguistic utility of locating the moral bias equally in each of them. I think that kind of relativism is just not a very useful way to describe things. I think for some reason this idea has formed that saying the bias is in X means we have to correct it in X, but I think it's that view that's mistaken. We can and should correct it wherever we feel is most efficient and effective.

lvxferre 5y ago

>that absent the ability to understand text, machines here are introducing novel prejudice where none existed prior. Kinda off-topic, but what haunts me is that this is equally true for people. People who don't understand what they read are always introducing new prejudices, where none existed prior.