7 ms·
The paper and title implies it's absorbing these stereotypes from humans. I think there is another explanation. Remember these models are trained on a dataset o
by Houshalter 10y ago
The paper and title implies it's absorbing these stereotypes from humans. I think there is another explanation. Remember these models are trained on a dataset of news or Wikipedia articles. And it's 'goal' is to find vectors that predict what contexts words are more likely to appear in.
So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".
And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.
This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.
- andreasvc 10y agoI think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.
- yummyfajitas 10y agoThe parent's point is that they may not be absorbing stereotypes from humans at all. They may be generating accurate beliefs about the world from text representations of the world.
- rmxt 10y agoSo, "plants are pleasant" or "insects are unpleasant" are universal truths?
- Houshalter 10y agoQuite possibly. Words relating to insects will occur in news articles about malaria, zika, crop destruction, etc. Words relating to plants might occur in articles about arbor day, spring time, environmentalism, etc.
- pharrington 10y agoAn exercise: Words relating to insects will occur in news articles about environmentalism, crop production, rituals of rebirth, etc. Words relating to plants might occur in articles about crop destruction, the international drug trade, people getting poisoned, etc. rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assertion against the discovered sentiments' universal truth.
- Houshalter 10y agoBut it is a universal truth that humans generally find plants pleasant and insects unpleasant. And the word "pleasant" is entirely based on human preferences after all.
- pharrington 10y agoWhat I'm probably missing indeed is that scoping of universality to humans. Lately I've been trying to be more explicit in my written communications in an attempt to understand both the limits of my knowledge and perceptions and the limits of the sources of information that I digest. Is suggesting that pleasantness is a sentiment that's not unique to humans really that controversial? super late edit: it's specifically flowers, not plants, that people are biased towards finding pleasant
- mcguire 10y ago"...it is a universal truth that humans generally find plants pleasant..." Ah, but those exceptions are really unpleasant.
- 10y ago
- pharrington 10y agoThe machines are generating accurate semantic graphs of human generated text representations of human observations.
- scribu 10y agoThe GP is saying that the bias isn't an attribute of the Wikipedia text, but of reality. If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?
- andreasvc 10y agoAgain, from humans and from reality is not different. Whether it is desirable to avoid stereotypes depends on your values.
- scribu 10y agoWhat humans think they know is not actually reality - fine. But then what's the difference between a fact and a stereotype, in your opinion?
- andreasvc 10y agoA fact is a statement corresponding with reality, a stereotype is a belief/generalization deemed harmful/undesirable.
- pharrington 10y agoNot answering for the parent: a fact is any instantaneous snapshot of reality. A stereotype is misapplying properties of specific and limited context to a universal scope.
- jrkatz 10y agoIt depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that. If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.
- scribu 10y agoI agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text. But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.
- Houshalter 10y agoThe distinction is very important. If it's just regurgitating human biases that would be bad. Humans often have very inaccurate and warped beliefs after all. If it's accurately modelling reality, then what's the problem? That's what we want it to do. Why would you want a less accurate model of reality? I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evidence that the humans who wrote the corups had any bias at all. As long as there are more news articles about female nurses than male nurses, the model will learn a correlation between the concepts.
- anigbrowl 10y agoWhy would we want to reproduce existing structures of oppression in mechanical form? Have you noticed how automation often vastly amplifies things? It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.' Surely you are aware that in real world world people rationalize decisions they don't actually understand all the time because they are not capable of or interested in improving upon the system within which they pursue their own economic interest on behalf of others whose interests do not seem coincident with their own.
- noir_lord 10y ago> Why would we want to reproduce existing structures of oppression in mechanical form? If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.
- arrrg 10y agoIf it’s saying that and only that then that’s obviously fine. However, if that knowledge is then applied in any other way, then that’s problematic.
- notahacker 10y agoIn an environment in which Blue people are banned from becoming doctors, its also inferring something about reality to conclude that 0% of Doctors are Blue. It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed. Obviously "structures of oppression" - real and imagined - which lead to fewer female doctors even in western liberal democracies where women wishing to become doctors are generally met with encouragement are less extreme, but that isn't to say they don't exist or that a computer output (or human interpretation of said computer output) is likely to draw correct inferences from it. And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you
- anigbrowl 10y agoThe result is expected but not desirable. However, I have little hope of you realizing why any time soon.