4 ms·
In Which We Define 'Biased' As Meaning 'Not Conforming To Our Ideas Of How The World Should Be'. Because it's unthinkable that movies with male main characters
by taneq 9y ago
In Which We Define 'Biased' As Meaning 'Not Conforming To Our Ideas Of How The World Should Be'. Because it's unthinkable that movies with male main characters could actually just be better than movies with female main characters.
(I'm not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?")
- ASpring 9y agoWe can quibble over some of the examples they give and their "ideas of how the world should be" but there are many cases where fairness may be explicitly legislated or defined that makes this research useful.
- rspeer 9y agoIf you see a biased result and your first thought is "is that result accurate?", it's because either you like the bias or you think data is a source of objective truth. If it's because you like the bias, what the fuck. If it's because you think data is a source of objective truth, what the fuck.
- blackbagboys 9y agoI'm not unsympathetic to your concerns, but the obvious retort is that if you already know which results are accurate, why bother doing analysis?
- jayd16 9y agoSo you can build a model and get a computer to do the analysis...
- rspeer 9y agoPeople who make word embedding models are not doing it to find out whether men are better than women. That is not the analysis they are doing. You always have to look at your results with a critical eye to recognize when something has gone wrong with your model. You can't recognize a flaw in the model if you believe everything the model says.
- blackbagboys 9y agoI'd be pretty amazed if it was even possible to construct a model that would answer that question. In the article's example, you're constructing a model to determine how a movie might be reviewed. The data indicates that movie reviewers may be reviewing movies with male leads more positively than movies with female leads. There are a lot of possible reasons for this other than "men are better than women", so why would you simply reject the result out of hand?
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- sanxiyn 9y agoData, of course, is a source of objective truth. It is a source, not the entire source, but it is a source.
- nl 9y agoIn NLP, data is very much NOT a source of objective truth. Language has too much implied context and meaning which is communicated outside the text under analysis. Embeddings attempt to capture this, but they aren't as good as humans. "XXXXX is a dog of a movie" - does this mean the movie was bad or was the author playing with words when the movie is about dogs?
- jayd16 9y ago>Because it's unthinkable that movies with male main characters could actually just be better than movies with female main characters. Its not unthinkable but its also very likely that bias does indeed have an impact. What do comments like these add to the discussion?
- sanxiyn 9y agoThey add to the discussion because they remind us that the phenomenon is likely to be mix of reality and bias and that we are very uncertain about proportion.
- zaphos 9y agoThe article does not advocate tuning out the results; it is advocating awareness of the results and careful consideration of how to proceed. See the detailing of options at the end of the second case study: >> "As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions."
- comex 9y agoThe article does briefly mention main characters as a hypothetical example, but the test case it provides data for is different: > In this case, she takes the 100 shortest reviews from her test set and appends the words "reviewed by _______", where the blank is filled in with a name. So for the model to be reflecting reality, it would have to be the case that movies that male critics tend to review are better, on average, than movies that female critics tend to review. Which is possible, I guess – but given the limited amount of training data (and the fact that the no-embedding model shows no bias), it seems more likely that the model is just picking up generalized associations from the embedding and applying them blithely to names it sees in the text, without really understanding the context. edit: Probably including associations as dumb as “this word is associated with positive/negative sentiment” (though more complex factors may also be involved).
- stfwn 9y ago> the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?" That depends on the product that you want to make. If you’re a company that wants to sail on the status quo and just make money, by all means be amoral. On the other hand, technology now allows us to analyze and steer social structure at scale. We could reproduce past inequalities in the name of efficiency, or we could think a bit longer about our axis of optimization. “But cultural relativism!” — I agree. It’s a tough problem, but we don’t all of the sudden need a definitive generalized social plan. Biases can be tackled one algorithm at a time.
- iguy 9y agoMaybe a useful way to think about this is Positive/Normative distinction. Science is "amoral" in your sense, it's trying to describe how things really are, and make predictions. So is the stock market: you're rewarded for correct predictions, whether or not others regard those outcomes as desirable. Of course we are also moral beings, interested in changing things for the better. And we have varying and contradictory ideas about what better means. As soon as you say "steer social structure at scale" the question is: who gets to steer? If this were obvious, then we wouldn't need democracy, we could just all work together not to "reproduce past inequalities"... but we can't.
- AnthonyMouse 9y ago> I'm not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?" Especially when you consider the weird feedback loops you can get by blindly fudging the numbers. For example, suppose the objectively best movies have a male lead, e.g. because Hollywood is biased and spends more resources producing those movies than ones with a female lead. So the bias is there, but it's at the production stage, not at the recommendation stage. What happens if we try to fix it at the recommendation stage? Hollywood sees that movies with a female lead are now being rated higher. But five stars is five stars. The occasional actually-great movie with a female lead can't get itself kicked up to six, so the incentive isn't to make more of those. It's to make more mediocre movies with a female lead to take advantage of the artificial boost. Which widens the gulf in the unmodified ratings even more and produces a vicious cycle where the number of garbage movies with a female lead explodes to take advantage of the fact that no matter how bad they get, the average will be adjusted upward to compensate.
- skybrian 9y agoSure, it's always good to dig deeper and find out where a correlation comes from. But the question isn't just whether it's true, but also whether it will stay true. As they say in finance, past performance does not guarantee future results. We are not talking about the laws of physics here. There are historical correlations that it's unsafe to rely on, particularly when people are trying to change them. Part of this is overfitting, but there's also the problem of drift. A model may stop working well because the world has changed. So some correlations can be rejected just because we don't have confidence in them. It's a dependency that looks too fragile.
- curun1r 9y ago> Because it's unthinkable that movies with male main characters could actually just be better than movies with female main characters Or, more realistically, it's not unthinkable that movies with a male lead could be better reviewed than movies with female main characters, especially when you open the field up to non-professional reviewers. We've see how ape-shit people went at movies like the recent Ghostbusters remake or even the new Star Wars series just because they put women in the kinds of roles that had traditionally been played by men. It's an interesting question of when machine learning models that reflect actual human biases are correct and when we should try to adjust them for that human bias. In the case of movie reviews, you could make the case that you'd want the models to reflect the biases of the reviewers. Just because a reviewer is being sexist doesn't mean it's wrong for the machine learning model to correctly classify the reviewer's sentiments.