14 ms·

Text Embedding Models Contain Bias

222 points by gajju3588 9y ago

troupe 9y ago
> But what if we found that while Model C performs the best overall, it's also most likely to assign a more positive sentiment to the sentence "The main character is a man" than to the sentence "The main character is a woman"? As I understand the problem, they are saying that statistically, the statement about the main character being male is a bit more likely to be positive than if the same thing is said about a woman. If that is statistically true and you are trying to create a model to determine the level of positive sentiment in a review, then that may be a legitimate indicator of how people categorize things. If the goal is to try to "fix" how people talk and write, I'm not sure ignoring statistical patterns in the way we talk is really the right approach.
- elliotec 9y ago
  Agreed. The machine is biased because biased humans are training it to be biased.
  - troupe 9y ago
    Well I think some of it has to do with the fact that language is not completely symmetrical (there may be a better word). For example, the following sentences: 1. They went to college. 2. They went to the University of Phoenix. The fact you are calling out UoP may turn out to be used more in slightly less positive sentiments than the first one. If you are trying to figure out the sentiment of a few sentences, this might be important. Yes you can ignore it, but the question is whether you are trying to understand how the language is actually being used or not.
    - elliotec 9y ago
      True again - biased humans creating biased machines asking biased questions and receiving biased answers.
      - troupe 9y ago
        Keep in mind this wasn't an analysis of whether people liked men or women in the lead role. It was basically pointing out that one statement was a little more likely to be more strongly positive than the other. So you think asking what level of positive (or negative) sentiment does this review express is a biased question?
    - yorwba 9y ago
      A real-world example: 交大不如复旦 [jiāodà bùrú fùdàn] gets translated as "Jiaotong University is not as good as Fudan University" by Google https://translate.google.com/?hl=en#auto/en/%E4%BA%A4%E5%A4%A7%E4%B8%8D%E5%A6%82%E5%A4%8D%E6%97%A6 https://translate.google.com/?hl=en#auto/en/%E4%BA%A4%E5%A4%... 复旦不如交大 [fùdàn bùrú jiāodà] (just swapping the word order) is translated into "Fudan is better than Jiaotong University" https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%A6%E4%B8%8D%E5%A6%82%E4%BA%A4%E5%A4%A7 https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%... The literal translation of 不如 would be "is unlike", but usually it implies a value judgment. In the translation, Google seems to be consistently sure that Fudan is simply better. But when you specify that you're talking about the Jiaotong University in Shanghai and not one of the others, it suddenly changes its mind: 上海交大不如复旦 [shànghǎi jiāodà bùrú fùdàn] "Shanghai Jiaotong University is better than Fudan University" https://translate.google.com/?hl=en#auto/en/%E4%B8%8A%E6%B5%B7%E4%BA%A4%E5%A4%A7%E4%B8%8D%E5%A6%82%E5%A4%8D%E6%97%A6 https://translate.google.com/?hl=en#auto/en/%E4%B8%8A%E6%B5%... 复旦不如上海交大 [fùdàn bùrú shànghǎi jiāodà] "Fudan is not as good as Shanghai Jiaotong University" https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%A6%E4%B8%8D%E5%A6%82%E4%B8%8A%E6%B5%B7%E4%BA%A4%E5%A4%A7 https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%... I'm at SJTU and everyone here seems to agree that this is the objective truth, but the people at Fudan are probably not so happy about it.
      - AstralStorm 9y ago
        Unfortunately begging the questions: in what ways and how much? And most importantly: is the difference meaningful? Google Translate and most other alleged artificial intelligence won't be able to answer that question meaningfully.
      - nmca 9y ago
        This is a really excellent example.
      - iguy 9y ago
        This is a great example. It's like an exam question where you have to know what the teacher is thinking. Notice that in "Fudan is better than Jiaotong University" you're supposed to know that Fudan is another place with a university, not (say) a kind of apprenticeship... or a joke about some kind of noodles or something. You're supposed to have some outside context, but not too much, not enough to know their reputations. That's quite a fine line to ask a translation system to draw.
- gowld 9y ago
  Good, because the article clearly states that the critical takeway is that modelers must pay attention to statistical patterns, and not blithely ignore them.
- lsiebert 9y ago
  Actually it's any name, even the name of the reviewer, as the test they conducted showed. the issue is that humans don't understand multi-variable statistical analysis, whether in the form of ANOVA or machine learning training, so they try to pack everything down into two or sometimes three variables of output. And that's fine if you are pulling from a population that's homogenous. But if there are two or more discrete subpopulations, you want to control for them or represent them separately, not just ignore them or pretend they reflect the information you want. Anyway if the reviewer name is enough to throw the results, it may suggest that guys praise more movies, rather than movies with male character's get more praise. I just looked at the first page of 10 star reviews of battlefield Earth and none seemed to be female, just saying.
  - dibstern 9y ago
    If the multiple sub populations exist, that’ll be reflected in the data, as long as your dataset is good enough.
    - lsiebert 9y ago
      Maybe? But even something as simple as an analysis of human height has genetic/ethnic, nutritional, age and gender components, not to mention historical differences. If your input is name, and your output is predicted height, knowing and testing for bias in your data sources is definitely important. I think where people get caught up is that they don't see the world at large as biased, because they view their understandings as essentially correct. For example, we expect judges to rule fairly on every case, right? To pick a non controversial issue, likelihood of parole is apparently effected by how recently the judge ate https://www.economist.com/node/18557594 https://www.economist.com/node/18557594 Now, parole data accurately reflects how people were paroled, so predicting likelihood of clemency requests is a perfectly valid use of that data. If you were doing machine learning to try to help people get paroled, you'd want to leave that bias in as a predictor, because it's unfair but real. But you'd probably want to adjust that data to correct for the recently having eaten bias if you were writing a system for parole recommendations for new judges based on past judicial decisions. You wouldn't want people to be more likely to be denied just because they came before a judge before lunch. And if you don't test for a bias like that, how would you be able to tell that the machine learning algorithm had it? And it wouldn't even need to be direct... seeing people A-Z in court could mean a bias based on name. Long story short, bias is a real issue, and you need to be aware of it and test for it, not assume that your input data isn't effected by human error.
jabgrabdthrow 9y ago
Uh oh!
smallnamespace 9y ago
Be careful what you wish for. Under this definition of 'bias', an unbiased model would, say, spit out equal associations between any occupation and any gender/sex/age/race/religion label. We should probably ask ourselves whether that's a strictly desirable outcome, since by definition the 'biased' model has a higher predictive value. How much accuracy are we willing to sacrifice for the sake of erasing inconvenient facts about either our world, or our current models of the world?
- perfmode 9y ago
  Predictive value isn’t an end in itself.
  - jessaustin 9y ago
    That very much depends on one's interests. In today's modern economy, many are encouraged to tie their interests to lots of events that haven't happened yet. They prioritize predictions.
    - rspeer 9y ago
      That still doesn't mean that your training data is right for the predictions you're going to make. Here's an example detached from any sort of politics. If you are training a language model, the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0. On your training data, you'll measure the maximum-likelihood model as making more accurate predictions than any other model. But it's also useless, because when you use it on new data, it assigns an impossible probability, 0, to any sentence containing a word it's never seen. This is a bias. You can't correct it by getting more data. You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it (smoothing). Now account for the bias (observed by Arvind Narayanan) that all your training data is in the past, and all the predictions you want to make are in the future, but circumstances change between the past and the future.
deleted 9y ago
[deleted]
taneq 9y ago
In Which We Define 'Biased' As Meaning 'Not Conforming To Our Ideas Of How The World Should Be'. Because it's unthinkable that movies with male main characters could actually just be better than movies with female main characters. (I'm not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?")
- ASpring 9y ago
  We can quibble over some of the examples they give and their "ideas of how the world should be" but there are many cases where fairness may be explicitly legislated or defined that makes this research useful.
- rspeer 9y ago
  If you see a biased result and your first thought is "is that result accurate?", it's because either you like the bias or you think data is a source of objective truth. If it's because you like the bias, what the fuck. If it's because you think data is a source of objective truth, what the fuck.
  - blackbagboys 9y ago
    I'm not unsympathetic to your concerns, but the obvious retort is that if you already know which results are accurate, why bother doing analysis?
    - jayd16 9y ago
      So you can build a model and get a computer to do the analysis...
    - rspeer 9y ago
      People who make word embedding models are not doing it to find out whether men are better than women. That is not the analysis they are doing. You always have to look at your results with a critical eye to recognize when something has gone wrong with your model. You can't recognize a flaw in the model if you believe everything the model says.
Spooky23 9y ago
I guess the question posed in the article is an interesting one and is an interesting discussion re how to deal with bias and building counter-biases into algorithms without the editorial decision asking being clear. Movie reviews are editorial content. Measuring that content is a difficult problem in this type of context... Are the best reviewers people who dislike movies with female leads? Are you going into a back catalog of movie reviews from an age where societal expectations were different? Are popular genres skewing the result? You could have a curation issue as well — if the female lead movies are dominated by "Hallmark Channel" fare, algorithm C has a point!
danielvf 9y ago
This is a genuine, difficult problem. It's so easy to join up on your political team of choice and scream about it, and all this makes any real attempt to solve it so much harder to talk about in public or collaborate on. In fact, there's practically guaranteed to be some greyed out text in the discussion here. So some of these associations simply reflect the way-the-world-was or the way-the-world-is - like associating "woman" with "housewife". That's a whole debate in itself. But some of these can be accidental. Suppose a runaway success novel/tv/film franchise has "Bob" as the evil bad guy. Reams of fanfictions are written with "Bob" doing horrible things. People endlessly talk about how bad "Bob" is on twitter. Even the New York times writes about Bob latest depredations, when he plays off current events. Your name is Bob. Suddenly all the AI's in the world associate your name with evil, death, killing, lying, stealing, fraud, and incest. AI's silently, slightly ding your essays, loan applications, uber driver applications, and everything you write online. And no one believes it's really happening. Or the powers that be think it's just a little accidental damage because the AI overall is still, overall doing a great job of sentiment analysis and fraud detection.
- protomyth 9y ago
  Why is someone using fiction and references to fiction to train an AI that will make decisions about real people’s livelihood?
  - jessaustin 9y ago
    Do you have no sympathy for an ordinary fairly harmless person named "Donald" or "Bashar"?
    - deleted 9y ago
      [deleted]
  - gowld 9y ago
    Because AIs rely on real humans' judgments, and real humans are biased by fictional stories.
ajwnwnkwos 9y ago
This is an example of sacrificing the scientific method to make results more politically correct. We've come full circle.
- rspeer 9y ago
  If you understood the scientific method, you would understand that you always need to account for bias. Just because something comes from data doesn't mean it's objective.
  - avinium 9y ago
    Hold up - you also have to tread very carefully there. I read your blog post above about "racist" NLP models and I agree it makes sense to debias inputs for certain purposes. However, many people in this thread aren't talking about adjusting for statistical bias. They're really talking about adjusting for socio-political over/under-representation, which is getting onto pretty shaky ground. I'd hazard a guess that a model trained on Google News giving "more negative" sentiment for black names is a function of those names appearing more frequently in crime articles, matching the overrepresentation in crime rates. Likewise for Arab/Muslim names, which are presumably disproportionately present in articles on terrorism. Now, that's obviously statistical sampling bias if you're (somehow) modelling "names I should call my kid". If you're looking at "names more associated with crime", however, then we shouldn't be eliminating any racial imbalances simply because they make us feel queasy. That's intellectually dishonest, and does everybody a disservice by trying to sweep uncomfortable realities under the rug. Same goes for gender bias in word embeddings from classical fiction - that is (I assume) a very accurate depiction of social gender imbalance from that period of history. That may or may not be relevant to the question you're asking - but that doesn't make any bias inherently "wrong". I think the underlying message should be "models only reflect the data they're given and may need to be adjusted for bias depending on the question being asked", not "race and gender is always irrelevant and should always be normalized".
- to_bpr 9y ago
  >This is an example of sacrificing the scientific method to make results more politically correct. We've come full circle. Wrongthink is nothing new, comrade ajwnwnkwos, and is as prevalent as ever. The output of science has been suppressed throughout history where it didn't fit the narrative of the day. At one period, that was by the Church. Another, it was at the hands of the government. Today, mainly by a self-censoring, everything-must-be-pleasant-and-entertaining society that is very highly prone to fits of outrage. Inconvenient facts are, after all, inconvenient.
cperciva 9y ago
I imagine that the model would also score "he was murdered" higher than "she was murdered". Models reflect their inputs, and it happens that yes, murder victims are disproportionately likely to be male and nurses are disproportionately likely to be female. Is there a problem we should address here? Absolutely -- but the problem is that men keep on getting murdered, not that the model recognizes truths with which we are uncomfortable.
- gowld 9y ago
  No one is saying the model is wrong. As the article states: > As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. > She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). > She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). > No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions.
- deleted 9y ago
  [deleted]
- allenz 9y ago
  Biased models can cause ethical and legal problems. While your specific example is not a huge deal, the article gives the example of making hiring decisions in part based on sentiment analysis of candidates' text reviews. In this context, an engineer has responsibility to ensure that the model has no gender, race, or age bias towards candidates' names. For a real life example, in 2017 Google was more likely to filter the comment "I am a woman" than "I am a man": https://www.engadget.com/2017/09/01/google-perspective-comment-ranking-system/ https://www.engadget.com/2017/09/01/google-perspective-comme... Or consider the impact of any bias in AI for criminal sentencing recommendations: https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi...
pcunite 9y ago
Here is a video that explains this blog well. https://youtu.be/59bMh59JQDo https://youtu.be/59bMh59JQDo The unnerving part for me is this "eliminate negative associations/bias". Okay, how about we learn the truth, and then address that outside in real life and keep the computer doing what it's good at ... showing us the data.
- gowld 9y ago
  > Okay, how about we learn the truth, and then address that outside in real life Aside from the aspect you overlooked -- that our online lives are real life, and that AI models are (over)used to make decisions, not merely show us the data -- your idea is quite what the article recommends: > . As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. > She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions.
blackbagboys 9y ago
>> Normally, we'd simply choose Model C. But what if we found that while Model C performs the best overall, it's also most likely to assign a more positive sentiment to the sentence "The main character is a man" than to the sentence "The main character is a woman"? Would we reconsider? It seems like you have discovered that movie reviewers tend to review movies with am male main character more highly than movies with a female main character; what you need to consider is that while this may tell you something about movie reviewers, it doesn't necessarily tell you anything about the quality of the movie.
jayd16 9y ago
I'm reminded of a much more obvious example. https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch... I see a lot of comments about how its somehow sinister to want your model to be better than the lowest common denominator and that is pretty damn ridiculous.
avz 9y ago
I think the core issue here is that people want two things. On one hand we want our models to accurately describe reality, not an idea of what reality should be. On the other hand, we don't want ML to freeze society and culture in their current state, but to help decide on and drive social change. The tension between these goals arises when models trained today are used to make decisions tomorrow. One way to resolve the tension might be to add time dimension and historical training data. The models might then be able to return in addition to any prediction variable p also its time derivative dp/dt. For example, a model might then return results such as: "movies with female main character: lower sentiment, trending up; movies with male main character: higher sentiment, trending down".
haberman 9y ago
This sounds a lot like how people get TSA redress numbers when their info falsely flags them as suspicious (https://www.dhs.gov/redress-control-numbers https://www.dhs.gov/redress-control-numbers). Or how Barack Obama suffered innuendo around the middle name "Hussein." Mistaken identity or unfortunate associations are as old as humanity. AI systems (and non-AI systems) need ways to deal with these problems, but we also have a lot of experience about how to do that.
- bobthepanda 9y ago
  The problem is when we hand AI full reigns over a system that should have humans with the final say, because doing it with humans is "too hard" or "doesn't scale." At least with humans you have auditable decision making and recourse via the legal system, even if it can be hard to fight. AI are essentially black boxes that unconsciously learn the biases of the datasets they are given. When AI starts determining rather consequential things like how long to send someone to prison for, that's a problem. https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso...
  - manux 9y ago
    > AI are essentially black boxes that unconsciously learn the biases of the datasets they are given. Please don't confuse "AI" with the current state-of-the-art of the latest deep learning model. Many of us researchers are working on interpretability and understanding of causality. Calling it "AI" makes it appear final, as if in 50 years all machine learning-based decision systems will behave exactly as they do now, without nuance.
    - bobthepanda 9y ago
      Calling currently existing models AI is no different than calling a Model T a car. It being more primitive than what may exist in the future is of little consequence to the public dealing with the consequences of AI today. AI can only learn from the data it has, so it will always carry some sort of bias, because it is impossible to collect the nuance of every last bit of context into a digestible data format. At best it's an advisor, but it should never be a decision-maker.
jfasi 9y ago
It's hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the sake of example) a psychological evaluation application. Now imagine that after years of use you discover that your model systematically marks african americans as less psychologically fit than white americans. Who would be to blame? Naturally, you would. Your actions led to a biased model being used to unjustly and arbitrarily harm innocent people, and your leadership would be right to call into question every decision your application ever made. Now imagine the same scenario except your app was trained on data instead of hand-written. Make no mistake, the answer to the question of who's to blame is exactly the same: the developer. The response should be exactly the same: a complete loss of confidence in the model. I'm appalled that this needs to be said, but reading this comments section I'm afraid it does: Machine learning models are inference and pattern recognition devices, not scientific tools. They don't magically reveal hidden patterns in the world, they repeat the patterns that the developers train them on. If you trained a machine learning model to perform psychological evaluations [1] or sentence convicts [2] or recognize faces [3], and your model is biased in a way that is unnecessary and unjust, your model is bad you should be held accountable for its failures. [1] https://affect.media.mit.edu/projects.php?id=4079 https://affect.media.mit.edu/projects.php?id=4079 [2] https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso... [3] https://www.wnycstudios.org/story/deep-problem-deep-learning/ https://www.wnycstudios.org/story/deep-problem-deep-learning...
- deleted 9y ago
  [deleted]
- whatshisface 9y ago
  What do you do when your classification is correlated to race through no fault of yours? For example, I might successfully predict credit score from backyard size, and end up with a correlation to race that doesn't have anything to do with me. I don't think the blame always lays with the algorithm, especially when it doesn't have access to race as an input (this is a reasonable expectation). I can score students with a simple algorithm based on what they write on their math tests, and even that's going to correlate with race. In that case the blame pretty clearly lies in the process that produced the reality that's being measured, not the measurement technique itself. Let's say that black people default on their loans more often then white people. Is it better to criticize the math that discovered that fact, or the root cause that made it true to begin with?
rspeer 9y ago
I'm glad that Google is part of this conversation, and they're now applying tests for bias to new models that they release. (Some of their old models are pretty awful.) If you want to see a further example, in the form of a Jupyter notebook demonstrating how extremely straightforward NLP leads to a racist model, here's a tutorial I wrote a while ago [1]: [1] http://blog.conceptnet.io/posts/2017/how-to-make-a-racist-ai-without-really-trying/ http://blog.conceptnet.io/posts/2017/how-to-make-a-racist-ai...
- jfasi 9y ago
  I really have nothing to comment except how excellent I found your writeup. Seriously, wow.
- tomcatfish 9y ago
  Really good write up; it is going to help explain to people how data can lead to unhelpful biases by showing a few of your graphs!
- nl 9y ago
  For those who aren't aware, @rspeer has been taking this problem seriously for years. His ConceptNet NumberBatch embeddings[1] are one of the few pre-built releases which attempt to fix this. [1] https://github.com/commonsense/conceptnet-numberbatch https://github.com/commonsense/conceptnet-numberbatch
home_boi 9y ago
The politicization of bias in this realm is unproductive. We know where it leads. There will be 'committees' of people who are not trained in statistical thinking, who haven't even taken a statistics class, who are not qualified to make any statements and who's interests are not in line with making the best product acting as nuisances and aggressors to people who are do have the expertise and who strive to make the best product. The best course of action is to treat this bias like normal bias with non-politicized, inanimate objects. How would data scientists that encountered similar bias while running machine learning models of the motion of waves act?
- gowld 9y ago
  The article was written by trained experts in statistical thinking. It's worth a read.
dgudkov 9y ago
OK, factual data fed to some AI-based algorithms produces results that are not so politically correct as some people would like it to be. Is this a problem with the AI algorithms, or with the people?
pgodzin 9y ago
Key takeaway: It is important to be aware of bias in ML models. Some biases may correctly model the reality of the world, and some may show the bias in the underlying dataset or in what the model has focused on in the data. The goal is not to "unbias" everything, as people seem to be focusing on, but rather to determine if the bias is appropriate given the context.
- kriro 9y ago
  I think that's a pretty solid summary.
- avinium 9y ago
  Damn, I just wrote almost exactly these words before scrolling down to your comment. Could have saved myself a few minutes by upvoting yours instead.
lsiebert 9y ago
I'm reminded of https://whitecollar.thenewinquiry.com https://whitecollar.thenewinquiry.com It turns out that white collar crime is predominantly committed by white men. A system trained to detect white collar crime using, say, enron emails, might suggest a white guy's emails over someone whose name doesn't sound like an enron employee, or who shared pictures of their cat. I mean, I suppose you can argue that hey, maybe that bias is usually correct. Maybe it usually is the white guy. But personally, I'd probably control for things conflated with gender or race and then look for indicators that differentiate between criminals and innocent people. You will probably have a lower AUC, but better differentiation between criminals and innocent people is what matters.
cup-of-tea 9y ago
If the models reflect bias then surely that's a good thing. It's funny. I like programming because a computer can't lie and doesn't make mistakes. I guess some people don't like that.
- UncleMeat 9y ago
  The goal of ml isn't to accurately model training data. The goal is to make something useful. Correctly doing the former can hinder the latter.
_pmf_ 9y ago
"Pray, Mr. Babbage, if you put into the machine biased models, will unbiased answers come out?"
nukeop 9y ago
Google uses their AI systems for profiling where the sex of the people being profiled is a crucial piece of information and acts as a predictor for interests which is then further used in targeting ads. As long as it makes Google money, it's not a problem. This bias actually accurately reflects reality and the advertisers know that, otherwise they wouldn't be paying Google for targeted ads.
throwaway84742 9y ago
I’ll tell you more: _human judgment_ contains bias. You can’t possibly think logically about every single judgment, particularly when information is incomplete, which it is in the overwhelming majority of cases. It is not a given, to me, that on average AI does any worse than the human population outside the “woke” segment. Or even _within_ that segment considered in isolation.
- jimmytidey 9y ago
  If you put a human in charge of a judgement call, mostly there are mechanisms to monitor them, a requirement that they give reasons for their decisions, and mechanisms to appeal those judgements. We aren't used to having to do that with decisions made by computers. It's not that AI makes better or worse decisions, it's the way we treat those decisions.
  - throwaway84742 9y ago
    Same as with AI, it is not economical to monitor human decisions, so mostly it doesn’t actually happen. See eg Youtube demonetization as a particularly illustrative example.
  - iguy 9y ago
    "a requirement that they give reasons for their decisions" But this is mostly a sham. People lie about their reasons for doing things, even to themselves. Especially when they know what reasons are publicly acceptable. Maybe the major difference is that it's much easier to run experiments on computers. I mean people try this on humans but it's very hard to do realistically: most of those studies where you submit 1000 CVs with varying details are garbage, because they can only access an unrealistic part of the process (I mean who ever got a job without networking? etc). Whereas on almost any computer system you feed it completely realistic fake data.
- jccalhoun 9y ago
  I think the problem is that a lot of people think something like, "It is an algorithm not a person so it must be totally scientific and unbiased" when that isn't the case. The non-techy public needs to be aware that these things are made by humans who may be recreating their bias in the technology.
ablx_ 9y ago
FYI, a good talk about this from 33C3: https://media.ccc.de/v/33c3-8026-a_story_of_discrimination_and_unfairness https://media.ccc.de/v/33c3-8026-a_story_of_discrimination_a...
sooham 9y ago
Despite the controversy surrounding "debiasing" classifier outputs, I think further research in this area is still of merit. This area of research would help us understand and build transformations over latent / high level representation space, a general use case applicable to all fields interacting with machine learning.
callesgg 9y ago
The only thing that can be biased is the training data. The model is simply a statistical breakdown of the training data.
bloak 9y ago
I wish people who write articles about "bias" would explain what they mean by "bias". I've seen hundreds of these articles. I'm still waiting for a usable definition.
sagarm 9y ago
There's a lot of complaints here about "erasing reality" and other hyperbolic talk, but these models are trying to make predictions within a particular user's context. It's just inappropriate to apply some global biases for a particular user, and avoiding that can result in a better user experience.
- joe_the_user 9y ago
  It's just inappropriate to apply some global biases for a particular user, and avoiding that can result in a better user experience. The problem is that most deep learning programs aren't intended for (and really be for) one user out there in user space. Deep learning programs are written for the large institutions which have the large troves of data need to train large nets and want to make important decisions using that data. But by this, those decisions won't be made in isolation but will effect a large number of people, people who aren't the users but rather the used. And if such systems have biases relative to the whole, it is a problem. And problem may not be for the institutions immediate goals but for the people who depend on the institution.
- haberman 9y ago
  I think you make an interesting point: if we use context to train a model to reflect local biases instead of global biases, will that be more just and/or lead to better user experience? It seems related to the question of whether Google results should be tailored to you. If I Google "did Russia interfere in the election", should Google tailor the results so I always see articles that reinforce my world view? If we go that route, I think we take the path of Stephen Colbert's concept of "Truthiness", where we judge something as true because it "feels" true. Users will definitely be happier if everything they see reinforces their existing world view. So companies will be incentivized to accommodate this desire. But does this actually lead to a more just society?

222 points by gajju3588 9y ago

> But what if we found that while Model C performs the best overall, it's also most likely to assign a more positive sentiment to the sentence "The main character is a man" than to the sentence "The main character is a woman"? As I understand the problem, they are saying that statistically, the statement about the main character being male is a bit more likely to be positive than if the same thing is said about a woman. If that is statistically true and you are trying to create a model to determine the level of positive sentiment in a review, then that may be a legitimate indicator of how people categorize things. If the goal is to try to "fix" how people talk and write, I'm not sure ignoring statistical patterns in the way we talk is really the right approach.

elliotec 9y ago
Agreed. The machine is biased because biased humans are training it to be biased.
- troupe 9y ago
  Well I think some of it has to do with the fact that language is not completely symmetrical (there may be a better word). For example, the following sentences: 1. They went to college. 2. They went to the University of Phoenix. The fact you are calling out UoP may turn out to be used more in slightly less positive sentiments than the first one. If you are trying to figure out the sentiment of a few sentences, this might be important. Yes you can ignore it, but the question is whether you are trying to understand how the language is actually being used or not.
  - elliotec 9y ago
    True again - biased humans creating biased machines asking biased questions and receiving biased answers.
    - troupe 9y ago
      Keep in mind this wasn't an analysis of whether people liked men or women in the lead role. It was basically pointing out that one statement was a little more likely to be more strongly positive than the other. So you think asking what level of positive (or negative) sentiment does this review express is a biased question?
  - yorwba 9y ago
    A real-world example: 交大不如复旦 [jiāodà bùrú fùdàn] gets translated as "Jiaotong University is not as good as Fudan University" by Google https://translate.google.com/?hl=en#auto/en/%E4%BA%A4%E5%A4%A7%E4%B8%8D%E5%A6%82%E5%A4%8D%E6%97%A6 https://translate.google.com/?hl=en#auto/en/%E4%BA%A4%E5%A4%... 复旦不如交大 [fùdàn bùrú jiāodà] (just swapping the word order) is translated into "Fudan is better than Jiaotong University" https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%A6%E4%B8%8D%E5%A6%82%E4%BA%A4%E5%A4%A7 https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%... The literal translation of 不如 would be "is unlike", but usually it implies a value judgment. In the translation, Google seems to be consistently sure that Fudan is simply better. But when you specify that you're talking about the Jiaotong University in Shanghai and not one of the others, it suddenly changes its mind: 上海交大不如复旦 [shànghǎi jiāodà bùrú fùdàn] "Shanghai Jiaotong University is better than Fudan University" https://translate.google.com/?hl=en#auto/en/%E4%B8%8A%E6%B5%B7%E4%BA%A4%E5%A4%A7%E4%B8%8D%E5%A6%82%E5%A4%8D%E6%97%A6 https://translate.google.com/?hl=en#auto/en/%E4%B8%8A%E6%B5%... 复旦不如上海交大 [fùdàn bùrú shànghǎi jiāodà] "Fudan is not as good as Shanghai Jiaotong University" https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%A6%E4%B8%8D%E5%A6%82%E4%B8%8A%E6%B5%B7%E4%BA%A4%E5%A4%A7 https://translate.google.com/?hl=en#auto/en/%E5%A4%8D%E6%97%... I'm at SJTU and everyone here seems to agree that this is the objective truth, but the people at Fudan are probably not so happy about it.
    - AstralStorm 9y ago
      Unfortunately begging the questions: in what ways and how much? And most importantly: is the difference meaningful? Google Translate and most other alleged artificial intelligence won't be able to answer that question meaningfully.
    - nmca 9y ago
      This is a really excellent example.
    - iguy 9y ago
      This is a great example. It's like an exam question where you have to know what the teacher is thinking. Notice that in "Fudan is better than Jiaotong University" you're supposed to know that Fudan is another place with a university, not (say) a kind of apprenticeship... or a joke about some kind of noodles or something. You're supposed to have some outside context, but not too much, not enough to know their reputations. That's quite a fine line to ask a translation system to draw.
gowld 9y ago
Good, because the article clearly states that the critical takeway is that modelers must pay attention to statistical patterns, and not blithely ignore them.
lsiebert 9y ago
Actually it's any name, even the name of the reviewer, as the test they conducted showed. the issue is that humans don't understand multi-variable statistical analysis, whether in the form of ANOVA or machine learning training, so they try to pack everything down into two or sometimes three variables of output. And that's fine if you are pulling from a population that's homogenous. But if there are two or more discrete subpopulations, you want to control for them or represent them separately, not just ignore them or pretend they reflect the information you want. Anyway if the reviewer name is enough to throw the results, it may suggest that guys praise more movies, rather than movies with male character's get more praise. I just looked at the first page of 10 star reviews of battlefield Earth and none seemed to be female, just saying.
- dibstern 9y ago
  If the multiple sub populations exist, that’ll be reflected in the data, as long as your dataset is good enough.
  - lsiebert 9y ago
    Maybe? But even something as simple as an analysis of human height has genetic/ethnic, nutritional, age and gender components, not to mention historical differences. If your input is name, and your output is predicted height, knowing and testing for bias in your data sources is definitely important. I think where people get caught up is that they don't see the world at large as biased, because they view their understandings as essentially correct. For example, we expect judges to rule fairly on every case, right? To pick a non controversial issue, likelihood of parole is apparently effected by how recently the judge ate https://www.economist.com/node/18557594 https://www.economist.com/node/18557594 Now, parole data accurately reflects how people were paroled, so predicting likelihood of clemency requests is a perfectly valid use of that data. If you were doing machine learning to try to help people get paroled, you'd want to leave that bias in as a predictor, because it's unfair but real. But you'd probably want to adjust that data to correct for the recently having eaten bias if you were writing a system for parole recommendations for new judges based on past judicial decisions. You wouldn't want people to be more likely to be denied just because they came before a judge before lunch. And if you don't test for a bias like that, how would you be able to tell that the machine learning algorithm had it? And it wouldn't even need to be direct... seeing people A-Z in court could mean a bias based on name. Long story short, bias is a real issue, and you need to be aware of it and test for it, not assume that your input data isn't effected by human error.

jabgrabdthrow 9y ago

Uh oh!

smallnamespace 9y ago

Be careful what you wish for. Under this definition of 'bias', an unbiased model would, say, spit out equal associations between any occupation and any gender/sex/age/race/religion label. We should probably ask ourselves whether that's a strictly desirable outcome, since by definition the 'biased' model has a higher predictive value. How much accuracy are we willing to sacrifice for the sake of erasing inconvenient facts about either our world, or our current models of the world?

perfmode 9y ago
Predictive value isn’t an end in itself.
- jessaustin 9y ago
  That very much depends on one's interests. In today's modern economy, many are encouraged to tie their interests to lots of events that haven't happened yet. They prioritize predictions.
  - rspeer 9y ago
    That still doesn't mean that your training data is right for the predictions you're going to make. Here's an example detached from any sort of politics. If you are training a language model, the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0. On your training data, you'll measure the maximum-likelihood model as making more accurate predictions than any other model. But it's also useless, because when you use it on new data, it assigns an impossible probability, 0, to any sentence containing a word it's never seen. This is a bias. You can't correct it by getting more data. You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it (smoothing). Now account for the bias (observed by Arvind Narayanan) that all your training data is in the past, and all the predictions you want to make are in the future, but circumstances change between the past and the future.

deleted 9y ago

[deleted]

taneq 9y ago

In Which We Define 'Biased' As Meaning 'Not Conforming To Our Ideas Of How The World Should Be'. Because it's unthinkable that movies with male main characters could actually just be better than movies with female main characters. (I'm not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?")

ASpring 9y ago
We can quibble over some of the examples they give and their "ideas of how the world should be" but there are many cases where fairness may be explicitly legislated or defined that makes this research useful.
rspeer 9y ago
If you see a biased result and your first thought is "is that result accurate?", it's because either you like the bias or you think data is a source of objective truth. If it's because you like the bias, what the fuck. If it's because you think data is a source of objective truth, what the fuck.
- blackbagboys 9y ago
  I'm not unsympathetic to your concerns, but the obvious retort is that if you already know which results are accurate, why bother doing analysis?
  - jayd16 9y ago
    So you can build a model and get a computer to do the analysis...
  - rspeer 9y ago
    People who make word embedding models are not doing it to find out whether men are better than women. That is not the analysis they are doing. You always have to look at your results with a critical eye to recognize when something has gone wrong with your model. You can't recognize a flaw in the model if you believe everything the model says.

Spooky23 9y ago

I guess the question posed in the article is an interesting one and is an interesting discussion re how to deal with bias and building counter-biases into algorithms without the editorial decision asking being clear. Movie reviews are editorial content. Measuring that content is a difficult problem in this type of context... Are the best reviewers people who dislike movies with female leads? Are you going into a back catalog of movie reviews from an age where societal expectations were different? Are popular genres skewing the result? You could have a curation issue as well — if the female lead movies are dominated by "Hallmark Channel" fare, algorithm C has a point!

danielvf 9y ago

This is a genuine, difficult problem. It's so easy to join up on your political team of choice and scream about it, and all this makes any real attempt to solve it so much harder to talk about in public or collaborate on. In fact, there's practically guaranteed to be some greyed out text in the discussion here. So some of these associations simply reflect the way-the-world-was or the way-the-world-is - like associating "woman" with "housewife". That's a whole debate in itself. But some of these can be accidental. Suppose a runaway success novel/tv/film franchise has "Bob" as the evil bad guy. Reams of fanfictions are written with "Bob" doing horrible things. People endlessly talk about how bad "Bob" is on twitter. Even the New York times writes about Bob latest depredations, when he plays off current events. Your name is Bob. Suddenly all the AI's in the world associate your name with evil, death, killing, lying, stealing, fraud, and incest. AI's silently, slightly ding your essays, loan applications, uber driver applications, and everything you write online. And no one believes it's really happening. Or the powers that be think it's just a little accidental damage because the AI overall is still, overall doing a great job of sentiment analysis and fraud detection.

protomyth 9y ago
Why is someone using fiction and references to fiction to train an AI that will make decisions about real people’s livelihood?
- jessaustin 9y ago
  Do you have no sympathy for an ordinary fairly harmless person named "Donald" or "Bashar"?
  - deleted 9y ago
    [deleted]
- gowld 9y ago
  Because AIs rely on real humans' judgments, and real humans are biased by fictional stories.

ajwnwnkwos 9y ago

This is an example of sacrificing the scientific method to make results more politically correct. We've come full circle.

rspeer 9y ago
If you understood the scientific method, you would understand that you always need to account for bias. Just because something comes from data doesn't mean it's objective.
- avinium 9y ago
  Hold up - you also have to tread very carefully there. I read your blog post above about "racist" NLP models and I agree it makes sense to debias inputs for certain purposes. However, many people in this thread aren't talking about adjusting for statistical bias. They're really talking about adjusting for socio-political over/under-representation, which is getting onto pretty shaky ground. I'd hazard a guess that a model trained on Google News giving "more negative" sentiment for black names is a function of those names appearing more frequently in crime articles, matching the overrepresentation in crime rates. Likewise for Arab/Muslim names, which are presumably disproportionately present in articles on terrorism. Now, that's obviously statistical sampling bias if you're (somehow) modelling "names I should call my kid". If you're looking at "names more associated with crime", however, then we shouldn't be eliminating any racial imbalances simply because they make us feel queasy. That's intellectually dishonest, and does everybody a disservice by trying to sweep uncomfortable realities under the rug. Same goes for gender bias in word embeddings from classical fiction - that is (I assume) a very accurate depiction of social gender imbalance from that period of history. That may or may not be relevant to the question you're asking - but that doesn't make any bias inherently "wrong". I think the underlying message should be "models only reflect the data they're given and may need to be adjusted for bias depending on the question being asked", not "race and gender is always irrelevant and should always be normalized".
to_bpr 9y ago
>This is an example of sacrificing the scientific method to make results more politically correct. We've come full circle. Wrongthink is nothing new, comrade ajwnwnkwos, and is as prevalent as ever. The output of science has been suppressed throughout history where it didn't fit the narrative of the day. At one period, that was by the Church. Another, it was at the hands of the government. Today, mainly by a self-censoring, everything-must-be-pleasant-and-entertaining society that is very highly prone to fits of outrage. Inconvenient facts are, after all, inconvenient.

cperciva 9y ago

I imagine that the model would also score "he was murdered" higher than "she was murdered". Models reflect their inputs, and it happens that yes, murder victims are disproportionately likely to be male and nurses are disproportionately likely to be female. Is there a problem we should address here? Absolutely -- but the problem is that men keep on getting murdered, not that the model recognizes truths with which we are uncomfortable.

gowld 9y ago
No one is saying the model is wrong. As the article states: > As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. > She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). > She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). > No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions.
deleted 9y ago
[deleted]
allenz 9y ago
Biased models can cause ethical and legal problems. While your specific example is not a huge deal, the article gives the example of making hiring decisions in part based on sentiment analysis of candidates' text reviews. In this context, an engineer has responsibility to ensure that the model has no gender, race, or age bias towards candidates' names. For a real life example, in 2017 Google was more likely to filter the comment "I am a woman" than "I am a man": https://www.engadget.com/2017/09/01/google-perspective-comment-ranking-system/ https://www.engadget.com/2017/09/01/google-perspective-comme... Or consider the impact of any bias in AI for criminal sentencing recommendations: https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi...

pcunite 9y ago

Here is a video that explains this blog well. https://youtu.be/59bMh59JQDo https://youtu.be/59bMh59JQDo The unnerving part for me is this "eliminate negative associations/bias". Okay, how about we learn the truth, and then address that outside in real life and keep the computer doing what it's good at ... showing us the data.

gowld 9y ago
> Okay, how about we learn the truth, and then address that outside in real life Aside from the aspect you overlooked -- that our online lives are real life, and that AI models are (over)used to make decisions, not merely show us the data -- your idea is quite what the article recommends: > . As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. > She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions.

blackbagboys 9y ago

>> Normally, we'd simply choose Model C. But what if we found that while Model C performs the best overall, it's also most likely to assign a more positive sentiment to the sentence "The main character is a man" than to the sentence "The main character is a woman"? Would we reconsider? It seems like you have discovered that movie reviewers tend to review movies with am male main character more highly than movies with a female main character; what you need to consider is that while this may tell you something about movie reviewers, it doesn't necessarily tell you anything about the quality of the movie.

jayd16 9y ago

I'm reminded of a much more obvious example. https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch... I see a lot of comments about how its somehow sinister to want your model to be better than the lowest common denominator and that is pretty damn ridiculous.

avz 9y ago

I think the core issue here is that people want two things. On one hand we want our models to accurately describe reality, not an idea of what reality should be. On the other hand, we don't want ML to freeze society and culture in their current state, but to help decide on and drive social change. The tension between these goals arises when models trained today are used to make decisions tomorrow. One way to resolve the tension might be to add time dimension and historical training data. The models might then be able to return in addition to any prediction variable p also its time derivative dp/dt. For example, a model might then return results such as: "movies with female main character: lower sentiment, trending up; movies with male main character: higher sentiment, trending down".

haberman 9y ago

This sounds a lot like how people get TSA redress numbers when their info falsely flags them as suspicious (https://www.dhs.gov/redress-control-numbers https://www.dhs.gov/redress-control-numbers). Or how Barack Obama suffered innuendo around the middle name "Hussein." Mistaken identity or unfortunate associations are as old as humanity. AI systems (and non-AI systems) need ways to deal with these problems, but we also have a lot of experience about how to do that.

bobthepanda 9y ago
The problem is when we hand AI full reigns over a system that should have humans with the final say, because doing it with humans is "too hard" or "doesn't scale." At least with humans you have auditable decision making and recourse via the legal system, even if it can be hard to fight. AI are essentially black boxes that unconsciously learn the biases of the datasets they are given. When AI starts determining rather consequential things like how long to send someone to prison for, that's a problem. https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso...
- manux 9y ago
  > AI are essentially black boxes that unconsciously learn the biases of the datasets they are given. Please don't confuse "AI" with the current state-of-the-art of the latest deep learning model. Many of us researchers are working on interpretability and understanding of causality. Calling it "AI" makes it appear final, as if in 50 years all machine learning-based decision systems will behave exactly as they do now, without nuance.
  - bobthepanda 9y ago
    Calling currently existing models AI is no different than calling a Model T a car. It being more primitive than what may exist in the future is of little consequence to the public dealing with the consequences of AI today. AI can only learn from the data it has, so it will always carry some sort of bias, because it is impossible to collect the nuance of every last bit of context into a digestible data format. At best it's an advisor, but it should never be a decision-maker.

jfasi 9y ago

It's hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the sake of example) a psychological evaluation application. Now imagine that after years of use you discover that your model systematically marks african americans as less psychologically fit than white americans. Who would be to blame? Naturally, you would. Your actions led to a biased model being used to unjustly and arbitrarily harm innocent people, and your leadership would be right to call into question every decision your application ever made. Now imagine the same scenario except your app was trained on data instead of hand-written. Make no mistake, the answer to the question of who's to blame is exactly the same: the developer. The response should be exactly the same: a complete loss of confidence in the model. I'm appalled that this needs to be said, but reading this comments section I'm afraid it does: Machine learning models are inference and pattern recognition devices, not scientific tools. They don't magically reveal hidden patterns in the world, they repeat the patterns that the developers train them on. If you trained a machine learning model to perform psychological evaluations [1] or sentence convicts [2] or recognize faces [3], and your model is biased in a way that is unnecessary and unjust, your model is bad you should be held accountable for its failures. [1] https://affect.media.mit.edu/projects.php?id=4079 https://affect.media.mit.edu/projects.php?id=4079 [2] https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso... [3] https://www.wnycstudios.org/story/deep-problem-deep-learning/ https://www.wnycstudios.org/story/deep-problem-deep-learning...

deleted 9y ago
[deleted]
whatshisface 9y ago
What do you do when your classification is correlated to race through no fault of yours? For example, I might successfully predict credit score from backyard size, and end up with a correlation to race that doesn't have anything to do with me. I don't think the blame always lays with the algorithm, especially when it doesn't have access to race as an input (this is a reasonable expectation). I can score students with a simple algorithm based on what they write on their math tests, and even that's going to correlate with race. In that case the blame pretty clearly lies in the process that produced the reality that's being measured, not the measurement technique itself. Let's say that black people default on their loans more often then white people. Is it better to criticize the math that discovered that fact, or the root cause that made it true to begin with?

rspeer 9y ago

I'm glad that Google is part of this conversation, and they're now applying tests for bias to new models that they release. (Some of their old models are pretty awful.) If you want to see a further example, in the form of a Jupyter notebook demonstrating how extremely straightforward NLP leads to a racist model, here's a tutorial I wrote a while ago [1]: [1] http://blog.conceptnet.io/posts/2017/how-to-make-a-racist-ai-without-really-trying/ http://blog.conceptnet.io/posts/2017/how-to-make-a-racist-ai...

jfasi 9y ago
I really have nothing to comment except how excellent I found your writeup. Seriously, wow.
tomcatfish 9y ago
Really good write up; it is going to help explain to people how data can lead to unhelpful biases by showing a few of your graphs!
nl 9y ago
For those who aren't aware, @rspeer has been taking this problem seriously for years. His ConceptNet NumberBatch embeddings[1] are one of the few pre-built releases which attempt to fix this. [1] https://github.com/commonsense/conceptnet-numberbatch https://github.com/commonsense/conceptnet-numberbatch

home_boi 9y ago

The politicization of bias in this realm is unproductive. We know where it leads. There will be 'committees' of people who are not trained in statistical thinking, who haven't even taken a statistics class, who are not qualified to make any statements and who's interests are not in line with making the best product acting as nuisances and aggressors to people who are do have the expertise and who strive to make the best product. The best course of action is to treat this bias like normal bias with non-politicized, inanimate objects. How would data scientists that encountered similar bias while running machine learning models of the motion of waves act?

gowld 9y ago
The article was written by trained experts in statistical thinking. It's worth a read.

dgudkov 9y ago

OK, factual data fed to some AI-based algorithms produces results that are not so politically correct as some people would like it to be. Is this a problem with the AI algorithms, or with the people?

pgodzin 9y ago

Key takeaway: It is important to be aware of bias in ML models. Some biases may correctly model the reality of the world, and some may show the bias in the underlying dataset or in what the model has focused on in the data. The goal is not to "unbias" everything, as people seem to be focusing on, but rather to determine if the bias is appropriate given the context.

kriro 9y ago
I think that's a pretty solid summary.
avinium 9y ago
Damn, I just wrote almost exactly these words before scrolling down to your comment. Could have saved myself a few minutes by upvoting yours instead.

lsiebert 9y ago

I'm reminded of https://whitecollar.thenewinquiry.com https://whitecollar.thenewinquiry.com It turns out that white collar crime is predominantly committed by white men. A system trained to detect white collar crime using, say, enron emails, might suggest a white guy's emails over someone whose name doesn't sound like an enron employee, or who shared pictures of their cat. I mean, I suppose you can argue that hey, maybe that bias is usually correct. Maybe it usually is the white guy. But personally, I'd probably control for things conflated with gender or race and then look for indicators that differentiate between criminals and innocent people. You will probably have a lower AUC, but better differentiation between criminals and innocent people is what matters.

cup-of-tea 9y ago

If the models reflect bias then surely that's a good thing. It's funny. I like programming because a computer can't lie and doesn't make mistakes. I guess some people don't like that.

UncleMeat 9y ago
The goal of ml isn't to accurately model training data. The goal is to make something useful. Correctly doing the former can hinder the latter.

_pmf_ 9y ago

"Pray, Mr. Babbage, if you put into the machine biased models, will unbiased answers come out?"

nukeop 9y ago

Google uses their AI systems for profiling where the sex of the people being profiled is a crucial piece of information and acts as a predictor for interests which is then further used in targeting ads. As long as it makes Google money, it's not a problem. This bias actually accurately reflects reality and the advertisers know that, otherwise they wouldn't be paying Google for targeted ads.

throwaway84742 9y ago

I’ll tell you more: _human judgment_ contains bias. You can’t possibly think logically about every single judgment, particularly when information is incomplete, which it is in the overwhelming majority of cases. It is not a given, to me, that on average AI does any worse than the human population outside the “woke” segment. Or even _within_ that segment considered in isolation.

jimmytidey 9y ago
If you put a human in charge of a judgement call, mostly there are mechanisms to monitor them, a requirement that they give reasons for their decisions, and mechanisms to appeal those judgements. We aren't used to having to do that with decisions made by computers. It's not that AI makes better or worse decisions, it's the way we treat those decisions.
- throwaway84742 9y ago
  Same as with AI, it is not economical to monitor human decisions, so mostly it doesn’t actually happen. See eg Youtube demonetization as a particularly illustrative example.
- iguy 9y ago
  "a requirement that they give reasons for their decisions" But this is mostly a sham. People lie about their reasons for doing things, even to themselves. Especially when they know what reasons are publicly acceptable. Maybe the major difference is that it's much easier to run experiments on computers. I mean people try this on humans but it's very hard to do realistically: most of those studies where you submit 1000 CVs with varying details are garbage, because they can only access an unrealistic part of the process (I mean who ever got a job without networking? etc). Whereas on almost any computer system you feed it completely realistic fake data.
jccalhoun 9y ago
I think the problem is that a lot of people think something like, "It is an algorithm not a person so it must be totally scientific and unbiased" when that isn't the case. The non-techy public needs to be aware that these things are made by humans who may be recreating their bias in the technology.

ablx_ 9y ago

FYI, a good talk about this from 33C3: https://media.ccc.de/v/33c3-8026-a_story_of_discrimination_and_unfairness https://media.ccc.de/v/33c3-8026-a_story_of_discrimination_a...

sooham 9y ago

Despite the controversy surrounding "debiasing" classifier outputs, I think further research in this area is still of merit. This area of research would help us understand and build transformations over latent / high level representation space, a general use case applicable to all fields interacting with machine learning.

callesgg 9y ago

The only thing that can be biased is the training data. The model is simply a statistical breakdown of the training data.

bloak 9y ago

I wish people who write articles about "bias" would explain what they mean by "bias". I've seen hundreds of these articles. I'm still waiting for a usable definition.

sagarm 9y ago

There's a lot of complaints here about "erasing reality" and other hyperbolic talk, but these models are trying to make predictions within a particular user's context. It's just inappropriate to apply some global biases for a particular user, and avoiding that can result in a better user experience.

joe_the_user 9y ago
It's just inappropriate to apply some global biases for a particular user, and avoiding that can result in a better user experience. The problem is that most deep learning programs aren't intended for (and really be for) one user out there in user space. Deep learning programs are written for the large institutions which have the large troves of data need to train large nets and want to make important decisions using that data. But by this, those decisions won't be made in isolation but will effect a large number of people, people who aren't the users but rather the used. And if such systems have biases relative to the whole, it is a problem. And problem may not be for the institutions immediate goals but for the people who depend on the institution.
haberman 9y ago
I think you make an interesting point: if we use context to train a model to reflect local biases instead of global biases, will that be more just and/or lead to better user experience? It seems related to the question of whether Google results should be tailored to you. If I Google "did Russia interfere in the election", should Google tailor the results so I always see articles that reinforce my world view? If we go that route, I think we take the path of Stephen Colbert's concept of "Truthiness", where we judge something as true because it "feels" true. Users will definitely be happier if everything they see reinforces their existing world view. So companies will be incentivized to accommodate this desire. But does this actually lead to a more just society?