28 ms·

A.I. Systems Echo Biases They’re Fed, Putting Scientists on Guard

113 points by wglb 7y ago

shantly 7y ago
Am I wrong in thinking that "echo the biases they're fed" is a pretty complete, concise description for what "AI"/ML systems are supposed to do?
- wpasc 7y ago
  Kind of. AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. Echoing the biases they're fed is probably more akin to overfitting to their training data.
  - shantly 7y ago
    A glance over some dictionaries reveals I'm leaning on an essentially archaic and poetic sense of "bias" here to evade its meaning exclusively an unreasonable judgement, it seems.
  - Doingmything123 7y ago
    I guess the question is how well to we expect AI/ML systems to be able to generalize on a biased subset data. If you are trying to use one of these systems to make hiring decisions but you train it on data from the hires you made, should we really be surprised if biases in the original hiring(education level, gender, race, income, personality) are reflected in the AI's decisions?
  - bsanr2 7y ago
    The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. If the results are odious to us, it should be impetus to critically analyze not only the AI/ML systems, but also the underlying assumptions that they're built on. Instead, developers become defensive and cage-y about their processes. If you don't want systems to have disparate impact, you have to be adamant about it in your design. If you think society is better off with systems that reflect preexisting biases, then fine, but be ready for the backlash. At the end of the day, it really is up to what humans want to do with themselves. It's an opportunity to be truer to our intent, not a bug to be covered up.
    - im3w1l 7y ago
      > The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.
      - wan23 7y ago
        > When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
      - bsanr2 7y ago
        Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
  - gambler 7y ago
    >AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. ...which means that whether a model is "biased" depends on where and how it's applied. This is an important point that is missing from most discussions, articles and even research papers on the so-called "AI ethics".
    - int_19h 7y ago
      Such a model is objectively biased regardless of where and how it's applied - it prefers X over Y, and that's bias by definition. The hard question here is deciding whether any given bias is ethical or not.
  - rcthompson 7y ago
    No, it's not the same as overfitting. It's nothing to do with the algorithm at all. The problem is with the training data, which contains patterns induced by the way the data was collected rather than the reality of what you want to classify. For example, imagine that you wanted to train an algorithm to distinguish photos of dogs from photos humans. So you collect a bunch of photos of both dogs and humans and use them to train a classifier. You do all the proper cross-validation, bootstrapping, etc. to ensure that you are not overfitting, and you get really good results. Then, looking at the mis-classifications, you notice something: all the photos that are taken looking at an angle down toward the ground are classified as dog photos, and all the photos taken looking straight ahead are classified as human photos. It turns out that in your training set, most of the dog photos are taken at a downward angle while must of the human photos are taken facing straight ahead, because humans are taller than dogs, and your machine learning algorithm identified this feature as the most reliable way to distinguish the two groups of photos in your training set. In this hypothetical example, no overfitting occurred. The difference in photo angles is a real difference in the training sets that you provided to the algorithm, and the algorithm did its job and correctly identified this difference between the two groups of photos as a reliable predictor. The problem is that your training set has a variable (photo angle) that is highly correlated with what you want to classify (species). This is considered an unwanted bias (and not a reliable indicator) because the correlation is caused by the means of data collection (most photos are taken from human head height) and has nothing to do with the subject of the photos.
    - CardenB 7y ago
      I think you’re arguing semantics a bit. What you’re saying checks out, but one could say that overfitting was occurring but the test dataset distribution was not wide enough to catch it.
      - etbebl 7y ago
        As I understand it, if it's overfitting, testing on a second random sample gathered in the same way as the training set should degrade performance. That's not the case here. (Though maybe the term as used in industry is less strict.)
  - haberman 7y ago
    The test for overfitting is whether the model holds up for data outside the training set. If the biases are consistent with other real-world data, then it's not overfitting.
  - deleted 7y ago
    [deleted]
- kempbellt 7y ago
  This is exactly what they are designed to do. The scary part is, they can do so on a different level than your average human. Pouring over larger sets of data quickly, than you or I have time to consume in our entire lives. The largest issue, however, is that an AI can find and shed light on dislikable realities of the world. Racism, sexism, culturism, opposing political opinions... Perspectives that are not "PC" still exist and permeate the digital world along with the physical one. Creating unbiased data is, imo, impossible, as I am also biased, and so are you. I don't know what unbiased data is. I can certainly say that I have held racist, sexist, religious, and political views at various stages of my life - based on small sample set data, and biased trainers. I have grown a better understanding and no longer hold many of the naive beliefs that I held when I was younger, and will continue to realize how ignorant I am as I live. The same process will probably happen for any AI.
  - loopz 7y ago
    It's good to hear you are now no longer ignorant! :-D
    - kempbellt 7y ago
      "No longer ignorant", I certainly am not. Thank you for the positive reinforcement though. Becoming less ignorant is a life-long process, as far as I can tell. I'm just somewhat more aware of my capacity to over-generalize based on my individual experiences, and allow biases to settle in my subconscious in the form of racism, sexism, ageism, or what have you. I try to find where I have internalized these thoughts so I can do a little internal reforming. I also pay a lot more attention to "_______ is/are _______" statements, as they are almost always over-generalizations. Being aware of this mental mechanism doesn't really stop me from doing it though. I know I'll tend to cluster experiences to create generalizations indefinitely, as it seems to be an evolved trait (makes sense for survival reasons, to assume the worst until you find evidence otherwise) - even though it's not perfect.
  - repolfx 7y ago
    Creating unbiased data is, imo, impossible, as I am also biased, and so are you What about just learning based on the entire web? I think you're using the word "unbiased" to mean "heavily adjusted for US centric views on racism and sexism" which isn't what the word really means. If you train an AI on everything written - all books, all web pages, all newspaper articles etc ... a not impossible task these days - then you can argue you're as close to bias free as possible. However a small number of AI researchers don't like the results they get when they do this, because the AI learns the world that truly exists instead of the one they wish would exist. But that's not a bug in the software. It's a bug in the researchers.
- thrower123 7y ago
  There is often confusion about what AI systems are, and what people think they should be. What they are is Bayesian black boxes that use pretty basic statistics and probabilities. What people think they should be is far, far more complicated and nebulous. I'm not sure I fully understand, but people have been fed a lot of nonsense about AI systems, from Deep Blue to Watson to AlphaGo, etc, etc, showing them as being very powerful in limited domains, and extrapolating that out into overestimations of what they could do. The other main problem is that people seem to think that these AI systems will be a complete replacement for human thought and decision-making, which, frankly, knowing even a little about how the sausage of software is made, is completely terrifying.
- smeeth 7y ago
  It really depends on the specifics of the optimization problem in question. If ML is trained on biased data where an optimality exists only at an unbiased solution (think a shaped reward function in RL to disincentivize class bias or something similar), then no, the ML is most certainly not supposed to echo the bias they've been fed. On the other hand, if an optimal solution to the ML optimization problem exists at a biased solution, ex. a naive prediction of if a nurse is a man or a woman, then yeah, we would say that it was supposed to echo the bias. All too often I feel people forget that ML is just an optimization problem. What you're trying to optimize really matters - generalizing about all ML without talking about the optimization problem in question is pointless.
- 6gvONxR4sf7o 7y ago
  Nope. If I feed an NLP model a bunch of shakespeare, it'll "echo the biases it's fed," but it won't do what you probably wanted it to. A less silly example is feeding the system a bunch of wall street journal articles and then trying to use it on informal communications like twitter. The goal is to learn something that solves a particular problem. It turns out that the current methodology solves problems in a biased way.
- Accujack 7y ago
  It is, except that people are discovering that the programs faithfully learn from their training material, even including traits and biases present that are not consciously included by the person that selected the training data. It seems a lot of people don't realize that there's likely a large disconnect between what people think they're training an AI to do and what they're actually training it to do. If they're lucky, the two are close enough that the learning program will be useful for what was hoped, and if they're unlucky it will look like it's useful and correct but will include behavior learned from the training data that was not predicted and which taints the result. IE, if you train an AI with what you think is unbiased data but which includes a subtle bias, then the results you get from it may be biased... and the bias may be so subtle it's undetectable except by another AI, which is a problem if you assume that since your training data was unbiased, your AI must be unbiased. Putting it another way... garbage in, garbage out.
- jdashg 7y ago
  One big problem is there are biases we're supposed to ignore, but we're not good at (able to?) ensuring this deliberate blindness in ML. For example, we're generally not allowed to discriminate based on gender, but even without a gender field, ML will happily imply a gender out of its firehose of other data, and discriminate based on it anyway. "AI finds that employees are less likely to stay at the company if their names end in 'a' and they score higher on 'cultivation' rubrics" This same AI then recommends hiring applicant 2 over applicant 1. Is this allowable?
  - lotu 7y ago
    Are we suggesting that humans don’t do the same thing?
  - nitwit005 7y ago
    A bit of an aside, but humans do what you describe and have name letter preferences: https://en.m.wikipedia.org/wiki/Name-letter_effect https://en.m.wikipedia.org/wiki/Name-letter_effect
    - dmix 7y ago
      We have lots of inherent biological biases like that.
      - repolfx 7y ago
        Why would a name letter bias be biological caused? Or do you mean bias about biological factors? Bear in mind "bias" is just a perjorative for "generalisation" or alternatively "lesson learned". AI algorithms are good at detecting patterns in data and bad at being politically correct. This is not a flaw of the algorithms, it's a flaw in people who can't accept measured reality and go into denial. So an AI trying to hire programmers discriminates against female sounding names, because it's learned that this is correlated with success? Apply it to hiring nurses or primary school teachers and it'll probably do the opposite. This is only "bias" if you start from ideologically driven blank slate assumptions. Otherwise it's just common sense.
- buboard 7y ago
  more generally, don't we expect General AI systems to be racist if they have racist parents? That's just proof that the AI works human-like.
- m0zg 7y ago
  Such articles are crafted to create outrage among the uninitiated. They like to pretend that "bias" did not exist before, and AI is introducing it. Every time we, as humans, make any decision, we're "echoing biases we're fed". It's 100% intractable to rationally think about every decision, so we almost never bother, and when we do, we second guess like crazy because rational decisions often do not agree with our biases. And guess what, most of the time our biases are _right_. It wouldn't make any sense from an energy conservation standpoint to have them otherwise. What we should be aiming for instead is not the total lack of bias (an impossibility if you are to learn anything at all from the data), but _explainability_. A system must be able to show me the set of statistics that led to a particular decision. I.e. women in Los Angeles area are known to be much safer drivers according to this subset of data, so we offer a lower insurance rate to women in Los Angeles area, to use just one hypothetical example. Such systems are a rarity nowadays, and research into them is relatively sparse.
pixelbath 7y ago
TFA is not responding and Google's cache is showing a 404. So, at the risk of a post that's purely snark: "In other news, scientists discover water is wet."
Doingmything123 7y ago
Ironically, I think this shows how human-like we have been able to make AI systems.
- AstralStorm 7y ago
  Not entirely. Humans are able to use well described and backed logic in decision making. Ever seen AI write out its decision logic, in a form that's portable to other AI? People past few years old can output at least partial rationale for behavior or decisions. Systems like BERT are at best comparable to a pre-linguistic 2 year old.
  - Doingmything123 7y ago
    I admit that it's unfortunate that AI can't write out their decision logic but I would argue that is because there hasn't been enough resources put into explainable AI. Considering the increasing use of these algorithms, I don't know if that is even a high priority. I tend to think that people are not as logical as they like to think they are, myself included. Not to say there isn't good reasoning, just that much of our decision making is emotional and habitual over some pure sense of logic. Systems like BERT seem perfectly rational to me. Are they not just following a set of rules on a given input to modify a state?(In the most simplistic sense of computation). I think the confusion is more over what the goal of these programs are and how do we encode that. This reminds me of the ai system that would pause the game of tetris so that it could never lose. Not we it's programmers intended but still accomplished it's "goal".
    - bluGill 7y ago
      While people are not as logical as we like to be, we all are logical. I can teach someone the rules of math - my method of teaching might be (probably is) bad, but if the student tries he will learn those rules. Latter on when given a test the student can show his work and it will be much the same as every other student trying to explain his reasoning: operations like "complete the squares" have been well described and reasoned out. Likewise chess masters can explain their thought process while looking at the next move and other chess masters will agree the lines of thought are good (they will probably ask why not some other equally good line...). We know this explanation is good because students can watch the experts explain their thought process and replicate it in games to a small extent and to better.
40acres 7y ago
Good. I hope that AI can be used as a mirror for the engineers working with it. AI allows amplification of the subtle assumptions we make in design and hopefully that amplification leads to better understand and appropriate measures to reduce bias.
- strken 7y ago
  Less a mirror of the engineers working on it and more a mirror of the data it's fed, which is a really important difference. Nobody thinks "today I'll create a model that hates women", they just aim their shiny new model at e.g. every New York Times article since 1851, then only check for the results they're interested in.
acd 7y ago
I am concerned about AI algorithms and especially recommendations algorithms in regards to biases. If someone is in a temporary psychological condition. The ad recommendation algorithms will keep reinforcing that bias behavior in order to maximize ad revenue. As long as the viewer is hooked on the site and keep coming back for more content. The algorithms have little or no sense what bias they are reinforcing. I wish there was more public debate and awareness from programmers about that. Ie we have moral obligations on what algorithms recommend.
- x220 7y ago
  Engineers at Instagram, Facebook dot com, Youtube, and Twitter DO NOT want to think about how they are affecting the mental health of their users. All they want to do is spout some platitudes and wash their hands of all responsibility.
jimmaswell 7y ago
Wayback link: https://web.archive.org/web/20191113162629/https://www.oodaloop.com/technology/2019/11/12/a-i-systems-echo-biases-theyre-fed-putting-scientists-on-guard/ https://web.archive.org/web/20191113162629/https://www.oodal...
humanistbot 7y ago
"Garbage In, Garbage Out" has long been a foundational saying in computer science. I'm genuinely curious why so many in this generation of AI researchers and developers seem to think they're immune to this.
- LeifCarrotson 7y ago
  > On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. > Passages from the Life of a Philosopher (1864), ch. 5 "Difference Engine No. 1" A very long time indeed! I think the key source of confusion (both in 1864 and today) is that non-technical people hope that the machine has some wisdom correlated with its arithmetic prowess. Charles Babbage's mechanical calculator did not have wisdom. Even when you give a bleeding-edge algorithm Big Data and lots of processing power, it doesn't yet have wisdom. But hope springs eternal.
manfredo 7y ago
Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? For instance, people have detected that AI models associate technical words with men more often than women and point to this as evidence of bias. I would argue that this is the opposite situation. There's social stigma attached to acknowledging differences between groups, so people have developed biased against acknowledging those differences. The AI on the other hand sees that 70-80% of technologists are men, and associates accordingly. So the problem isn't that the AI is biased it's that it lacks the biases we expect. Now, there are good arguments why this bias may be a good thing. Unbiased does not automatically mean good.
- dmix 7y ago
  That's true, a lot social activists are very much in favour of putting biases in systems. That's also pretty much what most gov policy is, various competing special interests fighting to intervene in various systems in a favourable direction.
- bilbo0s 7y ago
  Wait? Isn't "biased in a way we don't expect", itself, a bias? Unless I'm misunderstanding you, you're saying the AI is biased, and people are biased. Unbiased is an AI that will not miss an outlier. My field is medical imaging. The most difficult thing to get the AI vendors that come calling on us to understand is that being able to recognize well known patterns in data is of limited use to us. That's what the radiologists at remote sites do all day. It's being able to recognize pathologies that 99.999% of rads would definitely have missed that has value. And the AI's always fail this test. I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. I've been going to RSNA for decades now, and I would say we haven't seen a single AI that could be said to pass this kind of a test yet.
  - pigscantfly 7y ago
    I've been working in data science or in academia for most of the last decade and would argue that the primary benefits of using statistical models for automation are that they are fast and cheap compared to a trained professional, which frees those trained professionals up to spend more time on the difficult judgment calls. I've seen very few examples of superhuman performance on complex, practical tasks (although it is very satisfying to train a model to that level).
swebs 7y ago
>and is more likely to associate men with computer programming So would any reasonable human. I think the real problem these advocacy groups have with AI models is that they're not biased. They reflect the real world based on data and evidence, rather than conforming to progressive dogma. I fear that instead of using AI to overcome any wrong assumptions we may have, we're just going to get AI diversity officers to "correct" models that draw any uncomfortable conclusions.
- rolltiide 7y ago
  Right, it is cognitively negligent to ignore these things, at least in the context of what an exact machine will tell you. That doesn't mean accept it, it means to implement the exceptions and a way to update them or understand sensitivities
  - loopz 7y ago
    The whole point of Machine Learning is that you don't, and shouldn't, implement exceptions, manual weights, any of that complex parametrization, etc. The machine should figure all that out itself, even if it needs millions of internal parameters to do it! Sounds neat, until you trust it with your life.
    - rolltiide 7y ago
      ah yes the cultured machine learning proponents and their reputation for picking up social cues
- 40acres 7y ago
  We know that boys and girls are conditioned by society to be more interested in certain fields than others. It's why my cousin asked me to get a robot for her son's birthday as opposed to a dress for her daughter. Do we want our biases embedded in AI? Or do we want a model that hopefully can eliminate some of the subjectivity that humans add (which in some cases we absolutely need)?
  - 7y ago
ropiwqefjnpoa 7y ago
I'm curious if some of these are actual discriminatory bias' or just harmless cultural norms. Something like associating men with football can hardly be considered bias.
- loopz 7y ago
  It goes both ways: Even though you remove the sex/gender variables, the system could correlate football to identify men, and sockermoms!
kempbellt 7y ago
Try and wrap your mind around the idea of an "unbiased" AI. It is not possible. As a trainer of an AI, simply looking at a picture and saying that it's a cat, and then telling the AI that it's a cat, is biased. You are automatically assuming you are correctly identifying it as a cat, and teaching the mind of an AI to follow your presumption. The only way to make an AI less biased (but still biased), is to diversify your labeled training data. And by diversify, you need to diversify the presumptions on the labels of the data themselves. What is the confidence among several people that this picture is of a cat? Rather than one trainer. This is a simplistic example. If you scale it to more complicated tasks, such as understanding natural language, body language, intent of users, or anything related to ethics, you'll likely end up with a crazy AI. And yes, I mean crazy. As in, simultaneously holding very seemingly opposing understandings of reality and spitting out data that reflects those oppositions. If you've ever had more than one authoritative figure in your life give you contradicting advice to another - one parent holding differing beliefs to another, or one teacher to another in school, you may notice yourself starting to qualify your teacher's competence as a factor in the fidelity of their teachings. A scary, but simultaneously cool thought: An AI that questions my capacity to teach it, while I'm teaching it. Bias will always remain an element in the equation however. One AI will likely have interacted with different trainers or data, and create contradicting understandings of reality to that of another AI.
- crooked-v 7y ago
  "Duck or rabbit" is another simple example that would require bias in visual interpretation to reliably decide. https://www.independent.co.uk/news/science/duck-or-rabbit-the-100-year-old-optical-illusion-that-tells-you-how-creative-you-are-a6873106.html https://www.independent.co.uk/news/science/duck-or-rabbit-th...
  - kempbellt 7y ago
    Hah. Thank you for this reminder. This is a great example to illustrate the point. I would say that a less biased (wiser) AI, would learn that this duck/rabbit image is, itself, questionable, and can be perceived both ways (high confidence values for both identifiers). Me: Is this a duck or a rabbit, AI? Padawan AI: It's a rabbit (51% confidence as rabbit. 49% as a duck). Master Yoda AI: Rethink your question, you must. Yes, the most correct answer is.
Digit-Al 7y ago
Is there an opening here for some enterprising persons to try to develop systems to try to determine the biases present in training data?
dhairya 7y ago
The oodaloop article is a summary of the NYtimes (https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-bias.html https://www.nytimes.com/2019/11/11/technology/artificial-int...) article which is more detailed. Also pleasantly surprised in the NYTimes accurate description of what BERT was trained to do (masked word prediction and next sentence prediction) and the implications of that. I have worked closely with BERT and other language models for our startup. There is a disconnect between the capabilities and the state of AI research today and the public's expectations and imagination. There's also fundamental confusion between scale and intelligence. That is the public largely believes the efficacy of many "AI" models on large scale problems is equivalent to intelligence. That assumption is problematic in both overestimating the capabilities of these models and misdirecting the focus of critical inquiry. Hopefully, there is more education around the limitations and capabilities of these technologies. We should be more cautious in apply them to usecases where there is high potential of negative consequences.
dmtroyer 7y ago
the new york times article that this links to: https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-bias.html https://www.nytimes.com/2019/11/11/technology/artificial-int...
kazinator 7y ago
Scientists should be sophisticated computer users who understand the GIGO (garbage-in, gabrage-out) principle; that's what should put them on guard with regard to any manner of massaging their data.
bobloblaw45 7y ago
This reminds of of Tay from a few years ago.
Proven 7y ago
So? Why would anyone create an AI system that doesn’t work the way they want it to work? What a ridiculous article and topic.

113 points by wglb 7y ago

Am I wrong in thinking that "echo the biases they're fed" is a pretty complete, concise description for what "AI"/ML systems are supposed to do?

wpasc 7y ago
Kind of. AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. Echoing the biases they're fed is probably more akin to overfitting to their training data.
- shantly 7y ago
  A glance over some dictionaries reveals I'm leaning on an essentially archaic and poetic sense of "bias" here to evade its meaning exclusively an unreasonable judgement, it seems.
- Doingmything123 7y ago
  I guess the question is how well to we expect AI/ML systems to be able to generalize on a biased subset data. If you are trying to use one of these systems to make hiring decisions but you train it on data from the hires you made, should we really be surprised if biases in the original hiring(education level, gender, race, income, personality) are reflected in the AI's decisions?
- bsanr2 7y ago
  The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. If the results are odious to us, it should be impetus to critically analyze not only the AI/ML systems, but also the underlying assumptions that they're built on. Instead, developers become defensive and cage-y about their processes. If you don't want systems to have disparate impact, you have to be adamant about it in your design. If you think society is better off with systems that reflect preexisting biases, then fine, but be ready for the backlash. At the end of the day, it really is up to what humans want to do with themselves. It's an opportunity to be truer to our intent, not a bug to be covered up.
  - im3w1l 7y ago
    > The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.
    - wan23 7y ago
      > When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
    - bsanr2 7y ago
      Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
- gambler 7y ago
  >AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. ...which means that whether a model is "biased" depends on where and how it's applied. This is an important point that is missing from most discussions, articles and even research papers on the so-called "AI ethics".
  - int_19h 7y ago
    Such a model is objectively biased regardless of where and how it's applied - it prefers X over Y, and that's bias by definition. The hard question here is deciding whether any given bias is ethical or not.
- rcthompson 7y ago
  No, it's not the same as overfitting. It's nothing to do with the algorithm at all. The problem is with the training data, which contains patterns induced by the way the data was collected rather than the reality of what you want to classify. For example, imagine that you wanted to train an algorithm to distinguish photos of dogs from photos humans. So you collect a bunch of photos of both dogs and humans and use them to train a classifier. You do all the proper cross-validation, bootstrapping, etc. to ensure that you are not overfitting, and you get really good results. Then, looking at the mis-classifications, you notice something: all the photos that are taken looking at an angle down toward the ground are classified as dog photos, and all the photos taken looking straight ahead are classified as human photos. It turns out that in your training set, most of the dog photos are taken at a downward angle while must of the human photos are taken facing straight ahead, because humans are taller than dogs, and your machine learning algorithm identified this feature as the most reliable way to distinguish the two groups of photos in your training set. In this hypothetical example, no overfitting occurred. The difference in photo angles is a real difference in the training sets that you provided to the algorithm, and the algorithm did its job and correctly identified this difference between the two groups of photos as a reliable predictor. The problem is that your training set has a variable (photo angle) that is highly correlated with what you want to classify (species). This is considered an unwanted bias (and not a reliable indicator) because the correlation is caused by the means of data collection (most photos are taken from human head height) and has nothing to do with the subject of the photos.
  - CardenB 7y ago
    I think you’re arguing semantics a bit. What you’re saying checks out, but one could say that overfitting was occurring but the test dataset distribution was not wide enough to catch it.
    - etbebl 7y ago
      As I understand it, if it's overfitting, testing on a second random sample gathered in the same way as the training set should degrade performance. That's not the case here. (Though maybe the term as used in industry is less strict.)
- haberman 7y ago
  The test for overfitting is whether the model holds up for data outside the training set. If the biases are consistent with other real-world data, then it's not overfitting.
- deleted 7y ago
  [deleted]
kempbellt 7y ago
This is exactly what they are designed to do. The scary part is, they can do so on a different level than your average human. Pouring over larger sets of data quickly, than you or I have time to consume in our entire lives. The largest issue, however, is that an AI can find and shed light on dislikable realities of the world. Racism, sexism, culturism, opposing political opinions... Perspectives that are not "PC" still exist and permeate the digital world along with the physical one. Creating unbiased data is, imo, impossible, as I am also biased, and so are you. I don't know what unbiased data is. I can certainly say that I have held racist, sexist, religious, and political views at various stages of my life - based on small sample set data, and biased trainers. I have grown a better understanding and no longer hold many of the naive beliefs that I held when I was younger, and will continue to realize how ignorant I am as I live. The same process will probably happen for any AI.
- loopz 7y ago
  It's good to hear you are now no longer ignorant! :-D
  - kempbellt 7y ago
    "No longer ignorant", I certainly am not. Thank you for the positive reinforcement though. Becoming less ignorant is a life-long process, as far as I can tell. I'm just somewhat more aware of my capacity to over-generalize based on my individual experiences, and allow biases to settle in my subconscious in the form of racism, sexism, ageism, or what have you. I try to find where I have internalized these thoughts so I can do a little internal reforming. I also pay a lot more attention to "_______ is/are _______" statements, as they are almost always over-generalizations. Being aware of this mental mechanism doesn't really stop me from doing it though. I know I'll tend to cluster experiences to create generalizations indefinitely, as it seems to be an evolved trait (makes sense for survival reasons, to assume the worst until you find evidence otherwise) - even though it's not perfect.
- repolfx 7y ago
  Creating unbiased data is, imo, impossible, as I am also biased, and so are you What about just learning based on the entire web? I think you're using the word "unbiased" to mean "heavily adjusted for US centric views on racism and sexism" which isn't what the word really means. If you train an AI on everything written - all books, all web pages, all newspaper articles etc ... a not impossible task these days - then you can argue you're as close to bias free as possible. However a small number of AI researchers don't like the results they get when they do this, because the AI learns the world that truly exists instead of the one they wish would exist. But that's not a bug in the software. It's a bug in the researchers.
thrower123 7y ago
There is often confusion about what AI systems are, and what people think they should be. What they are is Bayesian black boxes that use pretty basic statistics and probabilities. What people think they should be is far, far more complicated and nebulous. I'm not sure I fully understand, but people have been fed a lot of nonsense about AI systems, from Deep Blue to Watson to AlphaGo, etc, etc, showing them as being very powerful in limited domains, and extrapolating that out into overestimations of what they could do. The other main problem is that people seem to think that these AI systems will be a complete replacement for human thought and decision-making, which, frankly, knowing even a little about how the sausage of software is made, is completely terrifying.
smeeth 7y ago
It really depends on the specifics of the optimization problem in question. If ML is trained on biased data where an optimality exists only at an unbiased solution (think a shaped reward function in RL to disincentivize class bias or something similar), then no, the ML is most certainly not supposed to echo the bias they've been fed. On the other hand, if an optimal solution to the ML optimization problem exists at a biased solution, ex. a naive prediction of if a nurse is a man or a woman, then yeah, we would say that it was supposed to echo the bias. All too often I feel people forget that ML is just an optimization problem. What you're trying to optimize really matters - generalizing about all ML without talking about the optimization problem in question is pointless.
6gvONxR4sf7o 7y ago
Nope. If I feed an NLP model a bunch of shakespeare, it'll "echo the biases it's fed," but it won't do what you probably wanted it to. A less silly example is feeding the system a bunch of wall street journal articles and then trying to use it on informal communications like twitter. The goal is to learn something that solves a particular problem. It turns out that the current methodology solves problems in a biased way.
Accujack 7y ago
It is, except that people are discovering that the programs faithfully learn from their training material, even including traits and biases present that are not consciously included by the person that selected the training data. It seems a lot of people don't realize that there's likely a large disconnect between what people think they're training an AI to do and what they're actually training it to do. If they're lucky, the two are close enough that the learning program will be useful for what was hoped, and if they're unlucky it will look like it's useful and correct but will include behavior learned from the training data that was not predicted and which taints the result. IE, if you train an AI with what you think is unbiased data but which includes a subtle bias, then the results you get from it may be biased... and the bias may be so subtle it's undetectable except by another AI, which is a problem if you assume that since your training data was unbiased, your AI must be unbiased. Putting it another way... garbage in, garbage out.
jdashg 7y ago
One big problem is there are biases we're supposed to ignore, but we're not good at (able to?) ensuring this deliberate blindness in ML. For example, we're generally not allowed to discriminate based on gender, but even without a gender field, ML will happily imply a gender out of its firehose of other data, and discriminate based on it anyway. "AI finds that employees are less likely to stay at the company if their names end in 'a' and they score higher on 'cultivation' rubrics" This same AI then recommends hiring applicant 2 over applicant 1. Is this allowable?
- lotu 7y ago
  Are we suggesting that humans don’t do the same thing?
- nitwit005 7y ago
  A bit of an aside, but humans do what you describe and have name letter preferences: https://en.m.wikipedia.org/wiki/Name-letter_effect https://en.m.wikipedia.org/wiki/Name-letter_effect
  - dmix 7y ago
    We have lots of inherent biological biases like that.
    - repolfx 7y ago
      Why would a name letter bias be biological caused? Or do you mean bias about biological factors? Bear in mind "bias" is just a perjorative for "generalisation" or alternatively "lesson learned". AI algorithms are good at detecting patterns in data and bad at being politically correct. This is not a flaw of the algorithms, it's a flaw in people who can't accept measured reality and go into denial. So an AI trying to hire programmers discriminates against female sounding names, because it's learned that this is correlated with success? Apply it to hiring nurses or primary school teachers and it'll probably do the opposite. This is only "bias" if you start from ideologically driven blank slate assumptions. Otherwise it's just common sense.
buboard 7y ago
more generally, don't we expect General AI systems to be racist if they have racist parents? That's just proof that the AI works human-like.
m0zg 7y ago
Such articles are crafted to create outrage among the uninitiated. They like to pretend that "bias" did not exist before, and AI is introducing it. Every time we, as humans, make any decision, we're "echoing biases we're fed". It's 100% intractable to rationally think about every decision, so we almost never bother, and when we do, we second guess like crazy because rational decisions often do not agree with our biases. And guess what, most of the time our biases are _right_. It wouldn't make any sense from an energy conservation standpoint to have them otherwise. What we should be aiming for instead is not the total lack of bias (an impossibility if you are to learn anything at all from the data), but _explainability_. A system must be able to show me the set of statistics that led to a particular decision. I.e. women in Los Angeles area are known to be much safer drivers according to this subset of data, so we offer a lower insurance rate to women in Los Angeles area, to use just one hypothetical example. Such systems are a rarity nowadays, and research into them is relatively sparse.

pixelbath 7y ago

TFA is not responding and Google's cache is showing a 404. So, at the risk of a post that's purely snark: "In other news, scientists discover water is wet."

Doingmything123 7y ago

Ironically, I think this shows how human-like we have been able to make AI systems.

AstralStorm 7y ago
Not entirely. Humans are able to use well described and backed logic in decision making. Ever seen AI write out its decision logic, in a form that's portable to other AI? People past few years old can output at least partial rationale for behavior or decisions. Systems like BERT are at best comparable to a pre-linguistic 2 year old.
- Doingmything123 7y ago
  I admit that it's unfortunate that AI can't write out their decision logic but I would argue that is because there hasn't been enough resources put into explainable AI. Considering the increasing use of these algorithms, I don't know if that is even a high priority. I tend to think that people are not as logical as they like to think they are, myself included. Not to say there isn't good reasoning, just that much of our decision making is emotional and habitual over some pure sense of logic. Systems like BERT seem perfectly rational to me. Are they not just following a set of rules on a given input to modify a state?(In the most simplistic sense of computation). I think the confusion is more over what the goal of these programs are and how do we encode that. This reminds me of the ai system that would pause the game of tetris so that it could never lose. Not we it's programmers intended but still accomplished it's "goal".
  - bluGill 7y ago
    While people are not as logical as we like to be, we all are logical. I can teach someone the rules of math - my method of teaching might be (probably is) bad, but if the student tries he will learn those rules. Latter on when given a test the student can show his work and it will be much the same as every other student trying to explain his reasoning: operations like "complete the squares" have been well described and reasoned out. Likewise chess masters can explain their thought process while looking at the next move and other chess masters will agree the lines of thought are good (they will probably ask why not some other equally good line...). We know this explanation is good because students can watch the experts explain their thought process and replicate it in games to a small extent and to better.

40acres 7y ago

Good. I hope that AI can be used as a mirror for the engineers working with it. AI allows amplification of the subtle assumptions we make in design and hopefully that amplification leads to better understand and appropriate measures to reduce bias.

strken 7y ago
Less a mirror of the engineers working on it and more a mirror of the data it's fed, which is a really important difference. Nobody thinks "today I'll create a model that hates women", they just aim their shiny new model at e.g. every New York Times article since 1851, then only check for the results they're interested in.

acd 7y ago

I am concerned about AI algorithms and especially recommendations algorithms in regards to biases. If someone is in a temporary psychological condition. The ad recommendation algorithms will keep reinforcing that bias behavior in order to maximize ad revenue. As long as the viewer is hooked on the site and keep coming back for more content. The algorithms have little or no sense what bias they are reinforcing. I wish there was more public debate and awareness from programmers about that. Ie we have moral obligations on what algorithms recommend.

x220 7y ago
Engineers at Instagram, Facebook dot com, Youtube, and Twitter DO NOT want to think about how they are affecting the mental health of their users. All they want to do is spout some platitudes and wash their hands of all responsibility.

jimmaswell 7y ago

Wayback link: https://web.archive.org/web/20191113162629/https://www.oodaloop.com/technology/2019/11/12/a-i-systems-echo-biases-theyre-fed-putting-scientists-on-guard/ https://web.archive.org/web/20191113162629/https://www.oodal...

humanistbot 7y ago

"Garbage In, Garbage Out" has long been a foundational saying in computer science. I'm genuinely curious why so many in this generation of AI researchers and developers seem to think they're immune to this.

LeifCarrotson 7y ago
> On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. > Passages from the Life of a Philosopher (1864), ch. 5 "Difference Engine No. 1" A very long time indeed! I think the key source of confusion (both in 1864 and today) is that non-technical people hope that the machine has some wisdom correlated with its arithmetic prowess. Charles Babbage's mechanical calculator did not have wisdom. Even when you give a bleeding-edge algorithm Big Data and lots of processing power, it doesn't yet have wisdom. But hope springs eternal.

manfredo 7y ago

Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? For instance, people have detected that AI models associate technical words with men more often than women and point to this as evidence of bias. I would argue that this is the opposite situation. There's social stigma attached to acknowledging differences between groups, so people have developed biased against acknowledging those differences. The AI on the other hand sees that 70-80% of technologists are men, and associates accordingly. So the problem isn't that the AI is biased it's that it lacks the biases we expect. Now, there are good arguments why this bias may be a good thing. Unbiased does not automatically mean good.

dmix 7y ago
That's true, a lot social activists are very much in favour of putting biases in systems. That's also pretty much what most gov policy is, various competing special interests fighting to intervene in various systems in a favourable direction.
bilbo0s 7y ago
Wait? Isn't "biased in a way we don't expect", itself, a bias? Unless I'm misunderstanding you, you're saying the AI is biased, and people are biased. Unbiased is an AI that will not miss an outlier. My field is medical imaging. The most difficult thing to get the AI vendors that come calling on us to understand is that being able to recognize well known patterns in data is of limited use to us. That's what the radiologists at remote sites do all day. It's being able to recognize pathologies that 99.999% of rads would definitely have missed that has value. And the AI's always fail this test. I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. I've been going to RSNA for decades now, and I would say we haven't seen a single AI that could be said to pass this kind of a test yet.
- pigscantfly 7y ago
  I've been working in data science or in academia for most of the last decade and would argue that the primary benefits of using statistical models for automation are that they are fast and cheap compared to a trained professional, which frees those trained professionals up to spend more time on the difficult judgment calls. I've seen very few examples of superhuman performance on complex, practical tasks (although it is very satisfying to train a model to that level).

swebs 7y ago

>and is more likely to associate men with computer programming So would any reasonable human. I think the real problem these advocacy groups have with AI models is that they're not biased. They reflect the real world based on data and evidence, rather than conforming to progressive dogma. I fear that instead of using AI to overcome any wrong assumptions we may have, we're just going to get AI diversity officers to "correct" models that draw any uncomfortable conclusions.

rolltiide 7y ago
Right, it is cognitively negligent to ignore these things, at least in the context of what an exact machine will tell you. That doesn't mean accept it, it means to implement the exceptions and a way to update them or understand sensitivities
- loopz 7y ago
  The whole point of Machine Learning is that you don't, and shouldn't, implement exceptions, manual weights, any of that complex parametrization, etc. The machine should figure all that out itself, even if it needs millions of internal parameters to do it! Sounds neat, until you trust it with your life.
  - rolltiide 7y ago
    ah yes the cultured machine learning proponents and their reputation for picking up social cues
40acres 7y ago
We know that boys and girls are conditioned by society to be more interested in certain fields than others. It's why my cousin asked me to get a robot for her son's birthday as opposed to a dress for her daughter. Do we want our biases embedded in AI? Or do we want a model that hopefully can eliminate some of the subjectivity that humans add (which in some cases we absolutely need)?
- 7y ago

ropiwqefjnpoa 7y ago

I'm curious if some of these are actual discriminatory bias' or just harmless cultural norms. Something like associating men with football can hardly be considered bias.

loopz 7y ago
It goes both ways: Even though you remove the sex/gender variables, the system could correlate football to identify men, and sockermoms!

kempbellt 7y ago

Try and wrap your mind around the idea of an "unbiased" AI. It is not possible. As a trainer of an AI, simply looking at a picture and saying that it's a cat, and then telling the AI that it's a cat, is biased. You are automatically assuming you are correctly identifying it as a cat, and teaching the mind of an AI to follow your presumption. The only way to make an AI less biased (but still biased), is to diversify your labeled training data. And by diversify, you need to diversify the presumptions on the labels of the data themselves. What is the confidence among several people that this picture is of a cat? Rather than one trainer. This is a simplistic example. If you scale it to more complicated tasks, such as understanding natural language, body language, intent of users, or anything related to ethics, you'll likely end up with a crazy AI. And yes, I mean crazy. As in, simultaneously holding very seemingly opposing understandings of reality and spitting out data that reflects those oppositions. If you've ever had more than one authoritative figure in your life give you contradicting advice to another - one parent holding differing beliefs to another, or one teacher to another in school, you may notice yourself starting to qualify your teacher's competence as a factor in the fidelity of their teachings. A scary, but simultaneously cool thought: An AI that questions my capacity to teach it, while I'm teaching it. Bias will always remain an element in the equation however. One AI will likely have interacted with different trainers or data, and create contradicting understandings of reality to that of another AI.

crooked-v 7y ago
"Duck or rabbit" is another simple example that would require bias in visual interpretation to reliably decide. https://www.independent.co.uk/news/science/duck-or-rabbit-the-100-year-old-optical-illusion-that-tells-you-how-creative-you-are-a6873106.html https://www.independent.co.uk/news/science/duck-or-rabbit-th...
- kempbellt 7y ago
  Hah. Thank you for this reminder. This is a great example to illustrate the point. I would say that a less biased (wiser) AI, would learn that this duck/rabbit image is, itself, questionable, and can be perceived both ways (high confidence values for both identifiers). Me: Is this a duck or a rabbit, AI? Padawan AI: It's a rabbit (51% confidence as rabbit. 49% as a duck). Master Yoda AI: Rethink your question, you must. Yes, the most correct answer is.

Digit-Al 7y ago

Is there an opening here for some enterprising persons to try to develop systems to try to determine the biases present in training data?

dhairya 7y ago

The oodaloop article is a summary of the NYtimes (https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-bias.html https://www.nytimes.com/2019/11/11/technology/artificial-int...) article which is more detailed. Also pleasantly surprised in the NYTimes accurate description of what BERT was trained to do (masked word prediction and next sentence prediction) and the implications of that. I have worked closely with BERT and other language models for our startup. There is a disconnect between the capabilities and the state of AI research today and the public's expectations and imagination. There's also fundamental confusion between scale and intelligence. That is the public largely believes the efficacy of many "AI" models on large scale problems is equivalent to intelligence. That assumption is problematic in both overestimating the capabilities of these models and misdirecting the focus of critical inquiry. Hopefully, there is more education around the limitations and capabilities of these technologies. We should be more cautious in apply them to usecases where there is high potential of negative consequences.

dmtroyer 7y ago

the new york times article that this links to: https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-bias.html https://www.nytimes.com/2019/11/11/technology/artificial-int...

kazinator 7y ago

Scientists should be sophisticated computer users who understand the GIGO (garbage-in, gabrage-out) principle; that's what should put them on guard with regard to any manner of massaging their data.

bobloblaw45 7y ago

This reminds of of Tay from a few years ago.

Proven 7y ago

So? Why would anyone create an AI system that doesn’t work the way they want it to work? What a ridiculous article and topic.