28 ms·

Unexpected sources of bias in artificial intelligence

94 points by Parbeyjr 10y ago

vanderZwan 10y ago
> We tend to think of machines, in particular smart machines, as somehow cold, calculating and unbiased. We believe that self-driving cars will have no preference during life or death decisions between the driver and a random pedestrian. We trust that smart systems performing credit assessments will ignore everything except the genuinely impactful metrics, such as income and FICO scores. And we understand that learning systems will always converge on ground truth because unbiased algorithms drive them. I absolutely hate this myth, and I suspect it's just an excuse used by technologists (consciously or not) to wash their hands clean of taking social responsibility. There is no such thing as innocent technology, because it is made by and for humans, and with a purpose. Which means that before we even built anything we have already left the domain of "is" and entered "ought". Technology "encodes" values by being an answer to how we attempt to implement some intended purpose, and thus becomes entwined with the questions of responsibility that we all struggle with. Here's a great anthology of writings on that topic: Inside the Politics of Technology[0], mostly written from a post-phenomenological[1] perspective. Excerpt from the introduction: > It was eventually proven that the pilot acted as capably as possible, so he was not to blame. But that only partly settled the question. After this first clarification, however important it was to the pilot, the accident could still be ascribed either to human error, e.g., false instructions from the control tower, poor maintenance, or lack of knowledge about the migration cory patterns of birds, or, alternatively, to technical deficiencies, e.g., fuel problems or engine failure. > In the end, a year after the crash, an official research report established the “real” cause of the crash: the snapping-off of a 10-cm metal pin that regulated the position of a fin in one of the 13 cogs in the rotor of the F-16. This set off a chain reaction demolishing the cogs one by one, ending up in a complete breakdown of the engine. > But even then the problem of humanity versus technology was not solved. Who could be blamed for this technical defect: service engineers, the Air Force, or the manufacturer? The latter was ultimately left holding the bag. But still then, was it a production fault or a designer’s error? Again,different actors and different technicalities are involved. > Apparently, a definitive dividing line between technical and human causes cannot be drawn. However technical the cause of the crash appeared to be, human beings always come along with the technicalities – and vice versa. Purely technical causes are just as illusory as purely human faults. The idea of machines being unbiased and innocent is not just naive, but outright damaging. [0] Free PDF: https://oapen.org/download?type=document&docid=340207 https://oapen.org/download?type=document&docid=340207 [1] A friend of mine explained post-phenomenology as follows: "My understanding is that for Verbeek and others like him in general, postphenomenology continues the core traditions of phenomenology but with a deliberate emphasis on social science methods and empiricism over philosophical arguments - a positivist Heideggerian phenomenology"
- canjobear 10y ago
  I believe 'idlewords put it best: machine learning is money laundering for bias.
- andrewboooo 10y ago
  Are there any other writings you could point me to? I've never heard of post-phenomenology, but I quite like the perspective. The concept of AI certainly isn't an unfamiliar one, so I imagine there'd be some interesting takes on that -- hopefully from a realistic, non-AGI perspective.
  - vanderZwan 10y ago
    Hmm.. Well, if I'm completely honest I can't say I dove too deep into the material; this just happened to be part of the theory/philosophy that I had to study during my design bachelor/master that really resonated with me. But "What Things Do Philosophical Reflections on Technology, Agency, and Design" by Peter-Paul Verbeek is a touchpoint when it comes to post-phenomenology, AFAIK. http://www.psupress.org/books/titles/0-271-02539-5.html http://www.psupress.org/books/titles/0-271-02539-5.html
interpol_p 10y ago
Here is a paper titled Automated Inference on Criminality using Face Images [0]. The system is trained on existing conviction records. Inheriting all the prejudices and biases of our current justice system. [0] https://arxiv.org/abs/1611.04135 https://arxiv.org/abs/1611.04135
- yummyfajitas 10y ago
  What are the appearance-based prejudices and biases of the Chinese justice system? It would be great to hear about this; most of us are not from China and are unfamiliar with such things.
  - devoply 10y ago
    Probably being a Uighur or Tibetan in their respective regions regions. Also faces might correspond to certain disadvantaged ethnicities or groups. Or they might even correspond to a genotype that is expressed as a phenotype, so a genetic basis. We have identified a certain gene that makes aggression much more likely. And apparently it's more prevalent in African Americans in the US than others. But even that has context. In certain countries beating someone up, i.e. physical assault, is glossed over. Crime is highly subjective in terms of what people are willing to put up with... anything more than thou shalt not kill is highly subjective, and even that's ignored by military all around the world all the time. Good quote: Indeed, that was an apt and true reply which was given to Alexander the Great by a pirate who had been seized. For when that king had asked the man what he meant by keeping hostile possession of the sea, he answered with bold pride, "What do you mean by seizing the whole earth; because I do it with a petty ship, I am called a robber, while you who does it with a great fleet are styled emperor".
    - im3w1l 10y ago
      They "controlled for race, gender, age and facial expressions". I wonder how well they did it though, it seems to may that the non-criminal faces in Fig 10, have a happier expression.
    - tjalfi 10y ago
yummyfajitas 10y ago
I think we need to stop using the word "bias" and "prejudice". The term "bias" has a real, well-defined meaning in statistics, and both of these terms imply incorrect decisions are being systematically made. Only one example in this article is actually about incorrect decisions. Most examples are about unbiased learning systems doing exactly what they are designed to do, but people other than the designers wishing they did different things. For example, consider "conflicting goals bias". This isn't a bias - the system is maximizing clicks exactly as desired. It's just that some random third party wishes the system were actually trying to mitigate a nonexistent psychological problem (namely stereotype threat) instead [1]. What they call "similarity bias" is the same thing. The system is attempting to show people stories they like (and probably does a good job at it), but the author wishes instead that the selection of stories was closer to what a journalist might choose. Another social "bias" - namely "redlining"/redundant encoding/etc is actually the elimination of statistical bias. A lot of input metrics - e.g. SAT score, FICO score, etc - are biased in favor of non-Asian minorities (and against Asians) [2] and machine learning algorithms designed to find hidden patterns discover this bias and fix it. Conflating "someone is doing X but I wish they did Y" with bias is not useful. It's also not useful to conflate the elimination of socially desirable biases with introducing new biases. [1] Stereotype threat has repeatedly failed to replicate. Funnel plots suggest it only existed to begin with due to publication bias. http://www.sciencedirect.com.sci-hub.cc/science/article/pii/S0022440514000831 http://www.sciencedirect.com.sci-hub.cc/science/article/pii/... https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_replicate_and_publication_bias https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_... https://replicationindex.wordpress.com/tag/stereotype-threat-and-womens-math-performance/ https://replicationindex.wordpress.com/tag/stereotype-threat... [2] See for example Figure 7 here: https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ/view https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ...
- deleted 10y ago
  [deleted]
- zkhalique 10y ago
  Yeah, the word bias seems to be overloaded with many meanings. I actually wonder if there is any philosophical difference between the word "knowledge" and the word "bias". But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights. The problem is that we don't know whether the human decisions matched the reality. "Did the person commit the crime" for example. We might have to wait until more unbiased estimators for such activity come along, and throw away old historical data. It's sort of like when Black-Scholes became a self-fulfilling prophecy for valuing derivatives, but only after it became widespread. The market started using Black-Scholes to value derivatives, so it became the best model to predict the value of derivatives. But until then, other models might have fit the historical data better.
gosubpl 10y ago
Obligatory: go to http://www.catb.org/jargon/html/ http://www.catb.org/jargon/html/ and search for "Sussman attains enlightenment"
- Noseshine 10y ago
  Direct link: http://www.catb.org/jargon/html/koans.html#id3141241 http://www.catb.org/jargon/html/koans.html#id3141241
- Chris2048 10y ago
  is the take-away: bias is not in the initial conditions of the net, but in the implementation of the neurons and its structure; closing my eyes makes my eyes insensitive to the state of the room, it does not make the room empty; randomizing my network makes it insensitive to the biases of the AI, it does not mean there are no biases. I'm finding it hard to figure this one out...
  - AnimalMuppet 10y ago
    Bias is in the initial conditions of the net. It's just not deliberately biased by the author. But it still has bias. "Closing your eyes" to the initial content/state of the net does not make the initial state go away.
    - GFischer 10y ago
      Thank you both for the explanations, I saw the top comment (the Koan) and I didn't know what to make of it :)
dsjoerg 10y ago
You can gain insight into human problems by noting that these sources of bias all pertain to natural intelligence as well.
- tomquin 10y ago
  This. Should not come as a great surprise that AI is inherently biased, as are the people who create it and the people who use it. Is it possible to not be affected by so many nested layers of bias?
  - Chris2048 10y ago
    The bias of the people need't bleed into the AI, since the whole point is that AI isn't constructed by experts, but inferred from collected data.
    - prolways 10y ago
      The worry is that the training data has been affected by human bias, so that bias will bleed over into the ai.
jakosz 10y ago
I struggle to see how data-driven bias is unexpected.
_pdp_ 10y ago
Junk in Junk out - it applies for both machine and human learning.
antoinevg 10y ago
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied. "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?" Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.
- justinpombrio 10y ago
  With risk of spoiling the Haiku, could someone explain this in more detail? What is Sussman suggesting that Minsky do instead: wire the neural net in a fixed way instead of randomly? What does that have to do with the difference between viewing the room with your eyes open vs. closed?
  - JMStewy 10y ago
    The koan is commonly accompanied by the following: "What I actually said was, 'If you wire it randomly, it will still have preconceptions of how to play. But you just won't know what those preconceptions are.'" -- Marvin Minsky
UhUhUhUh 10y ago
The funny thing about bias is that it was invented, technically, as a way of... improving the accuracy of data in audio recordings. By adding something (current or frequency) to the signal you could increase the quality of the output. Without chasing the metaphor too much, it's also interesting to considerer that, maybe, adding something, yes a bias, to the data would also increase their quality. Why did authors (e.g. Azimov) came up with fundamental robotic laws?
- TeMPOraL 10y ago
  > Why did authors (e.g. Azimov) came up with fundamental robotic laws? To criticize them by highlighting their failings in the stories they wrote, in order to show us friendly AI is not an easy problem?
tudorw 10y ago
Cognitive bias cheat sheet :) https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a472476b18 https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a...
slowm5 10y ago
these comments got real deep, real quick.

94 points by Parbeyjr 10y ago

> We tend to think of machines, in particular smart machines, as somehow cold, calculating and unbiased. We believe that self-driving cars will have no preference during life or death decisions between the driver and a random pedestrian. We trust that smart systems performing credit assessments will ignore everything except the genuinely impactful metrics, such as income and FICO scores. And we understand that learning systems will always converge on ground truth because unbiased algorithms drive them. I absolutely hate this myth, and I suspect it's just an excuse used by technologists (consciously or not) to wash their hands clean of taking social responsibility. There is no such thing as innocent technology, because it is made by and for humans, and with a purpose. Which means that before we even built anything we have already left the domain of "is" and entered "ought". Technology "encodes" values by being an answer to how we attempt to implement some intended purpose, and thus becomes entwined with the questions of responsibility that we all struggle with. Here's a great anthology of writings on that topic: Inside the Politics of Technology[0], mostly written from a post-phenomenological[1] perspective. Excerpt from the introduction: > It was eventually proven that the pilot acted as capably as possible, so he was not to blame. But that only partly settled the question. After this first clarification, however important it was to the pilot, the accident could still be ascribed either to human error, e.g., false instructions from the control tower, poor maintenance, or lack of knowledge about the migration cory patterns of birds, or, alternatively, to technical deficiencies, e.g., fuel problems or engine failure. > In the end, a year after the crash, an official research report established the “real” cause of the crash: the snapping-off of a 10-cm metal pin that regulated the position of a fin in one of the 13 cogs in the rotor of the F-16. This set off a chain reaction demolishing the cogs one by one, ending up in a complete breakdown of the engine. > But even then the problem of humanity versus technology was not solved. Who could be blamed for this technical defect: service engineers, the Air Force, or the manufacturer? The latter was ultimately left holding the bag. But still then, was it a production fault or a designer’s error? Again,different actors and different technicalities are involved. > Apparently, a definitive dividing line between technical and human causes cannot be drawn. However technical the cause of the crash appeared to be, human beings always come along with the technicalities – and vice versa. Purely technical causes are just as illusory as purely human faults. The idea of machines being unbiased and innocent is not just naive, but outright damaging. [0] Free PDF: https://oapen.org/download?type=document&docid=340207 https://oapen.org/download?type=document&docid=340207 [1] A friend of mine explained post-phenomenology as follows: "My understanding is that for Verbeek and others like him in general, postphenomenology continues the core traditions of phenomenology but with a deliberate emphasis on social science methods and empiricism over philosophical arguments - a positivist Heideggerian phenomenology"

canjobear 10y ago
I believe 'idlewords put it best: machine learning is money laundering for bias.
andrewboooo 10y ago
Are there any other writings you could point me to? I've never heard of post-phenomenology, but I quite like the perspective. The concept of AI certainly isn't an unfamiliar one, so I imagine there'd be some interesting takes on that -- hopefully from a realistic, non-AGI perspective.
- vanderZwan 10y ago
  Hmm.. Well, if I'm completely honest I can't say I dove too deep into the material; this just happened to be part of the theory/philosophy that I had to study during my design bachelor/master that really resonated with me. But "What Things Do Philosophical Reflections on Technology, Agency, and Design" by Peter-Paul Verbeek is a touchpoint when it comes to post-phenomenology, AFAIK. http://www.psupress.org/books/titles/0-271-02539-5.html http://www.psupress.org/books/titles/0-271-02539-5.html

interpol_p 10y ago

Here is a paper titled Automated Inference on Criminality using Face Images [0]. The system is trained on existing conviction records. Inheriting all the prejudices and biases of our current justice system. [0] https://arxiv.org/abs/1611.04135 https://arxiv.org/abs/1611.04135

yummyfajitas 10y ago
What are the appearance-based prejudices and biases of the Chinese justice system? It would be great to hear about this; most of us are not from China and are unfamiliar with such things.
- devoply 10y ago
  Probably being a Uighur or Tibetan in their respective regions regions. Also faces might correspond to certain disadvantaged ethnicities or groups. Or they might even correspond to a genotype that is expressed as a phenotype, so a genetic basis. We have identified a certain gene that makes aggression much more likely. And apparently it's more prevalent in African Americans in the US than others. But even that has context. In certain countries beating someone up, i.e. physical assault, is glossed over. Crime is highly subjective in terms of what people are willing to put up with... anything more than thou shalt not kill is highly subjective, and even that's ignored by military all around the world all the time. Good quote: Indeed, that was an apt and true reply which was given to Alexander the Great by a pirate who had been seized. For when that king had asked the man what he meant by keeping hostile possession of the sea, he answered with bold pride, "What do you mean by seizing the whole earth; because I do it with a petty ship, I am called a robber, while you who does it with a great fleet are styled emperor".
  - im3w1l 10y ago
    They "controlled for race, gender, age and facial expressions". I wonder how well they did it though, it seems to may that the non-criminal faces in Fig 10, have a happier expression.
  - tjalfi 10y ago

yummyfajitas 10y ago

I think we need to stop using the word "bias" and "prejudice". The term "bias" has a real, well-defined meaning in statistics, and both of these terms imply incorrect decisions are being systematically made. Only one example in this article is actually about incorrect decisions. Most examples are about unbiased learning systems doing exactly what they are designed to do, but people other than the designers wishing they did different things. For example, consider "conflicting goals bias". This isn't a bias - the system is maximizing clicks exactly as desired. It's just that some random third party wishes the system were actually trying to mitigate a nonexistent psychological problem (namely stereotype threat) instead [1]. What they call "similarity bias" is the same thing. The system is attempting to show people stories they like (and probably does a good job at it), but the author wishes instead that the selection of stories was closer to what a journalist might choose. Another social "bias" - namely "redlining"/redundant encoding/etc is actually the elimination of statistical bias. A lot of input metrics - e.g. SAT score, FICO score, etc - are biased in favor of non-Asian minorities (and against Asians) [2] and machine learning algorithms designed to find hidden patterns discover this bias and fix it. Conflating "someone is doing X but I wish they did Y" with bias is not useful. It's also not useful to conflate the elimination of socially desirable biases with introducing new biases. [1] Stereotype threat has repeatedly failed to replicate. Funnel plots suggest it only existed to begin with due to publication bias. http://www.sciencedirect.com.sci-hub.cc/science/article/pii/S0022440514000831 http://www.sciencedirect.com.sci-hub.cc/science/article/pii/... https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_replicate_and_publication_bias https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_... https://replicationindex.wordpress.com/tag/stereotype-threat-and-womens-math-performance/ https://replicationindex.wordpress.com/tag/stereotype-threat... [2] See for example Figure 7 here: https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ/view https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ...

deleted 10y ago
[deleted]
zkhalique 10y ago
Yeah, the word bias seems to be overloaded with many meanings. I actually wonder if there is any philosophical difference between the word "knowledge" and the word "bias". But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights. The problem is that we don't know whether the human decisions matched the reality. "Did the person commit the crime" for example. We might have to wait until more unbiased estimators for such activity come along, and throw away old historical data. It's sort of like when Black-Scholes became a self-fulfilling prophecy for valuing derivatives, but only after it became widespread. The market started using Black-Scholes to value derivatives, so it became the best model to predict the value of derivatives. But until then, other models might have fit the historical data better.

gosubpl 10y ago

Obligatory: go to http://www.catb.org/jargon/html/ http://www.catb.org/jargon/html/ and search for "Sussman attains enlightenment"

Noseshine 10y ago
Direct link: http://www.catb.org/jargon/html/koans.html#id3141241 http://www.catb.org/jargon/html/koans.html#id3141241
Chris2048 10y ago
is the take-away: bias is not in the initial conditions of the net, but in the implementation of the neurons and its structure; closing my eyes makes my eyes insensitive to the state of the room, it does not make the room empty; randomizing my network makes it insensitive to the biases of the AI, it does not mean there are no biases. I'm finding it hard to figure this one out...
- AnimalMuppet 10y ago
  Bias is in the initial conditions of the net. It's just not deliberately biased by the author. But it still has bias. "Closing your eyes" to the initial content/state of the net does not make the initial state go away.
  - GFischer 10y ago
    Thank you both for the explanations, I saw the top comment (the Koan) and I didn't know what to make of it :)

dsjoerg 10y ago

You can gain insight into human problems by noting that these sources of bias all pertain to natural intelligence as well.

tomquin 10y ago
This. Should not come as a great surprise that AI is inherently biased, as are the people who create it and the people who use it. Is it possible to not be affected by so many nested layers of bias?
- Chris2048 10y ago
  The bias of the people need't bleed into the AI, since the whole point is that AI isn't constructed by experts, but inferred from collected data.
  - prolways 10y ago
    The worry is that the training data has been affected by human bias, so that bias will bleed over into the ai.

jakosz 10y ago

I struggle to see how data-driven bias is unexpected.

_pdp_ 10y ago

Junk in Junk out - it applies for both machine and human learning.

antoinevg 10y ago

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied. "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said. Minsky then shut his eyes. "Why do you close your eyes?" Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.

justinpombrio 10y ago
With risk of spoiling the Haiku, could someone explain this in more detail? What is Sussman suggesting that Minsky do instead: wire the neural net in a fixed way instead of randomly? What does that have to do with the difference between viewing the room with your eyes open vs. closed?
- JMStewy 10y ago
  The koan is commonly accompanied by the following: "What I actually said was, 'If you wire it randomly, it will still have preconceptions of how to play. But you just won't know what those preconceptions are.'" -- Marvin Minsky

UhUhUhUh 10y ago

The funny thing about bias is that it was invented, technically, as a way of... improving the accuracy of data in audio recordings. By adding something (current or frequency) to the signal you could increase the quality of the output. Without chasing the metaphor too much, it's also interesting to considerer that, maybe, adding something, yes a bias, to the data would also increase their quality. Why did authors (e.g. Azimov) came up with fundamental robotic laws?

TeMPOraL 10y ago
> Why did authors (e.g. Azimov) came up with fundamental robotic laws? To criticize them by highlighting their failings in the stories they wrote, in order to show us friendly AI is not an easy problem?

tudorw 10y ago

Cognitive bias cheat sheet :) https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a472476b18 https://betterhumans.coach.me/cognitive-bias-cheat-sheet-55a...

slowm5 10y ago

these comments got real deep, real quick.