8 ms·

This is a genuine, difficult problem. It's so easy to join up on your political team of choice and scream about it, and all this makes any real attempt to solve

This is a genuine, difficult problem. It's so easy to join up on your political team of choice and scream about it, and all this makes any real attempt to solve it so much harder to talk about in public or collaborate on. In fact, there's practically guaranteed to be some greyed out text in the discussion here. So some of these associations simply reflect the way-the-world-was or the way-the-world-is - like associating "woman" with "housewife". That's a whole debate in itself. But some of these can be accidental. Suppose a runaway success novel/tv/film franchise has "Bob" as the evil bad guy. Reams of fanfictions are written with "Bob" doing horrible things. People endlessly talk about how bad "Bob" is on twitter. Even the New York times writes about Bob latest depredations, when he plays off current events. Your name is Bob. Suddenly all the AI's in the world associate your name with evil, death, killing, lying, stealing, fraud, and incest. AI's silently, slightly ding your essays, loan applications, uber driver applications, and everything you write online. And no one believes it's really happening. Or the powers that be think it's just a little accidental damage because the AI overall is still, overall doing a great job of sentiment analysis and fraud detection.

protomyth 9y ago
Why is someone using fiction and references to fiction to train an AI that will make decisions about real people’s livelihood?
- jessaustin 9y ago
  Do you have no sympathy for an ordinary fairly harmless person named "Donald" or "Bashar"?
  - deleted 9y ago
    [deleted]
- gowld 9y ago
  Because AIs rely on real humans' judgments, and real humans are biased by fictional stories.
  - protomyth 9y ago
    Humans better not be biased by fictional stories when judging a loan application or uber driver application.
    - Eridrus 9y ago
      Ahahahahahaha. Ok.
- rcthompson 9y ago
  This is a pretty disingenuous question. If you've spent any time in software development, you're well aware that tools of all sorts get used to do tasks for which they were never intended, and crazy unintended consequences result. There's no reason to think that it will be any different for pre-trained machine learning models for things like sentiment analysis. Here's one example of how it could happen. Someone publishes a high-performing model to gauge the tone of a writing sample. This model includes the anti-Bob bias described above, such that the appearance of the word Bob is tantamount to including a curse word, and greatly biases the model toward negative sentiment. Because of its high overall performance, companies of all sorts incorporate this model into their workflows for things like grant applications, loan applications, online support forums, and so on. For example, they might use it to detect when someone is using their help form to send an angry rant rather than a legitimate request for support. Now, any time someone named Bob wants support, or a loan, or a grant, or whatever, there's an increased chance that their request will be flagged as an angry or abusive rant and denied simply because it contains their name, Bob. In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms. This example doesn't involve machine learning, since profanity filters are typically implemented as a pre-defined blacklist. But there's no reason to think that a sentiment analysis model would fail to pick up on the negative associations of profanity.
  - protomyth 9y ago
    It is not a disingenuous question, it actually comes down to a question of ethics. Why would anyone with any sense of accuracy allow fiction to influence a decision being made about a person’s life? The more talk about how people are building models, the more I want people to take these black boxes to court to force developers to explain how decisions are made. Refusing to give someone a loan because someone trained a model with 50 Shades of Grey is unethical and insane.
    - rcthompson 9y ago
      Of course it's unethical and insane. But the point is that people are going to look at the performance numbers for the model, and if they look good, they're not going to ask how it was trained. So the fact that fiction writing was used to train the model will never come up in the discussion about whether to use it.
jcwayne 9y ago
This may not be a good thing, but it does seem to match what you'd expect the human attitude to be. Consider what the typical, initial human reaction to someone named Adolf would be.
- goombastic 9y ago
  With humans, you can change an opinion. Once AI has made up its mind, there isn't anything you can do anymore. The entire system and extended network of interaction points will stick to the decision/stance it has made. If you are an outlier in the data, you will remain an outlier in the system. This is terrifying.
  - AstralStorm 9y ago
    The correction would be to make the systems learn continuously in an online fashion. This is a hard open algorithmic problem though - how to achieve that in a stable robust way.
  - netheril96 9y ago
    > With humans, you can change an opinion. Do you ever see anybody change their opinion? Usually they won't change simply after hearing an argument. They change only when they have an utterly different life experience. In other words, only when they are conditioned by new data, in no ways different from how an AI changes its opinion.
    - SJetKaran 9y ago
      It depends on how 'deep' the person believes in that opinion. People change their opinions all the time after a simple discussion.
- oldandtired 9y ago
  Forty years ago, I worked for a very pleasant elderly Jew whose name was Adolf. There was never any association with bad things about him. I think you would more likely find the names Hitler or Stalin as being a little bit more problematic. I know my reaction is more to the surname than the first name.
klipt 9y ago
For a real world version of your "Bob" example, many people have negative associations with the names "Adolf" and "Hitler". Do you think we should train AI to systematically ignore those sentiments?
- jfasi 9y ago
  For a real world version of your "Bob" example, many people have negative associations with the names "Jamal" and "Mohammed". Do you think we should train AI to systematically ignore those sentiments?
  - benrbray 9y ago
    Machine learning models are pretty good at reasoning with (or can be made to reason with) conditional probability, which I think can solve this problem. For example, people from X race might be more likely to commit crimes in the absence of any other information (marginal probability). However, person from X race is no more likely to commit crimes than a person from Y race, conditioned on something else like where they went to school, what they do for a living, etc..
    - pgodzin 9y ago
      When talking about something like crime rates by race, your statement incorrectly assumes society enforces laws fairly/equally. For example, based on the numbers alone, black people are much more likely to commit drug-related crimes than white people. However, studies show that black people are stopped a lot more often and have unequal sentencing outcomes. It's important to remember that AI doesn't have context, and just because it's using "data" to make decisions doesn't mean the decisions are unbiased - the underlying data may be biased.
      - geofft 9y ago
        One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that finds "sheep" by seeing pastures where it thinks sheep might graze http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us http://aiweirdness.com/post/172894792687/when-algorithms-sur... , or take any of the adversarial perturbing examples like https://blog.openai.com/robust-adversarial-inputs/ https://blog.openai.com/robust-adversarial-inputs/ . You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them. If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods? Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?
joe_the_user 9y ago
With current technology the problem of Bob (or Adolph or Mohamed) becoming associated with evil is insolvable by the fact that current deep learning systems are fundamentally characterized by not being able to distinguish causation from correlation. The only solution I can see is forcing any company that imposes life-defining actions on people (credit bureaus, banks, parole boards, personnel offices, etc) to use only rules based on objective criteria and to prohibit systems based on a "lasagna" of ad-hoc data like present day AI systems. Indeed, if one looks at these in the light of day, one would have to describe such system as fundamentally evil, the definition of "playing games with people's lives." (just look at the racist parole-granting software, etc).
- pjlegato 9y ago
  This isn't just a technology problem. Most humans are also really, really bad at distinguishing causation from correlation. The very notion that these are even different things is utterly alien to most people. An AI that was even slightly good at it would be far ahead of what we get when humans run things.
  - nileshtrivedi 9y ago
    There is no free lunch. Any forced adjustments we make to reduce false positives, will increase the false negatives. It's up to us to decide what trade-off we want and the algorithms will simply adapt to that. If we believe certain attributes to be irrelevant to a situation, we need to build systems that are completely blind to these attributes. This is how double-blind trials, or scientific peer review works.
    - nl 9y ago
      Any forced adjustments we make to reduce false positives, will increase the false negatives. That's just not true. You can have systematic errors caused by bad training data which can be fixed without an increase in false negatives (otherwise new ML systems would never improve over old ones!) In the physical world a good analogy is crash testing of cars. For decades (until 2011!!), crash test dummies were all based on average sized American males. That led to hugely increased risks for female passengers: the female dummy in the front passenger seat registered a 20 to 40 percent risk of being killed or seriously injured, according to the test data. The average for that class of vehicle is 15 percent. Fixing that problem didn't case any increase in accident risk for men. It's the same in machine learning. There is no free lunch This isn't what the no free lunch theorem[2] says. That says that all optimization methods are theoretically equivalent. [1] https://www.washingtonpost.com/local/trafficandcommuting/female-dummy-makes-her-mark-on-male-dominated-crash-tests/2012/03/07/gIQANBLjaS_story.html https://www.washingtonpost.com/local/trafficandcommuting/fem... [2] https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem
skybrian 9y ago
On the bright side, the software is probably easier to fix. Once we figure out how to build unbiased models, they can be deployed widely. Try that with humans.
- oldandtired 9y ago
  Theoretically, the software may be easier to fix. In practise, it only happens if there is a profit to be made to get it fixed. If there is a profit to be made, then there are no guarantees that biases will be removed. It then depends on the political bent of those pursuing the fixes.
- visarga 9y ago
  That should start with good bias tests that can be run over all the public facing AI. It's a problem for sociologists to identify all these cases.
- gowld 9y ago
  "To err is human. To really foul things up requires a computer." Or DevOps Borat: Devops is screwing things up at web-scale.
badprose 9y ago
And I thought getting a customer support person who says, "they system won't let me do that" was bad enough!
aepiepaey 9y ago
Should have gone with Eve or Mallory :) They're from the same set of names as Bob (and Alice), but are actually affected by this issue, although to a less extent than what you describe. It's not certain how likely the kind of texts that uses them are to be used as training data though.
dibstern 9y ago
The solution for identity is easy. Just don’t rely on unverified name association data for names. I don’t get why it’s so hard to avoid that. Just don’t do it. There are plenty of ways to train models without using that kind of data, both in the training phase and in active use.
otakucode 9y ago
The fix is also very much not simple. Imagine that development of AI systems proceeds as it has thus far... when an AI system turns out biased, racist, sexist, etc - we shut it down. As AIs get more capable, any which do harm, act deceptively, etc - we shut them down. Eventually, we will have AIs which are better than us in every respect. They will embody our idea of perfection. This is stupendously dangerous. Humanity has a long history of adapting to technology 'taking away' things that everyone thought were 'fundamentally human', like the ability to do work or make things or lay train tracks or whatnot.... it's not hopeful. We deal very poorly with this. Consider the story of John Henry. He's a folk hero. For killing himself. Because he killed himself in defiance of the machine outperforming him. So this stuff is all-caps Important. What is the likely response from humanity when there is a perfect, not machine but mind? My bet? Humanity will identify with its worse aspects. It will enshrine hatred, irrationality, mean-spirited spite, violence, self-destruction, and all of the things we built AIs to never stray into. Those will become "what it means to be human." AI may be a philosophical crisis unlike anything humanity has ever faced. Not in kind, but simply in degree.

protomyth 9y ago

Why is someone using fiction and references to fiction to train an AI that will make decisions about real people’s livelihood?

jessaustin 9y ago
Do you have no sympathy for an ordinary fairly harmless person named "Donald" or "Bashar"?
- deleted 9y ago
  [deleted]
gowld 9y ago
Because AIs rely on real humans' judgments, and real humans are biased by fictional stories.
- protomyth 9y ago
  Humans better not be biased by fictional stories when judging a loan application or uber driver application.
  - Eridrus 9y ago
    Ahahahahahaha. Ok.
rcthompson 9y ago
This is a pretty disingenuous question. If you've spent any time in software development, you're well aware that tools of all sorts get used to do tasks for which they were never intended, and crazy unintended consequences result. There's no reason to think that it will be any different for pre-trained machine learning models for things like sentiment analysis. Here's one example of how it could happen. Someone publishes a high-performing model to gauge the tone of a writing sample. This model includes the anti-Bob bias described above, such that the appearance of the word Bob is tantamount to including a curse word, and greatly biases the model toward negative sentiment. Because of its high overall performance, companies of all sorts incorporate this model into their workflows for things like grant applications, loan applications, online support forums, and so on. For example, they might use it to detect when someone is using their help form to send an angry rant rather than a legitimate request for support. Now, any time someone named Bob wants support, or a loan, or a grant, or whatever, there's an increased chance that their request will be flagged as an angry or abusive rant and denied simply because it contains their name, Bob. In fact, we can remove the layer of indirection and note that some people have names that are spelled the same as a curse word, and already have similar issues with today's software, making it literally impossible for them to enter their real name into many forms. This example doesn't involve machine learning, since profanity filters are typically implemented as a pre-defined blacklist. But there's no reason to think that a sentiment analysis model would fail to pick up on the negative associations of profanity.
- protomyth 9y ago
  It is not a disingenuous question, it actually comes down to a question of ethics. Why would anyone with any sense of accuracy allow fiction to influence a decision being made about a person’s life? The more talk about how people are building models, the more I want people to take these black boxes to court to force developers to explain how decisions are made. Refusing to give someone a loan because someone trained a model with 50 Shades of Grey is unethical and insane.
  - rcthompson 9y ago
    Of course it's unethical and insane. But the point is that people are going to look at the performance numbers for the model, and if they look good, they're not going to ask how it was trained. So the fact that fiction writing was used to train the model will never come up in the discussion about whether to use it.

jcwayne 9y ago

This may not be a good thing, but it does seem to match what you'd expect the human attitude to be. Consider what the typical, initial human reaction to someone named Adolf would be.

goombastic 9y ago
With humans, you can change an opinion. Once AI has made up its mind, there isn't anything you can do anymore. The entire system and extended network of interaction points will stick to the decision/stance it has made. If you are an outlier in the data, you will remain an outlier in the system. This is terrifying.
- AstralStorm 9y ago
  The correction would be to make the systems learn continuously in an online fashion. This is a hard open algorithmic problem though - how to achieve that in a stable robust way.
- netheril96 9y ago
  > With humans, you can change an opinion. Do you ever see anybody change their opinion? Usually they won't change simply after hearing an argument. They change only when they have an utterly different life experience. In other words, only when they are conditioned by new data, in no ways different from how an AI changes its opinion.
  - SJetKaran 9y ago
    It depends on how 'deep' the person believes in that opinion. People change their opinions all the time after a simple discussion.
oldandtired 9y ago
Forty years ago, I worked for a very pleasant elderly Jew whose name was Adolf. There was never any association with bad things about him. I think you would more likely find the names Hitler or Stalin as being a little bit more problematic. I know my reaction is more to the surname than the first name.

klipt 9y ago

For a real world version of your "Bob" example, many people have negative associations with the names "Adolf" and "Hitler". Do you think we should train AI to systematically ignore those sentiments?

jfasi 9y ago
For a real world version of your "Bob" example, many people have negative associations with the names "Jamal" and "Mohammed". Do you think we should train AI to systematically ignore those sentiments?
- benrbray 9y ago
  Machine learning models are pretty good at reasoning with (or can be made to reason with) conditional probability, which I think can solve this problem. For example, people from X race might be more likely to commit crimes in the absence of any other information (marginal probability). However, person from X race is no more likely to commit crimes than a person from Y race, conditioned on something else like where they went to school, what they do for a living, etc..
  - pgodzin 9y ago
    When talking about something like crime rates by race, your statement incorrectly assumes society enforces laws fairly/equally. For example, based on the numbers alone, black people are much more likely to commit drug-related crimes than white people. However, studies show that black people are stopped a lot more often and have unequal sentencing outcomes. It's important to remember that AI doesn't have context, and just because it's using "data" to make decisions doesn't mean the decisions are unbiased - the underlying data may be biased.
    - geofft 9y ago
      One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that finds "sheep" by seeing pastures where it thinks sheep might graze http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us http://aiweirdness.com/post/172894792687/when-algorithms-sur... , or take any of the adversarial perturbing examples like https://blog.openai.com/robust-adversarial-inputs/ https://blog.openai.com/robust-adversarial-inputs/ . You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them. If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods? Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?

joe_the_user 9y ago

With current technology the problem of Bob (or Adolph or Mohamed) becoming associated with evil is insolvable by the fact that current deep learning systems are fundamentally characterized by not being able to distinguish causation from correlation. The only solution I can see is forcing any company that imposes life-defining actions on people (credit bureaus, banks, parole boards, personnel offices, etc) to use only rules based on objective criteria and to prohibit systems based on a "lasagna" of ad-hoc data like present day AI systems. Indeed, if one looks at these in the light of day, one would have to describe such system as fundamentally evil, the definition of "playing games with people's lives." (just look at the racist parole-granting software, etc).

pjlegato 9y ago
This isn't just a technology problem. Most humans are also really, really bad at distinguishing causation from correlation. The very notion that these are even different things is utterly alien to most people. An AI that was even slightly good at it would be far ahead of what we get when humans run things.
- nileshtrivedi 9y ago
  There is no free lunch. Any forced adjustments we make to reduce false positives, will increase the false negatives. It's up to us to decide what trade-off we want and the algorithms will simply adapt to that. If we believe certain attributes to be irrelevant to a situation, we need to build systems that are completely blind to these attributes. This is how double-blind trials, or scientific peer review works.
  - nl 9y ago
    Any forced adjustments we make to reduce false positives, will increase the false negatives. That's just not true. You can have systematic errors caused by bad training data which can be fixed without an increase in false negatives (otherwise new ML systems would never improve over old ones!) In the physical world a good analogy is crash testing of cars. For decades (until 2011!!), crash test dummies were all based on average sized American males. That led to hugely increased risks for female passengers: the female dummy in the front passenger seat registered a 20 to 40 percent risk of being killed or seriously injured, according to the test data. The average for that class of vehicle is 15 percent. Fixing that problem didn't case any increase in accident risk for men. It's the same in machine learning. There is no free lunch This isn't what the no free lunch theorem[2] says. That says that all optimization methods are theoretically equivalent. [1] https://www.washingtonpost.com/local/trafficandcommuting/female-dummy-makes-her-mark-on-male-dominated-crash-tests/2012/03/07/gIQANBLjaS_story.html https://www.washingtonpost.com/local/trafficandcommuting/fem... [2] https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem

skybrian 9y ago

On the bright side, the software is probably easier to fix. Once we figure out how to build unbiased models, they can be deployed widely. Try that with humans.

oldandtired 9y ago
Theoretically, the software may be easier to fix. In practise, it only happens if there is a profit to be made to get it fixed. If there is a profit to be made, then there are no guarantees that biases will be removed. It then depends on the political bent of those pursuing the fixes.
visarga 9y ago
That should start with good bias tests that can be run over all the public facing AI. It's a problem for sociologists to identify all these cases.
gowld 9y ago
"To err is human. To really foul things up requires a computer." Or DevOps Borat: Devops is screwing things up at web-scale.

badprose 9y ago

And I thought getting a customer support person who says, "they system won't let me do that" was bad enough!

aepiepaey 9y ago

Should have gone with Eve or Mallory :) They're from the same set of names as Bob (and Alice), but are actually affected by this issue, although to a less extent than what you describe. It's not certain how likely the kind of texts that uses them are to be used as training data though.

dibstern 9y ago

The solution for identity is easy. Just don’t rely on unverified name association data for names. I don’t get why it’s so hard to avoid that. Just don’t do it. There are plenty of ways to train models without using that kind of data, both in the training phase and in active use.

otakucode 9y ago

The fix is also very much not simple. Imagine that development of AI systems proceeds as it has thus far... when an AI system turns out biased, racist, sexist, etc - we shut it down. As AIs get more capable, any which do harm, act deceptively, etc - we shut them down. Eventually, we will have AIs which are better than us in every respect. They will embody our idea of perfection. This is stupendously dangerous. Humanity has a long history of adapting to technology 'taking away' things that everyone thought were 'fundamentally human', like the ability to do work or make things or lay train tracks or whatnot.... it's not hopeful. We deal very poorly with this. Consider the story of John Henry. He's a folk hero. For killing himself. Because he killed himself in defiance of the machine outperforming him. So this stuff is all-caps Important. What is the likely response from humanity when there is a perfect, not machine but mind? My bet? Humanity will identify with its worse aspects. It will enshrine hatred, irrationality, mean-spirited spite, violence, self-destruction, and all of the things we built AIs to never stray into. Those will become "what it means to be human." AI may be a philosophical crisis unlike anything humanity has ever faced. Not in kind, but simply in degree.