7 ms·

At the end of a day, if you ask for a nurse, should the model output a male or female by default? If the input text lacks context/nuance, then the model must ha

At the end of a day, if you ask for a nurse, should the model output a male or female by default? If the input text lacks context/nuance, then the model must have some bias to infer the user's intent. This holds true for any image it generates; not just the politically sensitive ones. For example, if I ask for a picture of a person, and don't get one with pink hair, is that a shortcoming of the model? I'd say that bias is only an issue if it's unable to respond to additional nuance in the input text. For example, if I ask for a "male nurse" it should be able to generate the less likely combination. Same with other races, hair colors, etc... Trying to generate a model that's "free of correlative relationships" is impossible because the model would never have the infinitely pedantic input text to describe the exact output image.

karpierz 4y ago
> At the end of a day, if you ask for a nurse, should the model output a male or female by default? Randomly pick one. > Trying to generate a model that's "free of correlative relationships" is impossible because the model would never have the infinitely pedantic input text to describe the exact output image. Sure, and you can never make a medical procedure 100% safe. Doesn't mean that you don't try to make them safer. You can trim the obvious low hanging fruit though.
- calvinmorrison 4y ago
  what if I asked the model to show me a sunday school photograph of baptists in the National Baptist Convention?
  - rvnx 4y ago
    The pictures I got from a similar model when asking for a "sunday school photograph of baptists in the National Baptist Convention": https://ibb.co/sHGZwh7 https://ibb.co/sHGZwh7
    - calvinmorrison 4y ago
      and how do we _feel_ about that outcome?
      - andybak 4y ago
        It's gone now. What was it?
- pxmpxm 4y ago
  > Randomly pick one. How does the model back out the "certain people would like to pretend it's a fair coin toss that a randomly selected nurse is male or female" feature? It won't be in any representative training set, so you're back to fishing for stock photos on getty rather than generating things.
  - shadowgovt 4y ago
    Yep, that's the hard problem Google is not comfortable releasing the API to this until they have it solved.
    - zarzavat 4y ago
      But why is it a problem? The AI is just a mirror showing us ourselves. That’s a good thing. How does it help anyone to make an AI that presents a fake world so that we can pretend that we live in a world that we actually don’t? Disassociation from reality is more dangerous than bias.
      - Daishiman 4y ago
        The AI is a mirror of the text and image corpora it was presented, as parsed and sanitized by the team in question.
      - shadowgovt 4y ago
        > The AI is just a mirror showing us ourselves. That's one hypothesis.
      - astrange 4y ago
        In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play" Minsky shut his eyes, "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened. — The AI doesn’t know what’s common or not. You don’t know if it’s going to be correct unless you’ve tested it. Just assuming whatever it comes out with is right is going to work as well as asking a psychic for your future.
- nmfisher 4y ago
  What about preschool teacher? I say this because I’ve been visiting a number of childcare centres over the past few days and I still have yet to see a single male teacher.
slg 4y ago
This type of bias sounds a lot easier to explain away as a non-issue when we are using "nurse" as the hypothetical prompt. What if the prompt is "criminal", "rapist", or some other negative? Would that change your thought process or would you be okay with the system always returning a person of the same race and gender that statistics indicate is the most likely? Do you see how that could be a problem?
- tines 4y ago
  Not the person you responded to, but I do see how someone could be hurt by that, and I want to avoid hurting people. But is this the level at which we should do it? Could skewing search results, i.e. hiding the bias of the real world, give us the impression that everything is fine and we don't need to do anything to actually help people? I have a feeling that we need to be real with ourselves and solve problems and not paper over them. I feel like people generally expect search engines to tell them what's really there instead of what people wish were there. And if the engines do that, people can get agitated! I'd almost say that hurt feelings are prerequisite for real change, hard though that may be. These are all really interesting questions brought up by this technology, thanks for your thoughts. Disclaimer, I'm a fucking idiot with no idea what I'm talking about.
  - magicalist 4y ago
    > Could skewing search results, i.e. hiding the bias of the real world Which real world? The population you sample from is going to make a big difference. Do you expect it to reflect your day to day life in your own city? Own country? The entire world? Results will vary significantly.
    - sangnoir 4y ago
      For AI, "real world" is likely "the world, as seen by Silicon Valley."
    - tines 4y ago
      I'd say it doesn't actually matter, as long as the population sampled is made clear to the user. If I ask for pictures of Japanese people, I'm not shocked when all the results are of Japanese people. If I asked for "criminals in the United States" and all the results are black people, that should concern me, not because the data set is biased but because the real world is biased and we should do something about that. The difference is that I know what set I'm asking for a sample from, and I can react accordingly.
sangnoir 4y ago
> At the end of a day, if you ask for a nurse, should the model output a male or female by default? This depends on the application. As an example, it would be a problem if it's used as a CV-screening app that's implicitly down-ranking male-applicants to nurse positions, resulting in fewer interviews for them.
pshc 4y ago
Perhaps to avoid this issue, future versions of the model would throw an error like “bias leak: please specify a gender for the nurse at character 32”
webmaven 4y ago
> If the input text lacks context/nuance, then the model must have some bias to infer the user's intent. This holds true for any image it generates; not just the politically sensitive ones. For example, if I ask for a picture of a person, and don't get one with pink hair, is that a shortcoming of the model? You're ignoring that these models are stochastic. If I ask for a nurse and always get an image of a woman in scrubs, then yes, the model exhibits bias. If I get a male nurse half the time, we can say the model is unbiased WRT gender, at least. The same logic applies to CEOs always being old white men, criminals always being Black men, and so on. Stochastic models can output results that when aggregated exhibit a distribution from which we can infer bias or the lack thereof.

karpierz 4y ago

> At the end of a day, if you ask for a nurse, should the model output a male or female by default? Randomly pick one. > Trying to generate a model that's "free of correlative relationships" is impossible because the model would never have the infinitely pedantic input text to describe the exact output image. Sure, and you can never make a medical procedure 100% safe. Doesn't mean that you don't try to make them safer. You can trim the obvious low hanging fruit though.

calvinmorrison 4y ago
what if I asked the model to show me a sunday school photograph of baptists in the National Baptist Convention?
- rvnx 4y ago
  The pictures I got from a similar model when asking for a "sunday school photograph of baptists in the National Baptist Convention": https://ibb.co/sHGZwh7 https://ibb.co/sHGZwh7
  - calvinmorrison 4y ago
    and how do we _feel_ about that outcome?
    - andybak 4y ago
      It's gone now. What was it?
pxmpxm 4y ago
> Randomly pick one. How does the model back out the "certain people would like to pretend it's a fair coin toss that a randomly selected nurse is male or female" feature? It won't be in any representative training set, so you're back to fishing for stock photos on getty rather than generating things.
- shadowgovt 4y ago
  Yep, that's the hard problem Google is not comfortable releasing the API to this until they have it solved.
  - zarzavat 4y ago
    But why is it a problem? The AI is just a mirror showing us ourselves. That’s a good thing. How does it help anyone to make an AI that presents a fake world so that we can pretend that we live in a world that we actually don’t? Disassociation from reality is more dangerous than bias.
    - Daishiman 4y ago
      The AI is a mirror of the text and image corpora it was presented, as parsed and sanitized by the team in question.
    - shadowgovt 4y ago
      > The AI is just a mirror showing us ourselves. That's one hypothesis.
    - astrange 4y ago
      In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play" Minsky shut his eyes, "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened. — The AI doesn’t know what’s common or not. You don’t know if it’s going to be correct unless you’ve tested it. Just assuming whatever it comes out with is right is going to work as well as asking a psychic for your future.
nmfisher 4y ago
What about preschool teacher? I say this because I’ve been visiting a number of childcare centres over the past few days and I still have yet to see a single male teacher.

slg 4y ago

This type of bias sounds a lot easier to explain away as a non-issue when we are using "nurse" as the hypothetical prompt. What if the prompt is "criminal", "rapist", or some other negative? Would that change your thought process or would you be okay with the system always returning a person of the same race and gender that statistics indicate is the most likely? Do you see how that could be a problem?

tines 4y ago
Not the person you responded to, but I do see how someone could be hurt by that, and I want to avoid hurting people. But is this the level at which we should do it? Could skewing search results, i.e. hiding the bias of the real world, give us the impression that everything is fine and we don't need to do anything to actually help people? I have a feeling that we need to be real with ourselves and solve problems and not paper over them. I feel like people generally expect search engines to tell them what's really there instead of what people wish were there. And if the engines do that, people can get agitated! I'd almost say that hurt feelings are prerequisite for real change, hard though that may be. These are all really interesting questions brought up by this technology, thanks for your thoughts. Disclaimer, I'm a fucking idiot with no idea what I'm talking about.
- magicalist 4y ago
  > Could skewing search results, i.e. hiding the bias of the real world Which real world? The population you sample from is going to make a big difference. Do you expect it to reflect your day to day life in your own city? Own country? The entire world? Results will vary significantly.
  - sangnoir 4y ago
    For AI, "real world" is likely "the world, as seen by Silicon Valley."
  - tines 4y ago
    I'd say it doesn't actually matter, as long as the population sampled is made clear to the user. If I ask for pictures of Japanese people, I'm not shocked when all the results are of Japanese people. If I asked for "criminals in the United States" and all the results are black people, that should concern me, not because the data set is biased but because the real world is biased and we should do something about that. The difference is that I know what set I'm asking for a sample from, and I can react accordingly.

sangnoir 4y ago

> At the end of a day, if you ask for a nurse, should the model output a male or female by default? This depends on the application. As an example, it would be a problem if it's used as a CV-screening app that's implicitly down-ranking male-applicants to nurse positions, resulting in fewer interviews for them.

pshc 4y ago

Perhaps to avoid this issue, future versions of the model would throw an error like “bias leak: please specify a gender for the nurse at character 32”

webmaven 4y ago

> If the input text lacks context/nuance, then the model must have some bias to infer the user's intent. This holds true for any image it generates; not just the politically sensitive ones. For example, if I ask for a picture of a person, and don't get one with pink hair, is that a shortcoming of the model? You're ignoring that these models are stochastic. If I ask for a nurse and always get an image of a woman in scrubs, then yes, the model exhibits bias. If I get a male nurse half the time, we can say the model is unbiased WRT gender, at least. The same logic applies to CEOs always being old white men, criminals always being Black men, and so on. Stochastic models can output results that when aggregated exhibit a distribution from which we can infer bias or the lack thereof.