7 ms·

My takeaway is that people disagree that the problem is simply the choice of dataset, but rather, they think that the problem is using ML at all to perform cert

by baddox 6y ago

My takeaway is that people disagree that the problem is simply the choice of dataset, but rather, they think that the problem is using ML at all to perform certain tasks (including, perhaps, making important decisions based on a person’s appearance).

darawk 6y ago
I agree that some people appear to believe this. But what is the justification for it? I don't really get how it cashes out in concrete harm, assuming you de-bias the datasets. To be clear, I do get arguments that ML might increase the efficiency of policing, and there is some inherent tension between the efficiency of the state and freedom, to the extent that imperfect enforcement is a tacit feature of our legal system. But, this is not an inherently racial point.
- whymauri 6y ago
  Using appearance and superficial features to predict personality or aptitude is phrenology.
  - darawk 6y ago
    Phrenology was bad because it was scientifically invalid. I don't see any problem with predicting traits from appearances, at least, not intrinsically if the data is accurate.
    - whymauri 6y ago
      There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing. I really don't want my field to regress towards 19th-century pseudoscience when we can empower other scientific advancements that will yield immediate benefits.
      - darawk 6y ago
        > There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing. This is a strange point. Are personality types subjective? Is IQ? Maybe they are, but we still sometimes find them useful to quantify. If it's ok to predict these things from your answers to questions...why isn't it ok to predict them from pictures of your face?
        joshuamorton 6y ago
        Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason. So yes, I think I can state pretty concretely that there's no value in trying to predict IQ from face shape.
        darawk 6y ago
        > Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason. The presence of error in a prediction makes that prediction valueless?
        joshuamorton 6y ago
        When doing anything prediction-y, you need to take the relative value of false positives and negatives into account. Can you describe a situation where the harm caused to someone by mis-classifying their IQ is outweighed by the increased efficiency from...being able to more quickly identify someone's IQ? Like, can you even describe a situation where this will be used ethically at all? What use is there for predicting someone's IQ with dubious accuracy, assuming you already have their permission?
    - baddox 6y ago
      > Phrenology was bad because it was scientifically invalid. Well, that's not a very full story. It was scientifically invalid because it modeled mental traits based on the shape of the skull, and that's the important part. If you trained a model to predict traits based on accurate data about skull shape and traits, I'm confident that the algorithm would happily do that. Phrenology would still be bad regardless of what technical methods are used to do phrenology.
      - darawk 6y ago
        What is your argument that it's still bad? That was kind of a non-sequitur. If we have a model that takes in skull measurements and accurately predicts whether you are an introvert or extrovert...what is the problem with that?
- baddox 6y ago
  Well, what are you going to use this technique for? What’s the use case for generating a high-res photo of a face from a low-res photo of a face? It’s hard to think of any use case with real-world consequences that wouldn’t be a bad idea regardless of the training set.
  - prepend 6y ago
    My org allows profpics for email or messaging or stuff like that. Some people have old, tiny profpics that are pixellated at various sizes. This technique could help the pictures look nicer.
  - grandmczeb 6y ago
    From the author's github[1] > We have noticed a lot of concern that PULSE will be used to identify individuals whose faces have been blurred out. We want to emphasize that this is impossible - PULSE makes imaginary faces of people who do not exist, which should not be confused for real people. It will not help identify or reconstruct the original image. This is just another way to generate realistic faces that match a pattern. The "bad use cases" I can imagine involve unmasking real people, which is not what the model does. [1] https://github.com/tg-bomze/Face-Depixelizer https://github.com/tg-bomze/Face-Depixelizer
- int_19h 6y ago
  How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with? For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police. So ML takes those biases, and entrenches them further. Worse yet, because people tend to believe that "computers cannot be biased", the fact that it's ML that's producing a specific result, is taken as prima facie evidence that said result is immune to criticism from that perspective. And with most advanced models, you can't really crack it open and make sense of what's inside - so you can't easily prove that it is biased.
  - darawk 6y ago
    > How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with? I can think of at least one trick that might work. Build a classifier capable of identifying race from the data, and then reverse its gradient to remove its signals. This would deracialize a dataset, so that there is no detectable latent information in it. However, whether or not my particular solution works is somewhat beside the point. It's an engineering challenge to figure out how to do this, but that isn't an argument that ML is fundamentally bad for these purposes. > For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police. I totally agree that this is a problem. But it seems to me like the solution is more effort on fixing the biases, not reverting to human judgment. I think it's always important to make sure we understand what the baseline is we're comparing to. Humans are racist. Humans created the biases in these datasets. These algorithmic tools just bring those biases to the surface and make them quantifiable. It seems to me that that represents an incredible opportunity to actually fix the biases in a measurable way.
- deleted 6y ago
  [deleted]
- slg 6y ago
  But how do you de-bias that datasets when the world itself is biased? You can't just ignore that bias in the real world because any ML will at best mirror that bias if not amplify it. For example imagine some ML algorithm that predicts the potential value of a customer (or their likelihood to steal from you) as they walk in the door of your business. How do you remove the racial bias from a system like that when racial biases exist in society that result in the average White person being in a higher socioeconomic class than the average Black person? The ML algorithm will tell you White customers are more valuable than Black customers. It is doing exactly what you told it to do, but the end result is that Black people get worse service in your business.
  - karpierz 6y ago
    You're talking about a different problem than the original thread. The original thread isn't saying "the underlying world is biased"; Obama's face exists in the world. It's saying that the dataset isn't reflective of the world, and this is an issue because then you'll end up with voice tools[1], facial recognition software[2], and sentencing algorithms[3] end up performing disproportionately poorly on some minority groups. And this ends up reinforcing the poor positions many people are already in. 1) https://www.pnas.org/content/117/14/7684 https://www.pnas.org/content/117/14/7684 2) http://proceedings.mlr.press/v81/buolamwini18a.html?mod=article_inline http://proceedings.mlr.press/v81/buolamwini18a.html?mod=arti... 3) https://www.liebertpub.com/doi/full/10.1089/big.2016.0047 https://www.liebertpub.com/doi/full/10.1089/big.2016.0047
    - slg 6y ago
      The way I understood his original tweet was that he was taking the specific problem with this study to illustrate a point about ML overall. Therefore the criticism he receives can be related to how that idea applies to ML overall and not exclusively this problem.
      - im3w1l 6y ago
        How else are you going to solve it, if not one issue at a time? Or do people just want to rant without being interrupted by solutions?
AndrewKemendo 6y ago
I watched the thread and LeCun did a pretty thorough explanation of why his argument was more complex than "the dataset is biased" You can see in the responses however that many people were dissatisfied with his response. [1] https://twitter.com/ylecun/status/1275162528511860737 https://twitter.com/ylecun/status/1275162528511860737
- zucker42 6y ago
  Great link, the one part of LeCun's argument I don't understand (or maybe I disagree with it) is that in Tweet 9 he minimizes the effect of architectural bias mentioned in Tweet 6 by saying that modern DL models have "generic architectures". But his own Tweet (#11) apparently demonstrates a problem with that reasoning; modern DL architectures are bad at generalizing from rare categories, which would seem like an architectural issue that causes significant bias, especially in the sense of racial bias. I think other people may have touched on something similar by pointing that bias might be inherent in the metrics researchers optimize for when designing new models.
munchbunny 6y ago
I’d phrase it that “bias” in the social sense can occur in (1) the task itself based on how it’s defined, (2) preexisting within the problem domain, so representative datasets could still be biased, and (3) in when the system and the people operating it choose to use the outputs of the model. Which is all to say that “bias” problems related to the model can extend beyond the model itself. The solution isn’t necessarily to fix the data or the model. It might be banning use of facial recognition by police, for example, which some places are doing.

by baddox 6y ago

darawk 6y ago

I agree that some people appear to believe this. But what is the justification for it? I don't really get how it cashes out in concrete harm, assuming you de-bias the datasets. To be clear, I do get arguments that ML might increase the efficiency of policing, and there is some inherent tension between the efficiency of the state and freedom, to the extent that imperfect enforcement is a tacit feature of our legal system. But, this is not an inherently racial point.

whymauri 6y ago
Using appearance and superficial features to predict personality or aptitude is phrenology.
- darawk 6y ago
  Phrenology was bad because it was scientifically invalid. I don't see any problem with predicting traits from appearances, at least, not intrinsically if the data is accurate.
  - whymauri 6y ago
    There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing. I really don't want my field to regress towards 19th-century pseudoscience when we can empower other scientific advancements that will yield immediate benefits.
    - darawk 6y ago
      > There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing. This is a strange point. Are personality types subjective? Is IQ? Maybe they are, but we still sometimes find them useful to quantify. If it's ok to predict these things from your answers to questions...why isn't it ok to predict them from pictures of your face?
      - joshuamorton 6y ago
        Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason. So yes, I think I can state pretty concretely that there's no value in trying to predict IQ from face shape.
        darawk 6y ago
        > Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason. The presence of error in a prediction makes that prediction valueless?
        joshuamorton 6y ago
        When doing anything prediction-y, you need to take the relative value of false positives and negatives into account. Can you describe a situation where the harm caused to someone by mis-classifying their IQ is outweighed by the increased efficiency from...being able to more quickly identify someone's IQ? Like, can you even describe a situation where this will be used ethically at all? What use is there for predicting someone's IQ with dubious accuracy, assuming you already have their permission?
  - baddox 6y ago
    > Phrenology was bad because it was scientifically invalid. Well, that's not a very full story. It was scientifically invalid because it modeled mental traits based on the shape of the skull, and that's the important part. If you trained a model to predict traits based on accurate data about skull shape and traits, I'm confident that the algorithm would happily do that. Phrenology would still be bad regardless of what technical methods are used to do phrenology.
    - darawk 6y ago
      What is your argument that it's still bad? That was kind of a non-sequitur. If we have a model that takes in skull measurements and accurately predicts whether you are an introvert or extrovert...what is the problem with that?
baddox 6y ago
Well, what are you going to use this technique for? What’s the use case for generating a high-res photo of a face from a low-res photo of a face? It’s hard to think of any use case with real-world consequences that wouldn’t be a bad idea regardless of the training set.
- prepend 6y ago
  My org allows profpics for email or messaging or stuff like that. Some people have old, tiny profpics that are pixellated at various sizes. This technique could help the pictures look nicer.
- grandmczeb 6y ago
  From the author's github[1] > We have noticed a lot of concern that PULSE will be used to identify individuals whose faces have been blurred out. We want to emphasize that this is impossible - PULSE makes imaginary faces of people who do not exist, which should not be confused for real people. It will not help identify or reconstruct the original image. This is just another way to generate realistic faces that match a pattern. The "bad use cases" I can imagine involve unmasking real people, which is not what the model does. [1] https://github.com/tg-bomze/Face-Depixelizer https://github.com/tg-bomze/Face-Depixelizer
int_19h 6y ago
How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with? For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police. So ML takes those biases, and entrenches them further. Worse yet, because people tend to believe that "computers cannot be biased", the fact that it's ML that's producing a specific result, is taken as prima facie evidence that said result is immune to criticism from that perspective. And with most advanced models, you can't really crack it open and make sense of what's inside - so you can't easily prove that it is biased.
- darawk 6y ago
  > How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with? I can think of at least one trick that might work. Build a classifier capable of identifying race from the data, and then reverse its gradient to remove its signals. This would deracialize a dataset, so that there is no detectable latent information in it. However, whether or not my particular solution works is somewhat beside the point. It's an engineering challenge to figure out how to do this, but that isn't an argument that ML is fundamentally bad for these purposes. > For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police. I totally agree that this is a problem. But it seems to me like the solution is more effort on fixing the biases, not reverting to human judgment. I think it's always important to make sure we understand what the baseline is we're comparing to. Humans are racist. Humans created the biases in these datasets. These algorithmic tools just bring those biases to the surface and make them quantifiable. It seems to me that that represents an incredible opportunity to actually fix the biases in a measurable way.
deleted 6y ago
[deleted]
slg 6y ago
But how do you de-bias that datasets when the world itself is biased? You can't just ignore that bias in the real world because any ML will at best mirror that bias if not amplify it. For example imagine some ML algorithm that predicts the potential value of a customer (or their likelihood to steal from you) as they walk in the door of your business. How do you remove the racial bias from a system like that when racial biases exist in society that result in the average White person being in a higher socioeconomic class than the average Black person? The ML algorithm will tell you White customers are more valuable than Black customers. It is doing exactly what you told it to do, but the end result is that Black people get worse service in your business.
- karpierz 6y ago
  You're talking about a different problem than the original thread. The original thread isn't saying "the underlying world is biased"; Obama's face exists in the world. It's saying that the dataset isn't reflective of the world, and this is an issue because then you'll end up with voice tools[1], facial recognition software[2], and sentencing algorithms[3] end up performing disproportionately poorly on some minority groups. And this ends up reinforcing the poor positions many people are already in. 1) https://www.pnas.org/content/117/14/7684 https://www.pnas.org/content/117/14/7684 2) http://proceedings.mlr.press/v81/buolamwini18a.html?mod=article_inline http://proceedings.mlr.press/v81/buolamwini18a.html?mod=arti... 3) https://www.liebertpub.com/doi/full/10.1089/big.2016.0047 https://www.liebertpub.com/doi/full/10.1089/big.2016.0047
  - slg 6y ago
    The way I understood his original tweet was that he was taking the specific problem with this study to illustrate a point about ML overall. Therefore the criticism he receives can be related to how that idea applies to ML overall and not exclusively this problem.
    - im3w1l 6y ago
      How else are you going to solve it, if not one issue at a time? Or do people just want to rant without being interrupted by solutions?

AndrewKemendo 6y ago

I watched the thread and LeCun did a pretty thorough explanation of why his argument was more complex than "the dataset is biased" You can see in the responses however that many people were dissatisfied with his response. [1] https://twitter.com/ylecun/status/1275162528511860737 https://twitter.com/ylecun/status/1275162528511860737

zucker42 6y ago
Great link, the one part of LeCun's argument I don't understand (or maybe I disagree with it) is that in Tweet 9 he minimizes the effect of architectural bias mentioned in Tweet 6 by saying that modern DL models have "generic architectures". But his own Tweet (#11) apparently demonstrates a problem with that reasoning; modern DL architectures are bad at generalizing from rare categories, which would seem like an architectural issue that causes significant bias, especially in the sense of racial bias. I think other people may have touched on something similar by pointing that bias might be inherent in the metrics researchers optimize for when designing new models.

munchbunny 6y ago

I’d phrase it that “bias” in the social sense can occur in (1) the task itself based on how it’s defined, (2) preexisting within the problem domain, so representative datasets could still be biased, and (3) in when the system and the people operating it choose to use the outputs of the model. Which is all to say that “bias” problems related to the model can extend beyond the model itself. The solution isn’t necessarily to fix the data or the model. It might be banning use of facial recognition by police, for example, which some places are doing.