3 ms·

I think it is reasonable for one of the founders and most highly respected members of the field to make confident claims about something that is as open and shu

I think it is reasonable for one of the founders and most highly respected members of the field to make confident claims about something that is as open and shut as this. The paper in question was training a network to upres faces. The researchers trained it on a dataset containing primarily White faces and released it on GitHub for reproducibility. Subsequent people used it on Black faces and found it make them white. This is not surprising. Black faces and White faces are not the same data manifold. This is like training a network to upsample oranges, then running it on an apple and being surprised when the result is an unusual color and texture for an apple. I’m not sure what else there is possibly wrong here. Is the width of their convolutions racist? Their choice to work on super resolution? The fact that they released their work for reproducibility?

astrange 6y ago
> I’m not sure what else there is possibly wrong here. Is the width of their convolutions racist? Not that parameter specifically, but if you say the data is wrong then that means any other parameter might be wrong too. More data might mean the model is too small to fit it, or the hyperparameters might be wrong to train it, or you now have the wrong ratio of other phenotypes (let's say that instead of races…) in the training set and their results regress. Also, if you're adding people with darker skin, that of course means the pixel values are lower. That matters for image processing code, things like SAD thresholds or noise reduction will work differently. > Their choice to work on super resolution? Superresolution is only useful as a toy and they should have mentioned that when they put up a live colab, yes. They added a disclaimer later on and it was good - there was a big issue where people were convinced this was somehow a surveillance technology because they watched too many TV shows, even though it literally can't work that way!
- visarga 6y ago
  Occam's razor makes us think in this case it's the data, the simplest explanation.
  - astrange 6y ago
    The data isn’t a separate thing from the rest of the model (unless you’ve validated it works across different datasets). It’s all just a bunch of numbers that get jointly optimized.
trhway 6y ago
>This is not surprising. Black faces and White faces are not the same data manifold. in particular because features contrast (dynamic range) is lower for darker faces. >Is the width of their convolutions racist? Not width. As a result of the above mentioned lower contrast, the racist here is the sensitivity of the resulting Gabor filters produced by the training in the first convolution layers and the Gauss filters in the next layer's. I suspect to deal with that problem one would have to normalize dynamic range of the faces, i.e. it would look something like kind of lightening of the dark faces and/or darkening of the white ones.
renewiltord 6y ago
There is a discussion to be had here but it's not possible in public because of the escalation of the topic. This is also not the best thread to be having it. Since everyone else is just saying "you don't get it" without explaining what "it" is, I will provide some brief avenues of exploration. Because they are brief, they will be coarse and imperfect, and aiming to give you directional assistance on the topic. So with that massive preamble because I'm not seeking to argue, here you go: * Pre-trained models already encode much of this bias and if you don't use them you won't get very far very fast * Large available datasets also reflect these biases * AI model performance is generally measured against this dataset, further entrenching the bias. If you're better on Indian face generation it will give you barely any benefit on most scoring methods * Saying "the outcome is only biased because the data is biased" misses the fact that the performance is on the biased data There's a little more to it but I believe that's the meat of it. Anyway, not too keen on arguing this. Just sharing because it took me some work to figure out what they were talking about and I wish someone had explained it to me so doing so here.
- free_rms 6y ago
  Thanks for the additional detail, but those all still boil down to dataset eventually, right? Those sound more like "yes, and" additions than the "no, you're racist" shitfight we saw.
  - renewiltord 6y ago
    In the interest of explaining (not arguing), I believe the distinction is sort of that the community at large recognizes performance only on certain metrics. Let's set aside the notion of whether something or someone is "racist" because that's a bit charged. We'll instead just talk about whether we're creating generally capable systems. For instance, if I were to champion metric A which purports to measure performance on human faces but it really only rewards performance on Senegalese then models that do better in general may not be recognized for being better. In an isolated sense this is not a problem. However if the mainstream is that all metrics that are taken seriously are dataset-biased then we'll have an environment where the models will be trained on biased datasets in order to be successful. For instance if everyone uses LFW to determine how good facial recognition is, then Senegalese fine-tuned facial recognition tech will not be recognized as being good at facial recognition. So the argument is that dataset bias is built-in to our approach to the problem. I, personally, think that this isn't malice. We need benchmarks to judge approaches against each other. Benchmarks always have a first mover advantage and a massive path dependence issue. The first benchmarks aiming for things on humans do not reflect humanity accurately. These benchmarks became standard among the community. To be taken seriously you have to do well on benchmarks that are standard in the community. Dataset bias is then natural in newer approaches because the approaches are judged against how good they are against the inaccurate (if you will) benchmarks. So no one need be racist or anything for the end result of the field to end up being discriminatory. I don't think a successful approach is to call people racist over this. After all, it isn't malice that guides them. The discrimination comes from the sort of historical accident that has North facing up on a map. And no individual is really racist. It's sort of like the Bechdel test: no movie is crappy simply because it doesn't have two girls talking to each other, but if very few movies have two girls talking to each other about something other than boys, then it makes you think "hmmm, why's that the case".

astrange 6y ago

> I’m not sure what else there is possibly wrong here. Is the width of their convolutions racist? Not that parameter specifically, but if you say the data is wrong then that means any other parameter might be wrong too. More data might mean the model is too small to fit it, or the hyperparameters might be wrong to train it, or you now have the wrong ratio of other phenotypes (let's say that instead of races…) in the training set and their results regress. Also, if you're adding people with darker skin, that of course means the pixel values are lower. That matters for image processing code, things like SAD thresholds or noise reduction will work differently. > Their choice to work on super resolution? Superresolution is only useful as a toy and they should have mentioned that when they put up a live colab, yes. They added a disclaimer later on and it was good - there was a big issue where people were convinced this was somehow a surveillance technology because they watched too many TV shows, even though it literally can't work that way!

visarga 6y ago
Occam's razor makes us think in this case it's the data, the simplest explanation.
- astrange 6y ago
  The data isn’t a separate thing from the rest of the model (unless you’ve validated it works across different datasets). It’s all just a bunch of numbers that get jointly optimized.

trhway 6y ago

>This is not surprising. Black faces and White faces are not the same data manifold. in particular because features contrast (dynamic range) is lower for darker faces. >Is the width of their convolutions racist? Not width. As a result of the above mentioned lower contrast, the racist here is the sensitivity of the resulting Gabor filters produced by the training in the first convolution layers and the Gauss filters in the next layer's. I suspect to deal with that problem one would have to normalize dynamic range of the faces, i.e. it would look something like kind of lightening of the dark faces and/or darkening of the white ones.

renewiltord 6y ago

There is a discussion to be had here but it's not possible in public because of the escalation of the topic. This is also not the best thread to be having it. Since everyone else is just saying "you don't get it" without explaining what "it" is, I will provide some brief avenues of exploration. Because they are brief, they will be coarse and imperfect, and aiming to give you directional assistance on the topic. So with that massive preamble because I'm not seeking to argue, here you go: * Pre-trained models already encode much of this bias and if you don't use them you won't get very far very fast * Large available datasets also reflect these biases * AI model performance is generally measured against this dataset, further entrenching the bias. If you're better on Indian face generation it will give you barely any benefit on most scoring methods * Saying "the outcome is only biased because the data is biased" misses the fact that the performance is on the biased data There's a little more to it but I believe that's the meat of it. Anyway, not too keen on arguing this. Just sharing because it took me some work to figure out what they were talking about and I wish someone had explained it to me so doing so here.

free_rms 6y ago
Thanks for the additional detail, but those all still boil down to dataset eventually, right? Those sound more like "yes, and" additions than the "no, you're racist" shitfight we saw.
- renewiltord 6y ago
  In the interest of explaining (not arguing), I believe the distinction is sort of that the community at large recognizes performance only on certain metrics. Let's set aside the notion of whether something or someone is "racist" because that's a bit charged. We'll instead just talk about whether we're creating generally capable systems. For instance, if I were to champion metric A which purports to measure performance on human faces but it really only rewards performance on Senegalese then models that do better in general may not be recognized for being better. In an isolated sense this is not a problem. However if the mainstream is that all metrics that are taken seriously are dataset-biased then we'll have an environment where the models will be trained on biased datasets in order to be successful. For instance if everyone uses LFW to determine how good facial recognition is, then Senegalese fine-tuned facial recognition tech will not be recognized as being good at facial recognition. So the argument is that dataset bias is built-in to our approach to the problem. I, personally, think that this isn't malice. We need benchmarks to judge approaches against each other. Benchmarks always have a first mover advantage and a massive path dependence issue. The first benchmarks aiming for things on humans do not reflect humanity accurately. These benchmarks became standard among the community. To be taken seriously you have to do well on benchmarks that are standard in the community. Dataset bias is then natural in newer approaches because the approaches are judged against how good they are against the inaccurate (if you will) benchmarks. So no one need be racist or anything for the end result of the field to end up being discriminatory. I don't think a successful approach is to call people racist over this. After all, it isn't malice that guides them. The discrimination comes from the sort of historical accident that has North facing up on a map. And no individual is really racist. It's sort of like the Bechdel test: no movie is crappy simply because it doesn't have two girls talking to each other, but if very few movies have two girls talking to each other about something other than boys, then it makes you think "hmmm, why's that the case".