4 ms·

I find it interesting that you said: “Only an idiot would say the output of a model has anything to do with the moral question that of how things should be.” M

I find it interesting that you said: “Only an idiot would say the output of a model has anything to do with the moral question that of how things should be.” My understanding of the article is that people often share this sentiment, but they often believe this property implies that an optimization procedure will only reflect bias rather than amplify it. Then the author goes on to show that this perceived implication is incorrect. I.e. your algorithm not understanding an existing bias does not mean it won’t amplify bias — it means it won’t care, which is precisely what allows it to amplify the bias. Because the bias exists in the underlying data, your algorithm can discover and overfit to that bias. Without a “regularizer” to control for this, it’s probably a bad idea to think the algorithm does not amplify bias. So if my understanding of your comment and the author’s post are right, I think you would both agree that at the end of the day: something needs to explicitly control for the moral question of how things should be, because the general optimization procedures we use do/will not. Is that a fair statement?

gotoeleven 6y ago
How is the existence of a model amplifying any bias? The example the author gives is that conditioning a linear model on some "biased" variable gives a more accurate model that predicts large differences between these groups. But the large difference in the groups is right there in the data. It's not amplified in any way. And then somehow this is the modeller's fault for choosing a naughty variable, as if the output of a model has anything to do with the world the modeller wishes existed. The author's beef should be with the idiots who would take this model and then say "yep looks like since these two groups have different regression lines it's totally great that we see large differences between these groups"
- sbpayne 6y ago
  I was mostly referring to this bit from the article: “Notice how the model that has the lowest mean squared error is also the model that causes the most bias between groups for higher skill levels. This increase in bias cannot be blamed merely on the data. It’s the choice of the model that increases this bias which is the responsibility of the algorithm designer.” My reading of the plots is that the difference between the groups seems to grow beyond what is present in the data. Do you not think this is amplified? As for conditioning on a biased variable; this post uses a contrived example for sure. But the same thing happens with variables that correlate with the “naughty” variable (perhaps author should have explicitly showed this to drive the point?) Removing all variables that correlate with “naughty variables” is a pretty difficult task itself without having something to detect the correlation and tell you. At least that’s been my experience.
  - mr_toad 6y ago
    > This increase in bias cannot be blamed merely on the data. This is debatable in this case. The grouped model might be a better fit to the data: it indicates a higher bias in salaries at higher skill levels. It sounds completely plausible that the gender bias for unskilled workers is less than for high skilled workers. Models can be under-fit as well as over fit. Just using a dummy variable for gender might not properly capture interactions like this. But of course it’s always possible that this is just an artefact of the training data.

gotoeleven 6y ago

How is the existence of a model amplifying any bias? The example the author gives is that conditioning a linear model on some "biased" variable gives a more accurate model that predicts large differences between these groups. But the large difference in the groups is right there in the data. It's not amplified in any way. And then somehow this is the modeller's fault for choosing a naughty variable, as if the output of a model has anything to do with the world the modeller wishes existed. The author's beef should be with the idiots who would take this model and then say "yep looks like since these two groups have different regression lines it's totally great that we see large differences between these groups"

sbpayne 6y ago
I was mostly referring to this bit from the article: “Notice how the model that has the lowest mean squared error is also the model that causes the most bias between groups for higher skill levels. This increase in bias cannot be blamed merely on the data. It’s the choice of the model that increases this bias which is the responsibility of the algorithm designer.” My reading of the plots is that the difference between the groups seems to grow beyond what is present in the data. Do you not think this is amplified? As for conditioning on a biased variable; this post uses a contrived example for sure. But the same thing happens with variables that correlate with the “naughty” variable (perhaps author should have explicitly showed this to drive the point?) Removing all variables that correlate with “naughty variables” is a pretty difficult task itself without having something to detect the correlation and tell you. At least that’s been my experience.
- mr_toad 6y ago
  > This increase in bias cannot be blamed merely on the data. This is debatable in this case. The grouped model might be a better fit to the data: it indicates a higher bias in salaries at higher skill levels. It sounds completely plausible that the gender bias for unskilled workers is less than for high skilled workers. Models can be under-fit as well as over fit. Just using a dummy variable for gender might not properly capture interactions like this. But of course it’s always possible that this is just an artefact of the training data.