3 ms·

This article is very dumb. Models are descriptions of data--they describe how things are to some level of accuracy, not how things ought to be according to som

This article is very dumb. Models are descriptions of data--they describe how things are to some level of accuracy, not how things ought to be according to some or other concept of cosmic justice. From the article: "Notice how the grid search would tell us that there should be a big payment gap between the two groups even if they display high levels of skill" Only an idiot would say that the output of a model has anything to do with the moral question of how things should be. Using "biased" inputs to your model may or may not help you describe your data and answer questions about what it is, but the inputs have nothing to do with what ought to be.

beakerbreaker 6y ago
Models are not just descriptions of data, which is why overfitting is not considered success. They are an attempt to generalize patterns from a given datatset. What patterns they generalize is of interest to several research fields, including fairness, domain adaptation, and robustness. A useful but incomplete resource is below. https://fairmlbook.org/ https://fairmlbook.org/
- gotoeleven 6y ago
  "Models are not just descriptions of data, which is why overfitting is not considered success." The degree to which a model is overfit is a measure of how accurate the model is, but that doesn't make it any more or less than a description of the data. I get what everyone's goal is here, it would be shitty to build and rely on a bunch of racist models. But the nonsense in this article is dangerous--again models are descriptions of what is and this weird glorification of them as some sort of source of normative judgement is bizarre. Models have to be used carefully with understanding of the limitations of their structure and the data that went into them. That doesn't make any particular modeling technique racist.
sbpayne 6y ago
I find it interesting that you said: “Only an idiot would say the output of a model has anything to do with the moral question that of how things should be.” My understanding of the article is that people often share this sentiment, but they often believe this property implies that an optimization procedure will only reflect bias rather than amplify it. Then the author goes on to show that this perceived implication is incorrect. I.e. your algorithm not understanding an existing bias does not mean it won’t amplify bias — it means it won’t care, which is precisely what allows it to amplify the bias. Because the bias exists in the underlying data, your algorithm can discover and overfit to that bias. Without a “regularizer” to control for this, it’s probably a bad idea to think the algorithm does not amplify bias. So if my understanding of your comment and the author’s post are right, I think you would both agree that at the end of the day: something needs to explicitly control for the moral question of how things should be, because the general optimization procedures we use do/will not. Is that a fair statement?
- gotoeleven 6y ago
  How is the existence of a model amplifying any bias? The example the author gives is that conditioning a linear model on some "biased" variable gives a more accurate model that predicts large differences between these groups. But the large difference in the groups is right there in the data. It's not amplified in any way. And then somehow this is the modeller's fault for choosing a naughty variable, as if the output of a model has anything to do with the world the modeller wishes existed. The author's beef should be with the idiots who would take this model and then say "yep looks like since these two groups have different regression lines it's totally great that we see large differences between these groups"
  - sbpayne 6y ago
    I was mostly referring to this bit from the article: “Notice how the model that has the lowest mean squared error is also the model that causes the most bias between groups for higher skill levels. This increase in bias cannot be blamed merely on the data. It’s the choice of the model that increases this bias which is the responsibility of the algorithm designer.” My reading of the plots is that the difference between the groups seems to grow beyond what is present in the data. Do you not think this is amplified? As for conditioning on a biased variable; this post uses a contrived example for sure. But the same thing happens with variables that correlate with the “naughty” variable (perhaps author should have explicitly showed this to drive the point?) Removing all variables that correlate with “naughty variables” is a pretty difficult task itself without having something to detect the correlation and tell you. At least that’s been my experience.
    - mr_toad 6y ago
      > This increase in bias cannot be blamed merely on the data. This is debatable in this case. The grouped model might be a better fit to the data: it indicates a higher bias in salaries at higher skill levels. It sounds completely plausible that the gender bias for unskilled workers is less than for high skilled workers. Models can be under-fit as well as over fit. Just using a dummy variable for gender might not properly capture interactions like this. But of course it’s always possible that this is just an artefact of the training data.
berndi 6y ago
Please just stop posting articles from this blog. There was another one a while back (https://news.ycombinator.com/item?id=23004026 https://news.ycombinator.com/item?id=23004026) where the same author "discovered" basic results in convex optimization but failed to recognize them as such. The author seems to be, I'm sorry to say, clueless but confident and this combination leads to misguided articles that will be quite confusing to newcomers.

beakerbreaker 6y ago

Models are not just descriptions of data, which is why overfitting is not considered success. They are an attempt to generalize patterns from a given datatset. What patterns they generalize is of interest to several research fields, including fairness, domain adaptation, and robustness. A useful but incomplete resource is below. https://fairmlbook.org/ https://fairmlbook.org/

gotoeleven 6y ago
"Models are not just descriptions of data, which is why overfitting is not considered success." The degree to which a model is overfit is a measure of how accurate the model is, but that doesn't make it any more or less than a description of the data. I get what everyone's goal is here, it would be shitty to build and rely on a bunch of racist models. But the nonsense in this article is dangerous--again models are descriptions of what is and this weird glorification of them as some sort of source of normative judgement is bizarre. Models have to be used carefully with understanding of the limitations of their structure and the data that went into them. That doesn't make any particular modeling technique racist.

sbpayne 6y ago

I find it interesting that you said: “Only an idiot would say the output of a model has anything to do with the moral question that of how things should be.” My understanding of the article is that people often share this sentiment, but they often believe this property implies that an optimization procedure will only reflect bias rather than amplify it. Then the author goes on to show that this perceived implication is incorrect. I.e. your algorithm not understanding an existing bias does not mean it won’t amplify bias — it means it won’t care, which is precisely what allows it to amplify the bias. Because the bias exists in the underlying data, your algorithm can discover and overfit to that bias. Without a “regularizer” to control for this, it’s probably a bad idea to think the algorithm does not amplify bias. So if my understanding of your comment and the author’s post are right, I think you would both agree that at the end of the day: something needs to explicitly control for the moral question of how things should be, because the general optimization procedures we use do/will not. Is that a fair statement?

gotoeleven 6y ago
How is the existence of a model amplifying any bias? The example the author gives is that conditioning a linear model on some "biased" variable gives a more accurate model that predicts large differences between these groups. But the large difference in the groups is right there in the data. It's not amplified in any way. And then somehow this is the modeller's fault for choosing a naughty variable, as if the output of a model has anything to do with the world the modeller wishes existed. The author's beef should be with the idiots who would take this model and then say "yep looks like since these two groups have different regression lines it's totally great that we see large differences between these groups"
- sbpayne 6y ago
  I was mostly referring to this bit from the article: “Notice how the model that has the lowest mean squared error is also the model that causes the most bias between groups for higher skill levels. This increase in bias cannot be blamed merely on the data. It’s the choice of the model that increases this bias which is the responsibility of the algorithm designer.” My reading of the plots is that the difference between the groups seems to grow beyond what is present in the data. Do you not think this is amplified? As for conditioning on a biased variable; this post uses a contrived example for sure. But the same thing happens with variables that correlate with the “naughty” variable (perhaps author should have explicitly showed this to drive the point?) Removing all variables that correlate with “naughty variables” is a pretty difficult task itself without having something to detect the correlation and tell you. At least that’s been my experience.
  - mr_toad 6y ago
    > This increase in bias cannot be blamed merely on the data. This is debatable in this case. The grouped model might be a better fit to the data: it indicates a higher bias in salaries at higher skill levels. It sounds completely plausible that the gender bias for unskilled workers is less than for high skilled workers. Models can be under-fit as well as over fit. Just using a dummy variable for gender might not properly capture interactions like this. But of course it’s always possible that this is just an artefact of the training data.

berndi 6y ago

Please just stop posting articles from this blog. There was another one a while back (https://news.ycombinator.com/item?id=23004026 https://news.ycombinator.com/item?id=23004026) where the same author "discovered" basic results in convex optimization but failed to recognize them as such. The author seems to be, I'm sorry to say, clueless but confident and this combination leads to misguided articles that will be quite confusing to newcomers.