6 ms·

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people ar

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and specially if the models have high impact on people's lives (applying to loans, university, scholarship, insurance), we cannot afford to blindly use existing annotations in black box models. All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. Edit: here is code for the curious https://github.com/sergioisidoro/aequalis https://github.com/sergioisidoro/aequalis

AnthonyMouse 10y ago
> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time travel to tell you with 100.0% accuracy who would pay back their loans, there would be a racial disparity in the result, but the cause of it is not the fault of the algorithm. And you can't fix it there because that's not where the problem is. Suppose you have a group of 800 white middle managers, 100 white sales clerks and 100 black sales clerks. Clearly the algorithm is going to have a racial disparity in outcome if the middle managers are at 500% of the cutoff and the sales clerks are right on the line, because it will accept all of the middle managers and half of the sales clerks which means it will accept >94% of the white applicants and 50% of the black applicants. But the source of the disparity is that black people are underrepresented as middle managers and overrepresented as sales clerks. The algorithm is just telling you that. It can't change it. And inserting bias into the algorithm to "balance" the outcome doesn't actually do that, all you're doing is creating bias against white sales clerks who had previously been on the same footing as black sales clerks. The white middle managers will be unaffected because they're sufficiently far above the cutoff that the change doesn't affect them, even though that they're the source of the imbalance.
- throwawaysocks 10y ago
  > maximizing prediction accuracy is inherently unbiased This assumes that you're actually maximizing prediction accuracy, rather than taking the easiest route toward sufficiently high predictive power. Not-so-hypothetical: you can invest $N and create a profitable model that (unfairly and inaccurately) discriminates directly based upon race, or you can invest $N*M and create a profitable model that does not discriminate on race (regardless of whether its results are racially equitable). Given the choice, a lot of people will choose the $N approach.
  - basseq 10y ago
    > regardless of whether its results are racially equitable Some would call this discrimination, if not outright racism. As others on this thread have pointed out, at some point the algorithm follows the underlying data, which may also be discriminatory (zip code, etc.).
- tombone12 10y ago
  You entirely miss the point! The point is that in supervised learning for example, if you optimize prediction accuracy with respect to your human generated examples, you will get a model that exactly reproduces the racist judgment of the human that generated your training set.
  - mcguire 10y ago
    Yes, and you can get similar effects with unsupervised learning if the data set is biased.
  - Houshalter 10y ago
    Thats silly though. You should use ground truth of actual outcomes, not predict that useless humans would do. But even then I bet the algorithm would be less racist than the humans, if you don't give them race as a feature.
  - hx87 10y ago
    Then the problem is in the racist human(s), not the algorithm.
penguinduck 10y ago
You wrote a program that rigs the input data in favor of one class in such a way that the classifier results are more or less uncorrelated with membership in that class. Am I understanding the code correctly? I don't see how you eliminated "human biases present in the data annotations" here. It seems like your program merely rigs the input in order to get the kind of output you like to see. Obviously you can tamper with the data to get any kind of result you want. You can force equal outcomes for "Never-worked" and "Private" but that doesn't mean you're removing some inherent bias in the original data. Also, this sentence: > All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. is kind of weird in the context of your comment and program. If you are correct in your fears of biased data, then accounting for the bias in the data maximizes profit for the companies. Ethics shouldn't be necessary if it's just about bias in the input data.
- sergioisidoro 10y ago
  Yes, in the last example you can see it as rigging the input data. I replicated the results of the experiment in the paper, and I made some remarks of this sorts in my own report and discussion. The idea is that a sensitive parameter should not contribute to a decision, so, for example, the probability of group A or group B having access to a loan should be the same: P(Loan|A) should be the same as P(Loan|B) This is dangerous, as the discrimination metric is sensitive to biases present in the data. It can, effectively, make it easier for the discriminated group A to get a Loan in comparison to a person in group B in same situations. This happens if the bias is not in the annotations, but in the demographics of the dataset. This is a really interesting problem, and I don't have answers for it.
dwmtm 10y ago
Capitalism leads to some unfavorable outcomes. Algorithms do capitalism better, leading to more unfavorable outcomes. Therefore, algorithms are the problem. I don't know what I missed, but it looks like algorithms merely provide the reductio for something you already don't like.
zeveb 10y ago
> All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. If companies are maximising profit, then they will treat a person _exactly_ according to everything they can observe about that person. How is that unethical? The problem that people have is that they don't want objectivity: they want to be able to pretend that something false is true, because it is comfortable.

AnthonyMouse 10y ago

> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time travel to tell you with 100.0% accuracy who would pay back their loans, there would be a racial disparity in the result, but the cause of it is not the fault of the algorithm. And you can't fix it there because that's not where the problem is. Suppose you have a group of 800 white middle managers, 100 white sales clerks and 100 black sales clerks. Clearly the algorithm is going to have a racial disparity in outcome if the middle managers are at 500% of the cutoff and the sales clerks are right on the line, because it will accept all of the middle managers and half of the sales clerks which means it will accept >94% of the white applicants and 50% of the black applicants. But the source of the disparity is that black people are underrepresented as middle managers and overrepresented as sales clerks. The algorithm is just telling you that. It can't change it. And inserting bias into the algorithm to "balance" the outcome doesn't actually do that, all you're doing is creating bias against white sales clerks who had previously been on the same footing as black sales clerks. The white middle managers will be unaffected because they're sufficiently far above the cutoff that the change doesn't affect them, even though that they're the source of the imbalance.

throwawaysocks 10y ago
> maximizing prediction accuracy is inherently unbiased This assumes that you're actually maximizing prediction accuracy, rather than taking the easiest route toward sufficiently high predictive power. Not-so-hypothetical: you can invest $N and create a profitable model that (unfairly and inaccurately) discriminates directly based upon race, or you can invest $N*M and create a profitable model that does not discriminate on race (regardless of whether its results are racially equitable). Given the choice, a lot of people will choose the $N approach.
- basseq 10y ago
  > regardless of whether its results are racially equitable Some would call this discrimination, if not outright racism. As others on this thread have pointed out, at some point the algorithm follows the underlying data, which may also be discriminatory (zip code, etc.).
tombone12 10y ago
You entirely miss the point! The point is that in supervised learning for example, if you optimize prediction accuracy with respect to your human generated examples, you will get a model that exactly reproduces the racist judgment of the human that generated your training set.
- mcguire 10y ago
  Yes, and you can get similar effects with unsupervised learning if the data set is biased.
- Houshalter 10y ago
  Thats silly though. You should use ground truth of actual outcomes, not predict that useless humans would do. But even then I bet the algorithm would be less racist than the humans, if you don't give them race as a feature.
- hx87 10y ago
  Then the problem is in the racist human(s), not the algorithm.

penguinduck 10y ago

You wrote a program that rigs the input data in favor of one class in such a way that the classifier results are more or less uncorrelated with membership in that class. Am I understanding the code correctly? I don't see how you eliminated "human biases present in the data annotations" here. It seems like your program merely rigs the input in order to get the kind of output you like to see. Obviously you can tamper with the data to get any kind of result you want. You can force equal outcomes for "Never-worked" and "Private" but that doesn't mean you're removing some inherent bias in the original data. Also, this sentence: > All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. is kind of weird in the context of your comment and program. If you are correct in your fears of biased data, then accounting for the bias in the data maximizes profit for the companies. Ethics shouldn't be necessary if it's just about bias in the input data.

sergioisidoro 10y ago
Yes, in the last example you can see it as rigging the input data. I replicated the results of the experiment in the paper, and I made some remarks of this sorts in my own report and discussion. The idea is that a sensitive parameter should not contribute to a decision, so, for example, the probability of group A or group B having access to a loan should be the same: P(Loan|A) should be the same as P(Loan|B) This is dangerous, as the discrimination metric is sensitive to biases present in the data. It can, effectively, make it easier for the discriminated group A to get a Loan in comparison to a person in group B in same situations. This happens if the bias is not in the annotations, but in the demographics of the dataset. This is a really interesting problem, and I don't have answers for it.

dwmtm 10y ago

Capitalism leads to some unfavorable outcomes. Algorithms do capitalism better, leading to more unfavorable outcomes. Therefore, algorithms are the problem. I don't know what I missed, but it looks like algorithms merely provide the reductio for something you already don't like.

zeveb 10y ago

> All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. If companies are maximising profit, then they will treat a person _exactly_ according to everything they can observe about that person. How is that unethical? The problem that people have is that they don't want objectivity: they want to be able to pretend that something false is true, because it is comfortable.