4 ms·

I agree that the data will shift the results of the model and that some predictors can be proxies for others that are protected, but the claim was that the mode

I agree that the data will shift the results of the model and that some predictors can be proxies for others that are protected, but the claim was that the models themselves were biased (rather globally) and was curious whether that was shown and if so how.

slg 6y ago
I'm not OP, but I don't think the claim was specifically that models themselves are biased. It is that models are inherently biased because the data they are based on is biased. That might sound the same, but there is a nuanced difference. If you are able to strip the bias from the data, the models will work fine. The problem is the data and not the models.
- refenestrator 6y ago
  At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? Let's say you have a racially-neutral observation of lower income, maybe disability status or a criminal rap in the past. That looks like a bad bet for a loan regardless of color, it just so happens that our society's created a statistical imbalance in those metrics.
  - klyrs 6y ago
    > At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? AI shouldn't take the blame. Blame the folks collecting biased data, or those making biased decisions encoded in the data. The data is known to be tainted. Blame those using that data to train models, and sell/rent/apply those models for profit. Blame the researchers who know, or should know, better but make breathless claims about how their AI can be used without regard for the impact if people follow their advice
    - refenestrator 6y ago
      But the underlying situation is biased. The data could be both accurate and unfair. Is it, just don't do data, same interest rate for everyone, no denials and amortize defaults across higher rates for lower-risk borrowers? You'd need a law, the first bank to do that would be crushed by other banks that can attract the lower-risk borrowers with lower rates.
      - klyrs 6y ago
        > You'd need a law, the first bank to do that would be crushed by other banks that can attract the lower-risk borrowers with lower rates. Curious. Is your answer to the title "we need laws to regulate AI?" I'm not specifically agreeing or disagreeing.
        refenestrator 6y ago
        I'm just talking about the bank loan thing. If you want all people to have access to bank loans on equal terms regardless of income or other signals, you'd have to enforce that with a law.
  - slg 6y ago
    >At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? I mean it is never truly the "AI's fault". It is the fault of the systems that resulted in the biased data and the people who ignored that bias while still delegating the decision making to that AI allowing it to be a tool that propagates those biases. You end up with a dangerous and self-perpetuating system like this: 1. Data is biased against Black people due to historic racism. 2. AI is built of this biased data. 3. AI results in a biased system that disadvantages Black people. 4. Black people get discriminated against due to the results from the AI. 5. New data is now even more biased against Black people. 6. Go to step 1.
    - refenestrator 6y ago
      I hear you on this but I think it crosses the line from 'ethics' to 'activism'. The ethical concern would be, "don't build a racist AI system that assumes black people don't pay back loans". This could be mitigated by removing race and clear proxies such as zip code from training data, but it doesn't necessarily break your cycle, because the poverty and social problems are real and exist regardless of whether the AI considers race. The anti-racist approach, on the other hand, would specifically require incorporating race so you can perform affirmative action. Maybe that's good for society, maybe there's a debate about how much and in what way you do it, who pays for it, etc. But it's left "ethics" far behind and become "politics" at that point IMO.
      - slg 6y ago
        >This could be mitigated by removing race and clear proxies such as zip code from training data It isn't this simple. Not all these proxies are clear. Also what are we even left with once all these proxies are removed? The zip code is hugely important when trying to establish the value of a home. Does removing all data that is potentially racially biased leave us with a model that is nearly worthless? >But it's left "ethics" far behind and become "politics" at that point IMO. It depends entirely on what branch of moral philosophy to which you subscribe. For a lot of people their personal politics is just applied ethics meaning not acting would be ethically wrong in their opinion.
- unishark 6y ago
  In statistical terms, a biased model has error in its estimate even with infinite data. This can happen if the model is too simple, for example if it classifies samples into two classes when there are really three, or if it uses a linear curve for a nonlinear relationship. That relates to the ethical or political version of "bias" if this error benefits some group over others. Though if it was just some basic model off the shelf, that bias could just as well benefit/penalize anyone.
klyrs 6y ago
Do you think it's reasonable to expect a model, trained on biased data, to not display a similar degree of bias?
- deleted 6y ago
  [deleted]

slg 6y ago

I'm not OP, but I don't think the claim was specifically that models themselves are biased. It is that models are inherently biased because the data they are based on is biased. That might sound the same, but there is a nuanced difference. If you are able to strip the bias from the data, the models will work fine. The problem is the data and not the models.

refenestrator 6y ago
At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? Let's say you have a racially-neutral observation of lower income, maybe disability status or a criminal rap in the past. That looks like a bad bet for a loan regardless of color, it just so happens that our society's created a statistical imbalance in those metrics.
- klyrs 6y ago
  > At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? AI shouldn't take the blame. Blame the folks collecting biased data, or those making biased decisions encoded in the data. The data is known to be tainted. Blame those using that data to train models, and sell/rent/apply those models for profit. Blame the researchers who know, or should know, better but make breathless claims about how their AI can be used without regard for the impact if people follow their advice
  - refenestrator 6y ago
    But the underlying situation is biased. The data could be both accurate and unfair. Is it, just don't do data, same interest rate for everyone, no denials and amortize defaults across higher rates for lower-risk borrowers? You'd need a law, the first bank to do that would be crushed by other banks that can attract the lower-risk borrowers with lower rates.
    - klyrs 6y ago
      > You'd need a law, the first bank to do that would be crushed by other banks that can attract the lower-risk borrowers with lower rates. Curious. Is your answer to the title "we need laws to regulate AI?" I'm not specifically agreeing or disagreeing.
      - refenestrator 6y ago
        I'm just talking about the bank loan thing. If you want all people to have access to bank loans on equal terms regardless of income or other signals, you'd have to enforce that with a law.
- slg 6y ago
  >At what point does this stop being AI's fault and start being an accurate observation of things that are society's fault? I mean it is never truly the "AI's fault". It is the fault of the systems that resulted in the biased data and the people who ignored that bias while still delegating the decision making to that AI allowing it to be a tool that propagates those biases. You end up with a dangerous and self-perpetuating system like this: 1. Data is biased against Black people due to historic racism. 2. AI is built of this biased data. 3. AI results in a biased system that disadvantages Black people. 4. Black people get discriminated against due to the results from the AI. 5. New data is now even more biased against Black people. 6. Go to step 1.
  - refenestrator 6y ago
    I hear you on this but I think it crosses the line from 'ethics' to 'activism'. The ethical concern would be, "don't build a racist AI system that assumes black people don't pay back loans". This could be mitigated by removing race and clear proxies such as zip code from training data, but it doesn't necessarily break your cycle, because the poverty and social problems are real and exist regardless of whether the AI considers race. The anti-racist approach, on the other hand, would specifically require incorporating race so you can perform affirmative action. Maybe that's good for society, maybe there's a debate about how much and in what way you do it, who pays for it, etc. But it's left "ethics" far behind and become "politics" at that point IMO.
    - slg 6y ago
      >This could be mitigated by removing race and clear proxies such as zip code from training data It isn't this simple. Not all these proxies are clear. Also what are we even left with once all these proxies are removed? The zip code is hugely important when trying to establish the value of a home. Does removing all data that is potentially racially biased leave us with a model that is nearly worthless? >But it's left "ethics" far behind and become "politics" at that point IMO. It depends entirely on what branch of moral philosophy to which you subscribe. For a lot of people their personal politics is just applied ethics meaning not acting would be ethically wrong in their opinion.
unishark 6y ago
In statistical terms, a biased model has error in its estimate even with infinite data. This can happen if the model is too simple, for example if it classifies samples into two classes when there are really three, or if it uses a linear curve for a nonlinear relationship. That relates to the ethical or political version of "bias" if this error benefits some group over others. Though if it was just some basic model off the shelf, that bias could just as well benefit/penalize anyone.

klyrs 6y ago

Do you think it's reasonable to expect a model, trained on biased data, to not display a similar degree of bias?

deleted 6y ago
[deleted]