4 ms·

It's hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the

by jfasi 8y ago

It's hard to be believe these days, but once upon a time language models were written by hand. Imagine you hand-wrote such a model and put it to use in (for the sake of example) a psychological evaluation application. Now imagine that after years of use you discover that your model systematically marks african americans as less psychologically fit than white americans. Who would be to blame? Naturally, you would. Your actions led to a biased model being used to unjustly and arbitrarily harm innocent people, and your leadership would be right to call into question every decision your application ever made. Now imagine the same scenario except your app was trained on data instead of hand-written. Make no mistake, the answer to the question of who's to blame is exactly the same: the developer. The response should be exactly the same: a complete loss of confidence in the model. I'm appalled that this needs to be said, but reading this comments section I'm afraid it does: Machine learning models are inference and pattern recognition devices, not scientific tools. They don't magically reveal hidden patterns in the world, they repeat the patterns that the developers train them on. If you trained a machine learning model to perform psychological evaluations [1] or sentence convicts [2] or recognize faces [3], and your model is biased in a way that is unnecessary and unjust, your model is bad you should be held accountable for its failures. [1] https://affect.media.mit.edu/projects.php?id=4079 https://affect.media.mit.edu/projects.php?id=4079 [2] https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso... [3] https://www.wnycstudios.org/story/deep-problem-deep-learning/ https://www.wnycstudios.org/story/deep-problem-deep-learning...

deleted 8y ago
[deleted]
whatshisface 8y ago
What do you do when your classification is correlated to race through no fault of yours? For example, I might successfully predict credit score from backyard size, and end up with a correlation to race that doesn't have anything to do with me. I don't think the blame always lays with the algorithm, especially when it doesn't have access to race as an input (this is a reasonable expectation). I can score students with a simple algorithm based on what they write on their math tests, and even that's going to correlate with race. In that case the blame pretty clearly lies in the process that produced the reality that's being measured, not the measurement technique itself. Let's say that black people default on their loans more often then white people. Is it better to criticize the math that discovered that fact, or the root cause that made it true to begin with?
- DmenshunlAnlsis 8y ago
  It sounds like a way to set up a positive feedback loop with terrible consequences to me. I don’t know enough about ML to be sure, but I know GIGO. If you feed bias into a model then let the model guide decision making, that seems obviously flawed. You bias the model not to give loans or credit to black people, reinforcing their lack of credit, which leads to the model confirming its bias. Maybe you can claim it’s a socioeconomic bias and not racist, and in some countries it would be true, but not in America. How long ago was it that black people had to fight past mobs to integrate schools? How long since redlining? If socioeconomic are tied to a history of oppression based on race, then ignoring half of that equation isn’t honest. Correlation is not causation, but does “AI” know that?
  - whatshisface 8y ago
    If someone is poor but you want them to be given a chance anyways, you don't go to the banker and say, "please close your eyes to what by all means seems to be true." You say, "I know he's more likely to default, but society wants him lifted out of poverty and you're going to give him that loan."
    - DmenshunlAnlsis 8y ago
      If those banks had to turn to the government to keep them afloat, and those banks played a large role in creating the problem in the first place, maybe you do? Maybe not, but what you definitely don’t do is set up black boxes to take the heat for centuries of prejudice and mistreatment.
      - whatshisface 8y ago
        > don’t [...] set up black boxes to take the heat I think I see now how we agree. This whole "algorithms" thing is a smokescreen over what is really just the AA debate: should institutions optimize purely for their own goals (in which case they would do nothing to help with old wounds, whenever ignoring them would be cheaper), or should they be expected to mix in social responsibility into their cost functions? Whether it's AI or a cunning but amoral banker, it's the same question.

by jfasi 8y ago

deleted 8y ago

[deleted]

whatshisface 8y ago

What do you do when your classification is correlated to race through no fault of yours? For example, I might successfully predict credit score from backyard size, and end up with a correlation to race that doesn't have anything to do with me. I don't think the blame always lays with the algorithm, especially when it doesn't have access to race as an input (this is a reasonable expectation). I can score students with a simple algorithm based on what they write on their math tests, and even that's going to correlate with race. In that case the blame pretty clearly lies in the process that produced the reality that's being measured, not the measurement technique itself. Let's say that black people default on their loans more often then white people. Is it better to criticize the math that discovered that fact, or the root cause that made it true to begin with?

DmenshunlAnlsis 8y ago
It sounds like a way to set up a positive feedback loop with terrible consequences to me. I don’t know enough about ML to be sure, but I know GIGO. If you feed bias into a model then let the model guide decision making, that seems obviously flawed. You bias the model not to give loans or credit to black people, reinforcing their lack of credit, which leads to the model confirming its bias. Maybe you can claim it’s a socioeconomic bias and not racist, and in some countries it would be true, but not in America. How long ago was it that black people had to fight past mobs to integrate schools? How long since redlining? If socioeconomic are tied to a history of oppression based on race, then ignoring half of that equation isn’t honest. Correlation is not causation, but does “AI” know that?
- whatshisface 8y ago
  If someone is poor but you want them to be given a chance anyways, you don't go to the banker and say, "please close your eyes to what by all means seems to be true." You say, "I know he's more likely to default, but society wants him lifted out of poverty and you're going to give him that loan."
  - DmenshunlAnlsis 8y ago
    If those banks had to turn to the government to keep them afloat, and those banks played a large role in creating the problem in the first place, maybe you do? Maybe not, but what you definitely don’t do is set up black boxes to take the heat for centuries of prejudice and mistreatment.
    - whatshisface 8y ago
      > don’t [...] set up black boxes to take the heat I think I see now how we agree. This whole "algorithms" thing is a smokescreen over what is really just the AA debate: should institutions optimize purely for their own goals (in which case they would do nothing to help with old wounds, whenever ignoring them would be cheaper), or should they be expected to mix in social responsibility into their cost functions? Whether it's AI or a cunning but amoral banker, it's the same question.