3 ms·

>Now without any malice from the developer, the system has become racist: It saw the correlation that in the US blacks and hispanics live in poverty more often

by bko 9y ago

>Now without any malice from the developer, the system has become racist: It saw the correlation that in the US blacks and hispanics live in poverty more often that others [1]. If the data really suggests that these factors (poor areas, foreign names, social media) affect liklihood of default, should they be completely ignored? I'm sure credit scores and salaries are correlated to race in the US too. Should those be ignored? Just give out loans indiscriminately? They're not ignored now by loan approval. Worse, inaccurate stereotypes are used as a proxy for some of these factors that may be relevant to a borrowers ability to pay. We don't live in a perfect world where people ignore certain factors.

barrkel 9y ago
Are you familiar with correlation vs causation? Without a causal link, making an inference based on correlation is unsound. Encoding unsound reasoning in an AI model, particularly when it reinforces an existing social imbalance, would be less than ethical. "Weapons of Math Destruction" by Cathy O'Neil delves into this in more depth. It's a very valid concern, particularly in the way non-technical people are trained over time to give deference to the algorithm.
- bem94 9y ago
  +1 for Weapons of Math Destruction.
- bko 9y ago
  I'm not sure what you mean. AI doesn't involve "encoding unsound reasoning" into anything. That's more akin to how humans think and act, by making decisions based on heuristics learned from a lifetime of experience and societal influences. It would be unethical to encode a "race" factor to be considered in an algorithm, although even if it were included I think it would prove insignificant since race has no direct causation to ability to prepay when you consider all the other factors (e.g. poor white person in same neighborhood would have same default prob). However other factors might, such as the ones you have listed (where they live, credit score, etc) I am familiar with the book you mentioned. You still didn't answer my question. If credit score and income are correlated to race in America, should they be ignored from loan applications? What are valid factors to consider?
  - barrkel 9y ago
    > If credit score and income are correlated to race in America, should they be ignored from loan applications? Is correlation causation? Is it fair to encode a judgement based on correlation but not causation? Or to make your position more clear, do you support racial profiling? If not, why not, and how do you justify that position, and how does that justification apply to judgements based on correlation but not causation? (My position is that making decisions based on correlation but not causation is the general case for which racial profiling is a specific case.)
    - bko 9y ago
      I already answered that. No race should not be a factor as it does not convey any info not explained by other factors. What are factors that are causation? Presumably not where you're living and other considerations you determined to be implicitly racist

by bko 9y ago

barrkel 9y ago

Are you familiar with correlation vs causation? Without a causal link, making an inference based on correlation is unsound. Encoding unsound reasoning in an AI model, particularly when it reinforces an existing social imbalance, would be less than ethical. "Weapons of Math Destruction" by Cathy O'Neil delves into this in more depth. It's a very valid concern, particularly in the way non-technical people are trained over time to give deference to the algorithm.

bem94 9y ago
+1 for Weapons of Math Destruction.
bko 9y ago
I'm not sure what you mean. AI doesn't involve "encoding unsound reasoning" into anything. That's more akin to how humans think and act, by making decisions based on heuristics learned from a lifetime of experience and societal influences. It would be unethical to encode a "race" factor to be considered in an algorithm, although even if it were included I think it would prove insignificant since race has no direct causation to ability to prepay when you consider all the other factors (e.g. poor white person in same neighborhood would have same default prob). However other factors might, such as the ones you have listed (where they live, credit score, etc) I am familiar with the book you mentioned. You still didn't answer my question. If credit score and income are correlated to race in America, should they be ignored from loan applications? What are valid factors to consider?
- barrkel 9y ago
  > If credit score and income are correlated to race in America, should they be ignored from loan applications? Is correlation causation? Is it fair to encode a judgement based on correlation but not causation? Or to make your position more clear, do you support racial profiling? If not, why not, and how do you justify that position, and how does that justification apply to judgements based on correlation but not causation? (My position is that making decisions based on correlation but not causation is the general case for which racial profiling is a specific case.)
  - bko 9y ago
    I already answered that. No race should not be a factor as it does not convey any info not explained by other factors. What are factors that are causation? Presumably not where you're living and other considerations you determined to be implicitly racist