4 ms·

But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on

But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores. In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive. I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.

InTheArena 10y ago
True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.
- vkou 10y ago
  The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.
  - Houshalter 10y ago
    The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.
    - yummyfajitas 10y ago
      If you can come up with an algorithm that reproduces this conclusion - with accuracy even remotely close to the "racist" ones - banks will cross heaven and earth to pay you $billions. Unfortunately the only public analysis I'm aware of is from a blog using Zillow data: https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... This effect is reproduced in various walks of life, e.g. education. You'll be doing social good in addition to earning billions of dollars. What's not to like?
      - Houshalter 10y ago
        I do in fact have an algorithm to remove racism from models, but I doubt its worth "$billions". The whole point of my argument is that it shouldn't be necessary. Surely you don't really believe racist stereotypes are true?
        yummyfajitas 10y ago
        I do, in fact. There is extensive research supporting the fact that many (though not all) are accurate. The typical racist stereotype is about twice as likely to replicate as the typical sociology paper. Here are a couple of review articles to get you started: http://www.spsp.org/blog/stereotype-accuracy-response http://www.spsp.org/blog/stereotype-accuracy-response http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-unbearable-accuracy-of-stereotypes.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... I didn't say that simply removing "racism" (by which I assume you mean disparate impact) is worth billions. I said doing so with the same accuracy as models which are "racist" is worth billions. Obviously you can do anything you want if you give up accuracy. Why do you believe they are false? Simply because it's socially unacceptable to think otherwise?
        Houshalter 10y ago
        I'm not sure whether or not the stereotype is true. The point is you are forced to acknowledge a pretty unpopular belief to defend this idea. And that the kind of people super concerned about disparate impact are usually not the kind of people that believe that. And I think that if you do acknowledge there are differences, the argument that it's unfair and wrong is much less strong. In any case the link you posted earlier suggested it could be explained by different rates of numeracy and literacy. Which sound like obvious proxies for IQ. If you gave people an IQ test, or at least literacy or numeracy test, this would probably eliminate any racist bias. > I said doing so with the same accuracy as models which are "racist" is worth billions. You necessarily lose some accuracy if you really think race is predictive. But you need not lose all of it. The current methods for removing bias just fuzz features that correlate with race. But there is a way to make sure a model doesn't learn to use features just as a proxy for race, but instead to predict how much they matter independent of race.

InTheArena 10y ago

True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.

vkou 10y ago
The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.
- Houshalter 10y ago
  The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.
  - yummyfajitas 10y ago
    If you can come up with an algorithm that reproduces this conclusion - with accuracy even remotely close to the "racist" ones - banks will cross heaven and earth to pay you $billions. Unfortunately the only public analysis I'm aware of is from a blog using Zillow data: https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... This effect is reproduced in various walks of life, e.g. education. You'll be doing social good in addition to earning billions of dollars. What's not to like?
    - Houshalter 10y ago
      I do in fact have an algorithm to remove racism from models, but I doubt its worth "$billions". The whole point of my argument is that it shouldn't be necessary. Surely you don't really believe racist stereotypes are true?
      - yummyfajitas 10y ago
        I do, in fact. There is extensive research supporting the fact that many (though not all) are accurate. The typical racist stereotype is about twice as likely to replicate as the typical sociology paper. Here are a couple of review articles to get you started: http://www.spsp.org/blog/stereotype-accuracy-response http://www.spsp.org/blog/stereotype-accuracy-response http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-unbearable-accuracy-of-stereotypes.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... I didn't say that simply removing "racism" (by which I assume you mean disparate impact) is worth billions. I said doing so with the same accuracy as models which are "racist" is worth billions. Obviously you can do anything you want if you give up accuracy. Why do you believe they are false? Simply because it's socially unacceptable to think otherwise?
        Houshalter 10y ago
        I'm not sure whether or not the stereotype is true. The point is you are forced to acknowledge a pretty unpopular belief to defend this idea. And that the kind of people super concerned about disparate impact are usually not the kind of people that believe that. And I think that if you do acknowledge there are differences, the argument that it's unfair and wrong is much less strong. In any case the link you posted earlier suggested it could be explained by different rates of numeracy and literacy. Which sound like obvious proxies for IQ. If you gave people an IQ test, or at least literacy or numeracy test, this would probably eliminate any racist bias. > I said doing so with the same accuracy as models which are "racist" is worth billions. You necessarily lose some accuracy if you really think race is predictive. But you need not lose all of it. The current methods for removing bias just fuzz features that correlate with race. But there is a way to make sure a model doesn't learn to use features just as a proxy for race, but instead to predict how much they matter independent of race.