4 ms·

I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrim

I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)

jacquesm 10y ago
Interesting aside (but still On Topic): This works for IP addresses as well as zip codes.
yummyfajitas 10y ago
But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores. In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive. I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.
- InTheArena 10y ago
  True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.
  - vkou 10y ago
    The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.
    - Houshalter 10y ago
      The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.
      - yummyfajitas 10y ago
        If you can come up with an algorithm that reproduces this conclusion - with accuracy even remotely close to the "racist" ones - banks will cross heaven and earth to pay you $billions. Unfortunately the only public analysis I'm aware of is from a blog using Zillow data: https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... This effect is reproduced in various walks of life, e.g. education. You'll be doing social good in addition to earning billions of dollars. What's not to like?
        Houshalter 10y ago
        I do in fact have an algorithm to remove racism from models, but I doubt its worth "$billions". The whole point of my argument is that it shouldn't be necessary. Surely you don't really believe racist stereotypes are true?
        yummyfajitas 10y ago
        I do, in fact. There is extensive research supporting the fact that many (though not all) are accurate. The typical racist stereotype is about twice as likely to replicate as the typical sociology paper. Here are a couple of review articles to get you started: http://www.spsp.org/blog/stereotype-accuracy-response http://www.spsp.org/blog/stereotype-accuracy-response http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-unbearable-accuracy-of-stereotypes.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... I didn't say that simply removing "racism" (by which I assume you mean disparate impact) is worth billions. I said doing so with the same accuracy as models which are "racist" is worth billions. Obviously you can do anything you want if you give up accuracy. Why do you believe they are false? Simply because it's socially unacceptable to think otherwise?
analyst74 10y ago
I personally draw the line of acceptable and unacceptable discrimination at "if the disadvantaged person can change that aspect". Skin color cannot be changed, but zipcode certainly can. There is no rule I'm aware of that restricts visible minorities from living in rich neighbourhoods.
- InTheArena 10y ago
  Except of course of the fact that visible minorities can not in fact afford to live there, or get a loan to live there, because they currently live in a poor neighborhood.

jacquesm 10y ago

Interesting aside (but still On Topic): This works for IP addresses as well as zip codes.

yummyfajitas 10y ago

But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores. In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive. I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.

InTheArena 10y ago
True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.
- vkou 10y ago
  The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.
  - Houshalter 10y ago
    The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.
    - yummyfajitas 10y ago
      If you can come up with an algorithm that reproduces this conclusion - with accuracy even remotely close to the "racist" ones - banks will cross heaven and earth to pay you $billions. Unfortunately the only public analysis I'm aware of is from a blog using Zillow data: https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... This effect is reproduced in various walks of life, e.g. education. You'll be doing social good in addition to earning billions of dollars. What's not to like?
      - Houshalter 10y ago
        I do in fact have an algorithm to remove racism from models, but I doubt its worth "$billions". The whole point of my argument is that it shouldn't be necessary. Surely you don't really believe racist stereotypes are true?
        yummyfajitas 10y ago
        I do, in fact. There is extensive research supporting the fact that many (though not all) are accurate. The typical racist stereotype is about twice as likely to replicate as the typical sociology paper. Here are a couple of review articles to get you started: http://www.spsp.org/blog/stereotype-accuracy-response http://www.spsp.org/blog/stereotype-accuracy-response http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-unbearable-accuracy-of-stereotypes.pdf http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... I didn't say that simply removing "racism" (by which I assume you mean disparate impact) is worth billions. I said doing so with the same accuracy as models which are "racist" is worth billions. Obviously you can do anything you want if you give up accuracy. Why do you believe they are false? Simply because it's socially unacceptable to think otherwise?

analyst74 10y ago

I personally draw the line of acceptable and unacceptable discrimination at "if the disadvantaged person can change that aspect". Skin color cannot be changed, but zipcode certainly can. There is no rule I'm aware of that restricts visible minorities from living in rich neighbourhoods.

InTheArena 10y ago
Except of course of the fact that visible minorities can not in fact afford to live there, or get a loan to live there, because they currently live in a poor neighborhood.