6 ms·

Schneier makes a startling claim: It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. This

Schneier makes a startling claim: It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. This concept needs to be expanded. We as a society need to understand what we expect out of the algorithms that automatically judge us and ensure that those expectations are met. Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above. If we want to mandate the outcome of our decision process, let's just do that openly and honestly. E.g. for credit, just impose a quota of credit that must be issued to each protected class. If we want fair and unbiased decisions then we need to instead examine the internals of our process and accept the outcomes, whatever they may be. But we need to stop pretending we can do both.

Nutmog 11y ago
This is going to get really hairy. Who gets to decide the protected classes? The classic ones grouped by race, religion, gender, etc will be too crude when computers are judging. Imagine some weird class like people who are old but don't wear prescription glasses - perhaps indicating that they're too irresponsible with money to be able to pay for them. Nevermind that they're an exception whose eyesight is OK if that data isn't visible to the computer. There could be all kinds of unexpected special cases like this that lead to individuals getting generally bad treatment throughout life.
- barry-cotter 11y ago
  Protected classes aren't about justice, they're about identity politics. If a group can deliver votes or violence as a bloc and demand group representation then they'll get it. But invisible groups like left-handers or citizens who are not native born have immense difficulty coordinating and mostly don't.
masterzora 11y ago
> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. That's an interesting interpretation. To me it looks more like he's discussing that bias can (probably will) be baked into algorithms even unintentionally and often subtly. And that highlights a significant difficulty with the "unbiased predictors" thing you say: can you distinguish between an unbiased algorithm discovering that something that looks like bias isn't and bias being subtly baked into the algorithm? I think that's where the "we need to understand what we expect out of the algorithms and ensure the expectations are met" bit comes in, at least in part.
- yummyfajitas 11y ago
  Definitely - one makes more money than the other. If a bank is biased against some group they are turning away profitable customers. This is also purely a statistical problem; once some quant discovers they can make more money by fixing the bias, they'll do it. Understanding what to expect out of the algorithms is absolutely the wrong way to determine this. The fact is that we simply don't know apriori the optimal way to allocate credit. That's why we need an algorithm in the first place.
  - pron 11y ago
    Or, having realized that algorithms making predictions based on past data are politically conservative (pretty much by definition), if we really want to improve the economy, we should take political action to change the situation. This is how it's always been done, and it can work; sometimes rather well.
    - yummyfajitas 11y ago
      If you want to force banks to make suboptimal credit allocation decisions for political reasons, go ahead and openly advocate for it. And be sure to openly accept blame for it if those loans go bad and we have another banking crisis. Just don't try to claim you are attempting to eliminate bias. Eliminating bias is a job for statisticians, not politicians.
      - pron 11y ago
        > If you want to force banks to make suboptimal credit allocation decisions for political reasons, go ahead and openly advocate for it You and I both want to force banks to make suboptimal credit allocations for political reasons (or we both don't). Leaving the local optimum can only be done with coordinated action, but that doesn't mean that your algorithm (that keeps us there) is any less political. By not calling for social action, you are forcing banks to stay in their Nash equilibrium, which is suboptimal for them. What I want is for us to acknowledge that we have (possibly unintentionally) programmed our algorithms to be conservative, and then make the conscious decision whether we want conservative algorithms or progressive algorithms. Sadly, neutral algorithms are impossible[1] if based on their output we take actions that may affect the distribution of power. I'm a progressive and I want a progressive agenda. You're a conservative and want a conservative agenda. Both are perfectly fine, but you must understand that there's nothing neutral in your stance. I believe political action for social change often works and can be very helpful, so I wish to encourage it; you believe the opposite, and therefore wish to discourage it. But neither of us is neutral, and your position is not based on math; your math simply serves your position. Your own equations deliberately omit feedback that we know for a fact (through the study of history) to exist. You make assumptions that reflect your conservative values just as I make assumptions reflecting my liberal values. > Eliminating bias is a job for statisticians, not politicians. You think that when people talk of social bias they mean statistical bias? This may sometimes happen to be (also) true, but in general, when people talk of social bias they mean any human behavior that creates a dynamics for present and future unfair distributions of power. Sadly, statisticians can't eliminate that. Or, rather they can't eliminate that alone, because politics is not (just) what politicians do. Politics[2] is what we all do when we make decisions that affect the distribution of power in society. Interpreting statistical snapshots as predictors of future human behavior is not statistics but politics. Statistics tells us what is (or sometimes how things will likely be if we change nothing); certainly not what we should do. ------ [1]: Well, it's possible to pretend some are neutral if we artificially (and incorrectly) reduce the domain of our value functions. In an interconnected system like human society, it is is disingenuous to direct our action based on its outcome for a very limited sector (like a bank), without saying what we want the action to achieve for all other sectors. So, for example, if you ask me if I want bankers to be happy, then the answer is, of course I do. But that doesn't mean I'll favor any action that makes them happy if it also makes other people less happy. My point is that our value functions must be total; we can't be lazy and assign a value to a narrow look at an outcome, but we must assign a value for a complete outcome (i.e. for all of humanity). [2]: https://en.wikipedia.org/wiki/Politics https://en.wikipedia.org/wiki/Politics : Politics... is the practice and theory of influencing other people... Furthermore, politics is the study or practice of the distribution of power and resources within a given community... as well as the interrelationship(s) between communities.
laughinghan 11y ago
This sounds reasonable but you're missing a piece here. In addition to algorithms and output, there's a 3rd piece: input. We don't want to mandate the outcome of our decision process. We want a fair and unbiased decision process where if the input is fair, the output will be fair. We want fair and unbiased decisions, but in addition to the internals of our process, we need to ensure the input is fair before we'll accept the outcomes. The real problem is people pretending the input is fair, that the effects of historical discrimination has evaporated, against protected classes or otherwise (the queer community comes to mind).
- yummyfajitas 11y ago
  We don't need fair and unbiased inputs. In the real world these don't exist, and the field of statistics has devoted extensive effort to the problem of getting good answers out of bad inputs. E.g., for an unbiased measurement (m = x+g, g ~ N(0,\sigma^2)), you use the likelihood function G(x-m,\sigma^2). For a biased measurement g ~ N(b, \sigma^2), you use the likelihood function G(x-m+b, \sigma^2). Here G(x,\sigma^2) is the pdf of the distribution the noise g is drawn from. But this is mainly just something that can be determined by studying the process. Mandating specific outcomes is just a way of ensuring that the predictor can't discover certain facts.
  - laughinghan 11y ago
    I'm not sure what your example is supposed to prove. Besides the fact that your likelihood function isn't an "unbiased algorithm" like your original comment was talking about, it assumes the bias of the input is known and accepted rather than heavily contested. We certainly shouldn't mandate specific outcomes. We should analyze processes, outcomes, and inputs alike for bias.
    - yummyfajitas 11y ago
      The algorithm would be Bayesian inference using the aforementioned likelihood. The example is a simple illustration of how to fix bias. If bias is unknown then treat it as another unknown variable and use bayesian inference to identify it. If this is unfamiliar, I'm actually publishing a blog post about it tomorrow (how to find the bias in your phone's compass) which you might find helpful.
pron 11y ago
> unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors Absolutely. Or the exact opposite: that the algorithms simply reflect results caused by bias, a far more likely relationship given what we know of history. The reason we're against this is not because we think our biases are not supported by real data; we know that some of them most certainly are. But we also know that a snapshot of current society is often little more than a reflection of social forces that has made the society what it is at this particular point in time, and treating them as immutable (or simply placing faith in their persistence) is nothing more than a political policy that helps perpetuate them. Treating past correlations as future predictors in a dynamic system such as society is programming conservatism into your algorithm. Acting on them is a political policy which strives to maintain the status quo -- there's nothing wrong about it per se, if that's what you want, but we need to be aware of this fact. Alternatively, we could program the algorithms to predict or even assist with social change. The worst thing we can do is accidentally program politics into algorithms, and then fall for our own ruse and start believing that the algorithms are apolitical. That takes away our agency in making political choices.
- barry-cotter 11y ago
  > Or the exact opposite: that the algorithms simply reflect results caused by bias, a far more likely relationship given what we know of history. Unsupported assertion > But we also know that a snapshot of current society is often little more than a reflection of social forces that has made the society what it is at this particular point in time, and treating them as immutable (or simply placing faith in their persistence) is nothing more than a political policy that helps perpetuate them. This is anti-inductive. Knowledge can only be contingent based on past experience. This is John Stuart Mill stuff, empiricism. > Treating past correlations as future predictors in a dynamic system such as society is programming conservatism into your algorithm. Yes. All anyone has to go one in making predictions is past data. If something changes radically in the environment then the model may break. This does not mean we should just make shit up for some of our sample data/training set. > Acting on them is a political policy which strives to maintain the status quo -- there's nothing wrong about it per se, if that's what you want, but we need to be aware of this fact. A model of the world can be more or less accurate. That is apolitical. It can be biased. This can be a political matter. So reduce or eliminate the bias. > Alternatively, we could program the algorithms to predict or even assist with social change. If you want to enact social change go for it but stop pretending that this has anything to do with statistics, modeling, prediction or math, just go straight to the politics.
AnthonyMouse 11y ago
> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above. There is a legitimate question as to whether an algorithm is actually unbiased. It could be quite accurate or there could be a bias, e.g. because the algorithm penalizes drug convictions and drug convictions have an unjustified bias due to racist or sexist enforcement. The problem is it isn't a simple question to answer. And people are going to want to choose their answer based on their politics. But that can't work. We can't let a black box take discriminatory inputs and expect it to produce a just result. We also can't just assume that every unequal outcome had an unjust cause, or we'll end up e.g. causing loans to be given to uncreditworthy people and wrecking the economy (see also housing crash). Which means that Schneier is wrong about quotas but exactly right that what we need is transparency. Because if there is unjustified bias then we need to understand why so we can prevent it, and if there isn't then we need to understand why so we can accept it.
- yummyfajitas 11y ago
  Consider your drug conviction example, and lets say the algorithm is linear regression so I can give a very simple explanation. The model will then be FICO = stuff + (-1) x (drug conviction). I.e., your FICO score is lowered by 1 point if you've been convicted of drug use. If you are correct, some enterprising quant can take your hypothesis. He can then rerun his regression with (drug conviction, black) and (drug conviction, white) as variables. If your hypothesis is correct the result will be: FICO = stuff + (-0.5) x (drug conviction, black) + (-1.5) x (drug conviction, white). This is because the bias in drug convictions makes it less predictive for black people. If you are wrong the coefficients will be equal [1]. If you are right he just made millions of dollars for the bank he works for and probably added $100k to his bonus. What you are discussing is strictly a statistics problem. [1] For examples of this type of analysis on social indicators (I don't know of any on credit allocation), see these papers: https://www.rand.org/content/dam/rand/www/external/labor/seminars/adp/pdfs/adp_ajph.pdf https://www.rand.org/content/dam/rand/www/external/labor/sem... http://egov.ufsc.br/portal/sites/default/files/anexos/33027-41458-1-PB.pdf http://egov.ufsc.br/portal/sites/default/files/anexos/33027-... http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf
imh 11y ago
It's not just about bias. Even if we know there is predictive value, there's a notion of whether it's fair to include it. Regardless of what it correlates with, it seems unfair to make someone more/less likely to get a loan because of the color of their skin (or anything else they cannot change). There are some attributes that we should turn a blind eye to.
- nickff 11y ago
  Schneier's point is not only that those attributes should be ignored, but there should be no correlation between those attributes and the end result of the algorithm. You can quite easily find examples of algorithms which do not include race or gender as an input, yet have a disparate impact. Schneier is saying that algorithms with disparate impact should be avoided without regard to procedural fairness, proportionality of outcome, or correctness.
- Houshalter 11y ago
  Why should fairness be a goal though? If men are really are more dangerous drivers, why shouldn't they pay more in car insurance? Or teenagers? It's not like they can control their age, but they are objectively more likely to get into an accident. So are people with eyesight problems, etc. Algorithms should make the most accurate predictions possible. If somehow we had a time machine and could just see the future, we would do that. People who are more likely to get in accidents should pay higher insurance, or bank loans, or whatever. If you are less of a risk, then you should pay less. Omitting statistically relevant features is silly. It's not like the algorithm has human biases. It doesn't hate people of a different race, it's not sexist. It just looks at the cold hard facts and makes the best prediction possible.
msellout 11y ago
>> It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. > unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors That may be true, but Schneier is speaking on a different topic -- redlining. There's a long history of people discriminating based on race, sex, class, etc. and pretending they're using an unbiased algorithm instead. The word redlining comes from the practice of using geography to make decisions. It's obvious to most people that geography is highly correlated with race and class. Therefore, even if geography itself carries no intrinsic bias, to use geography as a variable in evaluating credit-worthiness is discriminatory. https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining
david_ar 11y ago
The problem is not so much about whether predictions are biased. It's a problem of whether the algorithm is reinforcing pre-existing biases. In other words, it's about taking short-term losses in return for long-term gains. Suppose a certain group of people have experienced some kind of historical discrimination, and as a result people in that group are starting from a disadvantaged position (wealth, etc) compared to the rest of the population. It may well be an unbiased prediction that such people will be a greater credit risk, etc. However, by making decisions solely on the basis of this, although the bank saves some money in the short-term, it reinforces the cycle of discrimination. If your goal is to improve society in the long-term, making decisions based on these short-term predictors is not a desirable thing to do.

Nutmog 11y ago

This is going to get really hairy. Who gets to decide the protected classes? The classic ones grouped by race, religion, gender, etc will be too crude when computers are judging. Imagine some weird class like people who are old but don't wear prescription glasses - perhaps indicating that they're too irresponsible with money to be able to pay for them. Nevermind that they're an exception whose eyesight is OK if that data isn't visible to the computer. There could be all kinds of unexpected special cases like this that lead to individuals getting generally bad treatment throughout life.

barry-cotter 11y ago
Protected classes aren't about justice, they're about identity politics. If a group can deliver votes or violence as a bloc and demand group representation then they'll get it. But invisible groups like left-handers or citizens who are not native born have immense difficulty coordinating and mostly don't.

masterzora 11y ago

> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. That's an interesting interpretation. To me it looks more like he's discussing that bias can (probably will) be baked into algorithms even unintentionally and often subtly. And that highlights a significant difficulty with the "unbiased predictors" thing you say: can you distinguish between an unbiased algorithm discovering that something that looks like bias isn't and bias being subtly baked into the algorithm? I think that's where the "we need to understand what we expect out of the algorithms and ensure the expectations are met" bit comes in, at least in part.

yummyfajitas 11y ago
Definitely - one makes more money than the other. If a bank is biased against some group they are turning away profitable customers. This is also purely a statistical problem; once some quant discovers they can make more money by fixing the bias, they'll do it. Understanding what to expect out of the algorithms is absolutely the wrong way to determine this. The fact is that we simply don't know apriori the optimal way to allocate credit. That's why we need an algorithm in the first place.
- pron 11y ago
  Or, having realized that algorithms making predictions based on past data are politically conservative (pretty much by definition), if we really want to improve the economy, we should take political action to change the situation. This is how it's always been done, and it can work; sometimes rather well.
  - yummyfajitas 11y ago
    If you want to force banks to make suboptimal credit allocation decisions for political reasons, go ahead and openly advocate for it. And be sure to openly accept blame for it if those loans go bad and we have another banking crisis. Just don't try to claim you are attempting to eliminate bias. Eliminating bias is a job for statisticians, not politicians.
    - pron 11y ago
      > If you want to force banks to make suboptimal credit allocation decisions for political reasons, go ahead and openly advocate for it You and I both want to force banks to make suboptimal credit allocations for political reasons (or we both don't). Leaving the local optimum can only be done with coordinated action, but that doesn't mean that your algorithm (that keeps us there) is any less political. By not calling for social action, you are forcing banks to stay in their Nash equilibrium, which is suboptimal for them. What I want is for us to acknowledge that we have (possibly unintentionally) programmed our algorithms to be conservative, and then make the conscious decision whether we want conservative algorithms or progressive algorithms. Sadly, neutral algorithms are impossible[1] if based on their output we take actions that may affect the distribution of power. I'm a progressive and I want a progressive agenda. You're a conservative and want a conservative agenda. Both are perfectly fine, but you must understand that there's nothing neutral in your stance. I believe political action for social change often works and can be very helpful, so I wish to encourage it; you believe the opposite, and therefore wish to discourage it. But neither of us is neutral, and your position is not based on math; your math simply serves your position. Your own equations deliberately omit feedback that we know for a fact (through the study of history) to exist. You make assumptions that reflect your conservative values just as I make assumptions reflecting my liberal values. > Eliminating bias is a job for statisticians, not politicians. You think that when people talk of social bias they mean statistical bias? This may sometimes happen to be (also) true, but in general, when people talk of social bias they mean any human behavior that creates a dynamics for present and future unfair distributions of power. Sadly, statisticians can't eliminate that. Or, rather they can't eliminate that alone, because politics is not (just) what politicians do. Politics[2] is what we all do when we make decisions that affect the distribution of power in society. Interpreting statistical snapshots as predictors of future human behavior is not statistics but politics. Statistics tells us what is (or sometimes how things will likely be if we change nothing); certainly not what we should do. ------ [1]: Well, it's possible to pretend some are neutral if we artificially (and incorrectly) reduce the domain of our value functions. In an interconnected system like human society, it is is disingenuous to direct our action based on its outcome for a very limited sector (like a bank), without saying what we want the action to achieve for all other sectors. So, for example, if you ask me if I want bankers to be happy, then the answer is, of course I do. But that doesn't mean I'll favor any action that makes them happy if it also makes other people less happy. My point is that our value functions must be total; we can't be lazy and assign a value to a narrow look at an outcome, but we must assign a value for a complete outcome (i.e. for all of humanity). [2]: https://en.wikipedia.org/wiki/Politics https://en.wikipedia.org/wiki/Politics : Politics... is the practice and theory of influencing other people... Furthermore, politics is the study or practice of the distribution of power and resources within a given community... as well as the interrelationship(s) between communities.

laughinghan 11y ago

This sounds reasonable but you're missing a piece here. In addition to algorithms and output, there's a 3rd piece: input. We don't want to mandate the outcome of our decision process. We want a fair and unbiased decision process where if the input is fair, the output will be fair. We want fair and unbiased decisions, but in addition to the internals of our process, we need to ensure the input is fair before we'll accept the outcomes. The real problem is people pretending the input is fair, that the effects of historical discrimination has evaporated, against protected classes or otherwise (the queer community comes to mind).

yummyfajitas 11y ago
We don't need fair and unbiased inputs. In the real world these don't exist, and the field of statistics has devoted extensive effort to the problem of getting good answers out of bad inputs. E.g., for an unbiased measurement (m = x+g, g ~ N(0,\sigma^2)), you use the likelihood function G(x-m,\sigma^2). For a biased measurement g ~ N(b, \sigma^2), you use the likelihood function G(x-m+b, \sigma^2). Here G(x,\sigma^2) is the pdf of the distribution the noise g is drawn from. But this is mainly just something that can be determined by studying the process. Mandating specific outcomes is just a way of ensuring that the predictor can't discover certain facts.
- laughinghan 11y ago
  I'm not sure what your example is supposed to prove. Besides the fact that your likelihood function isn't an "unbiased algorithm" like your original comment was talking about, it assumes the bias of the input is known and accepted rather than heavily contested. We certainly shouldn't mandate specific outcomes. We should analyze processes, outcomes, and inputs alike for bias.
  - yummyfajitas 11y ago
    The algorithm would be Bayesian inference using the aforementioned likelihood. The example is a simple illustration of how to fix bias. If bias is unknown then treat it as another unknown variable and use bayesian inference to identify it. If this is unfamiliar, I'm actually publishing a blog post about it tomorrow (how to find the bias in your phone's compass) which you might find helpful.

pron 11y ago

> unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors Absolutely. Or the exact opposite: that the algorithms simply reflect results caused by bias, a far more likely relationship given what we know of history. The reason we're against this is not because we think our biases are not supported by real data; we know that some of them most certainly are. But we also know that a snapshot of current society is often little more than a reflection of social forces that has made the society what it is at this particular point in time, and treating them as immutable (or simply placing faith in their persistence) is nothing more than a political policy that helps perpetuate them. Treating past correlations as future predictors in a dynamic system such as society is programming conservatism into your algorithm. Acting on them is a political policy which strives to maintain the status quo -- there's nothing wrong about it per se, if that's what you want, but we need to be aware of this fact. Alternatively, we could program the algorithms to predict or even assist with social change. The worst thing we can do is accidentally program politics into algorithms, and then fall for our own ruse and start believing that the algorithms are apolitical. That takes away our agency in making political choices.

barry-cotter 11y ago
> Or the exact opposite: that the algorithms simply reflect results caused by bias, a far more likely relationship given what we know of history. Unsupported assertion > But we also know that a snapshot of current society is often little more than a reflection of social forces that has made the society what it is at this particular point in time, and treating them as immutable (or simply placing faith in their persistence) is nothing more than a political policy that helps perpetuate them. This is anti-inductive. Knowledge can only be contingent based on past experience. This is John Stuart Mill stuff, empiricism. > Treating past correlations as future predictors in a dynamic system such as society is programming conservatism into your algorithm. Yes. All anyone has to go one in making predictions is past data. If something changes radically in the environment then the model may break. This does not mean we should just make shit up for some of our sample data/training set. > Acting on them is a political policy which strives to maintain the status quo -- there's nothing wrong about it per se, if that's what you want, but we need to be aware of this fact. A model of the world can be more or less accurate. That is apolitical. It can be biased. This can be a political matter. So reduce or eliminate the bias. > Alternatively, we could program the algorithms to predict or even assist with social change. If you want to enact social change go for it but stop pretending that this has anything to do with statistics, modeling, prediction or math, just go straight to the politics.

AnthonyMouse 11y ago

> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above. There is a legitimate question as to whether an algorithm is actually unbiased. It could be quite accurate or there could be a bias, e.g. because the algorithm penalizes drug convictions and drug convictions have an unjustified bias due to racist or sexist enforcement. The problem is it isn't a simple question to answer. And people are going to want to choose their answer based on their politics. But that can't work. We can't let a black box take discriminatory inputs and expect it to produce a just result. We also can't just assume that every unequal outcome had an unjust cause, or we'll end up e.g. causing loans to be given to uncreditworthy people and wrecking the economy (see also housing crash). Which means that Schneier is wrong about quotas but exactly right that what we need is transparency. Because if there is unjustified bias then we need to understand why so we can prevent it, and if there isn't then we need to understand why so we can accept it.

yummyfajitas 11y ago
Consider your drug conviction example, and lets say the algorithm is linear regression so I can give a very simple explanation. The model will then be FICO = stuff + (-1) x (drug conviction). I.e., your FICO score is lowered by 1 point if you've been convicted of drug use. If you are correct, some enterprising quant can take your hypothesis. He can then rerun his regression with (drug conviction, black) and (drug conviction, white) as variables. If your hypothesis is correct the result will be: FICO = stuff + (-0.5) x (drug conviction, black) + (-1.5) x (drug conviction, white). This is because the bias in drug convictions makes it less predictive for black people. If you are wrong the coefficients will be equal [1]. If you are right he just made millions of dollars for the bank he works for and probably added $100k to his bonus. What you are discussing is strictly a statistics problem. [1] For examples of this type of analysis on social indicators (I don't know of any on credit allocation), see these papers: https://www.rand.org/content/dam/rand/www/external/labor/seminars/adp/pdfs/adp_ajph.pdf https://www.rand.org/content/dam/rand/www/external/labor/sem... http://egov.ufsc.br/portal/sites/default/files/anexos/33027-41458-1-PB.pdf http://egov.ufsc.br/portal/sites/default/files/anexos/33027-... http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf

imh 11y ago

It's not just about bias. Even if we know there is predictive value, there's a notion of whether it's fair to include it. Regardless of what it correlates with, it seems unfair to make someone more/less likely to get a loan because of the color of their skin (or anything else they cannot change). There are some attributes that we should turn a blind eye to.

nickff 11y ago
Schneier's point is not only that those attributes should be ignored, but there should be no correlation between those attributes and the end result of the algorithm. You can quite easily find examples of algorithms which do not include race or gender as an input, yet have a disparate impact. Schneier is saying that algorithms with disparate impact should be avoided without regard to procedural fairness, proportionality of outcome, or correctness.
Houshalter 11y ago
Why should fairness be a goal though? If men are really are more dangerous drivers, why shouldn't they pay more in car insurance? Or teenagers? It's not like they can control their age, but they are objectively more likely to get into an accident. So are people with eyesight problems, etc. Algorithms should make the most accurate predictions possible. If somehow we had a time machine and could just see the future, we would do that. People who are more likely to get in accidents should pay higher insurance, or bank loans, or whatever. If you are less of a risk, then you should pay less. Omitting statistically relevant features is silly. It's not like the algorithm has human biases. It doesn't hate people of a different race, it's not sexist. It just looks at the cold hard facts and makes the best prediction possible.

msellout 11y ago

>> It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. > unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors That may be true, but Schneier is speaking on a different topic -- redlining. There's a long history of people discriminating based on race, sex, class, etc. and pretending they're using an unbiased algorithm instead. The word redlining comes from the practice of using geography to make decisions. It's obvious to most people that geography is highly correlated with race and class. Therefore, even if geography itself carries no intrinsic bias, to use geography as a variable in evaluating credit-worthiness is discriminatory. https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining

david_ar 11y ago

The problem is not so much about whether predictions are biased. It's a problem of whether the algorithm is reinforcing pre-existing biases. In other words, it's about taking short-term losses in return for long-term gains. Suppose a certain group of people have experienced some kind of historical discrimination, and as a result people in that group are starting from a disadvantaged position (wealth, etc) compared to the rest of the population. It may well be an unbiased prediction that such people will be a greater credit risk, etc. However, by making decisions solely on the basis of this, although the bank saves some money in the short-term, it reinforces the cycle of discrimination. If your goal is to improve society in the long-term, making decisions based on these short-term predictors is not a desirable thing to do.