4 ms·

An AI classifier doesn't know, or care, why a particular population has higher crime and lower income, it only recognizes and reports on the pattern.

gbrown 7y ago
EXACTLY
- manfredo 7y ago
  But I think you're missing the point. Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
  - crooked-v 7y ago
    It depends on what you're using it for and why. If you're concretely distributing police resources, you probably want short-term prediction of actual aggregate crime rates (but also want to consider the risk of overpolicing based on crime rates only recorded higher because of historical overpolicing). On the other hand, if you want to understand anything about the populations involved on some abstract level, merely plugging in predicted crime rates based on historical data won't help at all.
    - manfredo 7y ago
      This comment chain is in reference to law enforcement and the justice system. I would hope that our law enforcement and justice systems are operating on real world data, rather than a hypothetical world's data.
      - crooked-v 7y ago
        I would absolutely not want the justice system to use historical statistical information without considering factors like lead exposure and past overpolicing, and even pure boots-on-ground distributions of law enforcement resources should take into account historical over- and under-policing and its effects on the statistics being used to distribute those resources.
        manfredo 7y ago
        Okay, that's your position. Plenty of people do not want an AI that, say, gives parole to someone that is a high risk of re-offending because someone engineered the AI to be more lenient towards people who grew up in areas with greater lead exposure.
        crooked-v 7y ago
        Consider if someone grew up in an area of high lead exposure but was, one way or another, protected from it as a child [1]. Should they be treated as having the same recidivism risk factor as someone who suffered the full effects of lead exposure? If not, how do you separate the effects of lead exposure from other factors to recidivism rates? [1] See the 1999 documentary Blast from the Past.
        manfredo 7y ago
        The better approach is to define what inputs are going to be used, explicitly. Some like race and religion are going not going to be used, as discrimination on the basis of race or religion is off limits. But age, past offences, and the like probably are But I think we're diverging considerably from the original point: that forcing an AI to produce equal outcomes despite unequal behavior in the real world is not the elimination of bias, it's the deliberate introduction of bias. If we have an AI that predicts recidivism rates, and we engineer it to produce equal predicted rates across all groups despite different between rates groups in the real world then we are deliberate introducing bias. The truth, regrettable thought may be, is that a magical AI that operates with 100% accuracy - the only people who it flags would have re-offended - is going to produce disparities because recidivism rates are not equal.
  - gbrown 7y ago
    The answer entirely depends - "Deliberately altering the model to produce equal results despite unequal patterns in reality" is so vague as to be meaningless in this discussion. > Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world? The goal is to use the results of a model for something (pet peeve, I hate the use of "AI" to describe what are usually pretty standard statistical or ML models). The model you create, and how you apply/interpret it, depend entirely on what you're actually trying to accomplish or change with the results. Depending on what that is, the kind of "bias reflection" we're discussing is hugely problematic.
    - manfredo 7y ago
      And that "something" we want the AI to do is usually grounded in the real world in some way, be it shopping patterns, or crime. And the real world has disparities. For example, crime rates are not equal between men and women. If we force our AI to assign equal risk of crime to men and women then we will have introduced a bias that either under predicts the rate of male crime, or over prdicts the rate of female crime.
      - loopz 7y ago
        "Truth" is irrelevant and academic. What matters is: What are the outcomes and consequences of active systems, AI or not. For instance: How do the algo cope with derivatives of its own output being fed into itself as input at a later stage?
        manfredo 7y ago
        If truth is irrelevant then just return random output and call it a day. The reality is, truth is relevant and sometimes the truth is inconvenient. Tech workers may want to build an AI that measures risk of recidivism that produces uniform risks across race and gender. But the truth is, rates of recidivism is not the same across all groups. If we produce the desired outcome of equal reporting of risk, then the consequence is that men have their risk underreported to put them on parity with women, or vice versa.
  - 40acres 7y ago
    Is it introducing bias or is it introducing appropriate weighting? The most crime filled neighborhood in America may not be the inner city you have pictured in your mind but Lower Manhattan.
  - vharuck 7y ago
    >Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Statisticians refer to this process as "controlling for confounding factors." What really matters is what questions you're asking. Data is too often abused, not always intentionally, by people with vague questions.
  - codesushi42 7y ago
    Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? What a terrible example. There are many other features being omitted that would predict crime rates. If anything, this is an example of enforcing your own bias on the model by not including all relevant features.
    - manfredo 7y ago
      Great then go ahead and include additional features. The problem is, recidivism rates in real life are not equal so as the model increases in accuracy it will inevitably produce disparities in its results.
      - deleted 7y ago
        [deleted]
  - tsimionescu 7y ago
    You're missing the point when asking the final question. Predicting crime rates is not the goal in itself (unless you're running some kind of crime rate bet). If you're going to use a model trained on simple, biased data, to get, say, insurance estimates for a for-profit company, the model will probably successfully increase profits, so it was a good model. On the other hand, if you're going to use the same model to help with sentencing, where your goal is to see equality and justice, then the model will do very badly, since it will punish many people for the community/skin they happened to be born in.
    - manfredo 7y ago
      The problem is, some groups do commit crimes at higher rates than others. If we are engineering the model to produce outputs with no disparities when disparities do exist in the real world, then our model is going to be biased. For instance, men commit more crimes than women. If we are building an AI that predicts risk of committing crime (say, estimating rates of recidivism) and we forcibly make it report equal rates between men and women then we will be creating a discriminatory system because it will either under report the risk of men or over report the risk of women in order to achieve parity. Engineering parity of outcome in the model when the real world outcomes have disparities necessarily results in bias.
- nightski 7y ago
  If you already know what you want the model to say ahead of time and are tweaking it to fit that narrative then there is no point in creating a model in the first place.
crooked-v 7y ago
...which is exactly the kind of "treating culturally- and situationally-contextual results like inherent facts" bias the article is talking about.
- DuskStar 7y ago
  And if those results change, so will the algorithm's outputs! But asking the algorithm to make the change seems to be a bit much. Honestly, my preferred method of solving this would be to train the algorithm on a data set with all of the forbidden values included along with anything else the creator feels relevant - zip code, income, familial status, favorite sport, education - and then when running in production, against real people, don't give it the restricted information. Yes, you could theoretically extract race, gender and other protected stats from the information the algorithm actually uses in prod - but it has no incentive to, since a less-noisy signal is already provided. For instance, suppose the optimal algorithm for your data set is some linear function of X, Y and Z - let's say X+Y+Z to keep things simple. X,Y and Z are all normally distributed variables, mean of 0 and the same standard deviation. Y has a 0.5 correlation with X, and a -0.5 correlation with Z. If not provided Y, your algorithm might come up with 1.5X+0.5Z as an approximation - extracting a bit of the signal for Y from the things it does have access to. It's suboptimal, but better than just X+Z. Unfortunately, Y is verboten - we're not allowed to discriminate on it, and this approximation ends up with results that track Y. So instead we train with X, Y and Z as inputs, so the derived model is X+Y+Z - and we can drop Y from that model in production, leading to a model that (while less accurate) shouldn't unfairly track Y.
  - int_19h 7y ago
    > And if those results change, so will the algorithm's outputs! But asking the algorithm to make the change seems to be a bit much. The problem is that the output of those algorithms is used to drive decision making that has the effect of maintaining the status quo, by removing opportunities to change it.
    - manfredo 7y ago
      Which loops back around to my original point, which is that the notion that we should alter or influence the algorithm because its output does not match our worldview or politics is not the removal of bias it is the deliberate injection of our personal biases. The whole point of using the algorithm was to make sure personal biases aren't impacting the decision. If we're going to alter the algorithm because we don't like the result, then why are we bothering to use an algorithm in the first place? Just use a human to make the decision. At least in that scenario potential biases have an identifiable source, as opposed to an opaque program that may have been made by engineers that deliberately tuned it to avoid any disparities because they think any disparity in outcome is fundamentally problematic
40acres 7y ago
This is exactly the problem. An AI classifier without sufficient context is simply a very efficient discrimination machine. We must make sure that AI systems have enough context and protection from human bias before large scale deployment.

gbrown 7y ago

EXACTLY

manfredo 7y ago
But I think you're missing the point. Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
- crooked-v 7y ago
  It depends on what you're using it for and why. If you're concretely distributing police resources, you probably want short-term prediction of actual aggregate crime rates (but also want to consider the risk of overpolicing based on crime rates only recorded higher because of historical overpolicing). On the other hand, if you want to understand anything about the populations involved on some abstract level, merely plugging in predicted crime rates based on historical data won't help at all.
  - manfredo 7y ago
    This comment chain is in reference to law enforcement and the justice system. I would hope that our law enforcement and justice systems are operating on real world data, rather than a hypothetical world's data.
    - crooked-v 7y ago
      I would absolutely not want the justice system to use historical statistical information without considering factors like lead exposure and past overpolicing, and even pure boots-on-ground distributions of law enforcement resources should take into account historical over- and under-policing and its effects on the statistics being used to distribute those resources.
      - manfredo 7y ago
        Okay, that's your position. Plenty of people do not want an AI that, say, gives parole to someone that is a high risk of re-offending because someone engineered the AI to be more lenient towards people who grew up in areas with greater lead exposure.
        crooked-v 7y ago
        Consider if someone grew up in an area of high lead exposure but was, one way or another, protected from it as a child [1]. Should they be treated as having the same recidivism risk factor as someone who suffered the full effects of lead exposure? If not, how do you separate the effects of lead exposure from other factors to recidivism rates? [1] See the 1999 documentary Blast from the Past.
        manfredo 7y ago
        The better approach is to define what inputs are going to be used, explicitly. Some like race and religion are going not going to be used, as discrimination on the basis of race or religion is off limits. But age, past offences, and the like probably are But I think we're diverging considerably from the original point: that forcing an AI to produce equal outcomes despite unequal behavior in the real world is not the elimination of bias, it's the deliberate introduction of bias. If we have an AI that predicts recidivism rates, and we engineer it to produce equal predicted rates across all groups despite different between rates groups in the real world then we are deliberate introducing bias. The truth, regrettable thought may be, is that a magical AI that operates with 100% accuracy - the only people who it flags would have re-offended - is going to produce disparities because recidivism rates are not equal.
- gbrown 7y ago
  The answer entirely depends - "Deliberately altering the model to produce equal results despite unequal patterns in reality" is so vague as to be meaningless in this discussion. > Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world? The goal is to use the results of a model for something (pet peeve, I hate the use of "AI" to describe what are usually pretty standard statistical or ML models). The model you create, and how you apply/interpret it, depend entirely on what you're actually trying to accomplish or change with the results. Depending on what that is, the kind of "bias reflection" we're discussing is hugely problematic.
  - manfredo 7y ago
    And that "something" we want the AI to do is usually grounded in the real world in some way, be it shopping patterns, or crime. And the real world has disparities. For example, crime rates are not equal between men and women. If we force our AI to assign equal risk of crime to men and women then we will have introduced a bias that either under predicts the rate of male crime, or over prdicts the rate of female crime.
    - loopz 7y ago
      "Truth" is irrelevant and academic. What matters is: What are the outcomes and consequences of active systems, AI or not. For instance: How do the algo cope with derivatives of its own output being fed into itself as input at a later stage?
      - manfredo 7y ago
        If truth is irrelevant then just return random output and call it a day. The reality is, truth is relevant and sometimes the truth is inconvenient. Tech workers may want to build an AI that measures risk of recidivism that produces uniform risks across race and gender. But the truth is, rates of recidivism is not the same across all groups. If we produce the desired outcome of equal reporting of risk, then the consequence is that men have their risk underreported to put them on parity with women, or vice versa.
- 40acres 7y ago
  Is it introducing bias or is it introducing appropriate weighting? The most crime filled neighborhood in America may not be the inner city you have pictured in your mind but Lower Manhattan.
- vharuck 7y ago
  >Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Statisticians refer to this process as "controlling for confounding factors." What really matters is what questions you're asking. Data is too often abused, not always intentionally, by people with vague questions.
- codesushi42 7y ago
  Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? What a terrible example. There are many other features being omitted that would predict crime rates. If anything, this is an example of enforcing your own bias on the model by not including all relevant features.
  - manfredo 7y ago
    Great then go ahead and include additional features. The problem is, recidivism rates in real life are not equal so as the model increases in accuracy it will inevitably produce disparities in its results.
    - deleted 7y ago
      [deleted]
- tsimionescu 7y ago
  You're missing the point when asking the final question. Predicting crime rates is not the goal in itself (unless you're running some kind of crime rate bet). If you're going to use a model trained on simple, biased data, to get, say, insurance estimates for a for-profit company, the model will probably successfully increase profits, so it was a good model. On the other hand, if you're going to use the same model to help with sentencing, where your goal is to see equality and justice, then the model will do very badly, since it will punish many people for the community/skin they happened to be born in.
  - manfredo 7y ago
    The problem is, some groups do commit crimes at higher rates than others. If we are engineering the model to produce outputs with no disparities when disparities do exist in the real world, then our model is going to be biased. For instance, men commit more crimes than women. If we are building an AI that predicts risk of committing crime (say, estimating rates of recidivism) and we forcibly make it report equal rates between men and women then we will be creating a discriminatory system because it will either under report the risk of men or over report the risk of women in order to achieve parity. Engineering parity of outcome in the model when the real world outcomes have disparities necessarily results in bias.
nightski 7y ago
If you already know what you want the model to say ahead of time and are tweaking it to fit that narrative then there is no point in creating a model in the first place.

crooked-v 7y ago

...which is exactly the kind of "treating culturally- and situationally-contextual results like inherent facts" bias the article is talking about.

DuskStar 7y ago
And if those results change, so will the algorithm's outputs! But asking the algorithm to make the change seems to be a bit much. Honestly, my preferred method of solving this would be to train the algorithm on a data set with all of the forbidden values included along with anything else the creator feels relevant - zip code, income, familial status, favorite sport, education - and then when running in production, against real people, don't give it the restricted information. Yes, you could theoretically extract race, gender and other protected stats from the information the algorithm actually uses in prod - but it has no incentive to, since a less-noisy signal is already provided. For instance, suppose the optimal algorithm for your data set is some linear function of X, Y and Z - let's say X+Y+Z to keep things simple. X,Y and Z are all normally distributed variables, mean of 0 and the same standard deviation. Y has a 0.5 correlation with X, and a -0.5 correlation with Z. If not provided Y, your algorithm might come up with 1.5X+0.5Z as an approximation - extracting a bit of the signal for Y from the things it does have access to. It's suboptimal, but better than just X+Z. Unfortunately, Y is verboten - we're not allowed to discriminate on it, and this approximation ends up with results that track Y. So instead we train with X, Y and Z as inputs, so the derived model is X+Y+Z - and we can drop Y from that model in production, leading to a model that (while less accurate) shouldn't unfairly track Y.
- int_19h 7y ago
  > And if those results change, so will the algorithm's outputs! But asking the algorithm to make the change seems to be a bit much. The problem is that the output of those algorithms is used to drive decision making that has the effect of maintaining the status quo, by removing opportunities to change it.
  - manfredo 7y ago
    Which loops back around to my original point, which is that the notion that we should alter or influence the algorithm because its output does not match our worldview or politics is not the removal of bias it is the deliberate injection of our personal biases. The whole point of using the algorithm was to make sure personal biases aren't impacting the decision. If we're going to alter the algorithm because we don't like the result, then why are we bothering to use an algorithm in the first place? Just use a human to make the decision. At least in that scenario potential biases have an identifiable source, as opposed to an opaque program that may have been made by engineers that deliberately tuned it to avoid any disparities because they think any disparity in outcome is fundamentally problematic

40acres 7y ago

This is exactly the problem. An AI classifier without sufficient context is simply a very efficient discrimination machine. We must make sure that AI systems have enough context and protection from human bias before large scale deployment.