10 ms·

I think you've very severely missed the point. Differences like those you describe generally reflect biases in our socal structure, which are then (correctly, b

by gbrown 7y ago

I think you've very severely missed the point. Differences like those you describe generally reflect biases in our socal structure, which are then (correctly, but problematically) reflected in statistical models and algorithms of all stripes. This becomes more problematic when we overinterpret the resulting findings, or uncritically incorporate the resulting predictions into real systems. A good example is the criminal justice system in the US. Minority communities are much more intensively policed, prosecuted, and jailed. A model will, without care, use race or poxies thereof to predict "criminality" without understanding or accounting for the bias inherent in the label. If that is used in policy, it runs the risk of amplifying existing social problems and injustices. We alway have to ask: "bias for what?", or the conversation will be hopelessly confused.

nostromo 7y ago
It's unpopular to talk about, but there is quantifiably more crime in some minority communities in the US. There are some crimes, like drug use, that are indeed over-reported in some communities due to over-policing. But there are other crimes, like homicide, that are nearly universally reported, regardless of where they are committed. And those crimes are much more frequent in minority-majority parts of the country. Turning a blind eye to this problem is a disservice to those communities, because they are the same communities that are most commonly the victims of crimes.
- crooked-v 7y ago
  > there is quantifiably more crime in some minority communities Sure... and what about the chronic lead exposure issues in many of these areas, the effects of long-term overpolicing of minor crimes on family stability, and the impact of (lack of) generational wealth and education from being used as de facto or literal slaves for generations? The 'minority' part here is a correlative factor, not a causative one.
  - nostromo 7y ago
    An AI classifier doesn't know, or care, why a particular population has higher crime and lower income, it only recognizes and reports on the pattern.
    - gbrown 7y ago
      EXACTLY
      - manfredo 7y ago
        But I think you're missing the point. Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
        crooked-v 7y ago
        It depends on what you're using it for and why. If you're concretely distributing police resources, you probably want short-term prediction of actual aggregate crime rates (but also want to consider the risk of overpolicing based on crime rates only recorded higher because of historical overpolicing). On the other hand, if you want to understand anything about the populations involved on some abstract level, merely plugging in predicted crime rates based on historical data won't help at all.
        manfredo 7y ago
        This comment chain is in reference to law enforcement and the justice system. I would hope that our law enforcement and justice systems are operating on real world data, rather than a hypothetical world's data.
        crooked-v 7y ago
        I would absolutely not want the justice system to use historical statistical information without considering factors like lead exposure and past overpolicing, and even pure boots-on-ground distributions of law enforcement resources should take into account historical over- and under-policing and its effects on the statistics being used to distribute those resources.
- gbrown 7y ago
  You are correct about the distribution of certain crimes, but you've entirely missed the point.* Problems like violence have multiple causes, but are widely understood to be linked to problems of poverty, inequality, and marginalization. Violence is also self-perpetuating through social networks. When an incautious user of statistical tools fails to investigate the causal story behind such findings, they're going to get the wrong answer. When a policymaker acts on incomplete or misleading findings, they're going to make the problem worse. *Unless you're arguing that it's the fact of being non-white which makes people more violent, in which case I have nothing to say to you.
  - kempbellt 7y ago
    > Unless you're arguing that it's the fact of being non-white which makes people more violent This is not how I interpreted OPs comment at all. I just read it as a note that data shows correlation between minority communities and crime. Minority is a flexible, relative term, to my understanding - which, notably in recent years is frequently attributed to "non-white", but over time has been attributed to many different groups of people (caucasian as well). Interpreting it as meaning "non-white" in this context is yet another good example of our predisposed biases and relativistic understandings at play, I suppose. Not to harp on you - just noting the differences in our perceptions.
    - gbrown 7y ago
      I totally understand, and maybe I shouldn't have included that, but I do see phrases like "It's unpopular to talk about" get used as dog whistles all the time in other forums (and in real life). There's a whole subculture dedicated to dressing up racism in pseudo-scientific language (race realism etc.) At the end of the day, there's enough real, vile bigotry out there in our society, I think it's important to be extra clear when discussing topics like this.
      - kempbellt 7y ago
        I'm glad you did. It's a great example of the problem we are discussing. I agree, there is are many negative biases in the real world. More so, I'd argue, on the internet, where people feel safer at a distance to state disagreeable opinions. It's worth noting, especially when we are discussing the nature of how an AI learns.
- dmwallin 7y ago
  The problem is that these systems are fundamentally unable to distinguish causation from correlation. Which is admittedly a hard prob even for humans, but at least we have some capacity to tease these out. In this case, the increased crime in these communities are not caused by their minority status, but rather from a multitude of other factors with historical and societal origins.
  - whatshisface 7y ago
    Cynically, does a bank care why zipcode is correlated with failure to repay loans? Forcing the bank to act as if the probability distributions are different from how they really are sounds like a very awkward way to redistribute wealth. Maybe the right answer would be reparations for whatever society did to everyone in those communities and then total freedom of association for banks and others to give loans as they see fit. Another option would be to give a bad credit subsidy to people in the cases where their low estimated trustworthiness is judged to be due to someone else's error. For example if a factory gives me lead poisoning, my interest rates go up because people with high blood lead content are less likely to keep good credit, but the factory has to pay the difference because they were liable. This reminds me of the argument for the UBI. Instead of having thousands of tiny chairty programs sprinkled all over society, why not let the economy make itself as efficient as possible and then hand out the charity in units of dollars?
    - empath75 7y ago
      Redlining: https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining
      - deleted 7y ago
        [deleted]
  - hr5eaqhera 7y ago
    > unable to distinguish causation from correlation More specifically, the current breed of machine learning is correlation engine, the only strength these networks have is finding correlations in the absence of context or explanation.
- empath75 7y ago
  There is more crime in poor communities everywhere, and minority communities are more likely to be poor. Moreover, the sorts of crimes that poor people commit are more likely to be prosecuted, and minorities are more likely to be prosecuted for them.
manfredo 7y ago
Whether or not the differences in society are the product of bias is it's own can of worms. But what you're writing is demonstrative of what I'm saying. People, unlike machines, tend to regard differences between groups with suspicion and are biased against acknowledging these differences. Machines, on the other hand, do not harbor these biases and build a model that reflects the reality we live in. As I wrote in my original comment, there are potential justifications as to why we might want to introduce bias in our machine learning models to prevent the from identifying things we don't want. But the point remains: building a model that, for example, associates technology with men and women at equal rates despite the disparity that exists in the reality is not the elimination of bias. It's the deliberate introduction of bias to make the model produce a result that is in line with what we consider a more ideal worldview.
- crooked-v 7y ago
  > building a model that, for example, associates technology with men and women at equal rates despite the disparity that exists in the reality is not the elimination of bias You're assuming that that disparity is not itself a result of bias in any way.
  - manfredo 7y ago
    Maybe it is, maybe it isn't. But this is not relevant to what I'm saying. The goal of the AI is not to associate words in a hypothetical world where there is no discrimination or bias. The goal is to build a model that accurately reflects the real world. Again, there may be justifications to deliberately introduce bias in the model to produce higher rates of association between women and technology than would normally occur if the creators of the AI so desire. But this is not elimination of bias, this is deliberate introduction of bias to make th model match the creators' worldviews.
    - crooked-v 7y ago
      AI does not, in and of itself, have a goal. AI exists to be used for some human's goal. If the human's goal is to understand the level of association between women and technology without confounding factors like decades of marketing video games and computers to boys more than girls, the historical bias of such factors in the data fed to the AI becomes extremely important.
xamuel 7y ago
Hypothetical question. Suppose no-one ever noticed the lead in the Flint water supply. People put test scores into an AI and the AI emotionlessly concludes that people from Flint perform worse than average. Should you hack the AI to force it to stop saying something like that, because that's obviously racist? Or should you begin to wonder why people from Flint perform worse, and what is causing that (the undiagnosed lead in the water), and ultimately how to correct the problem (fix the water)?
- knzhou 7y ago
  Incidentally, some actual number crunching [0] shows that the lead levels in Flint were never that high, being less than the average level in many states, and like everywhere else in America had been declining significantly over decades. What's more, in the era of leaded gasoline, almost all children had blood lead levels about 10x what was seen in Flint. 0: https://www.nytimes.com/2018/07/22/opinion/flint-lead-poisoning-water.html https://www.nytimes.com/2018/07/22/opinion/flint-lead-poison...
- gbrown 7y ago
  As always, it depends entirely on what you're doing with the (predictions/estimates/inferences). This is an entirely false dilemma.
  - xamuel 7y ago
    When would you want to hack the AI? Not a rhetorical question. Reading through the comment thread I'm slowly shifting more towards being on the fence, as opposed to having a "don't hack the AI" position. But rather than hacking the AI, why not just get rid of the AI? What is the point of the AI in the first place if you're going to hack it to get the results you want anyway?
    - gbrown 7y ago
      > When would you want to hack the AI? This is not a meaningful phrase - it adds literally nothing to the conversation but confusion. To bend over backwards to give you a reasonable answer: when you're interested in conditional effects. Let's say you're interested in the risk of cancer associated with alcohol consumption. People who drink some are often found to have lower cancer rates than people who don't drink at all. Reasonable models adjust for wealth/income - estimating the risk of cancer association with alcohol CONDITIONING ON wealth/income changes the picture; the positive association is clear. Adjusting for confounding is "removing bias", changing the effect estimates. In a predictive context, using ensembles, NNs etc, the problems don't go away, they're just sometimes harder to detect (and they're dressed up in sexy marketing-speak like "AI"). Repeat after me three times: "AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle"

by gbrown 7y ago

nostromo 7y ago

It's unpopular to talk about, but there is quantifiably more crime in some minority communities in the US. There are some crimes, like drug use, that are indeed over-reported in some communities due to over-policing. But there are other crimes, like homicide, that are nearly universally reported, regardless of where they are committed. And those crimes are much more frequent in minority-majority parts of the country. Turning a blind eye to this problem is a disservice to those communities, because they are the same communities that are most commonly the victims of crimes.

crooked-v 7y ago
> there is quantifiably more crime in some minority communities Sure... and what about the chronic lead exposure issues in many of these areas, the effects of long-term overpolicing of minor crimes on family stability, and the impact of (lack of) generational wealth and education from being used as de facto or literal slaves for generations? The 'minority' part here is a correlative factor, not a causative one.
- nostromo 7y ago
  An AI classifier doesn't know, or care, why a particular population has higher crime and lower income, it only recognizes and reports on the pattern.
  - gbrown 7y ago
    EXACTLY
    - manfredo 7y ago
      But I think you're missing the point. Deliberately altering the model to produce equal results despite unequal patterns in reality is not the elimination of bias, it's introducing bias. Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
      - crooked-v 7y ago
        It depends on what you're using it for and why. If you're concretely distributing police resources, you probably want short-term prediction of actual aggregate crime rates (but also want to consider the risk of overpolicing based on crime rates only recorded higher because of historical overpolicing). On the other hand, if you want to understand anything about the populations involved on some abstract level, merely plugging in predicted crime rates based on historical data won't help at all.
        manfredo 7y ago
        This comment chain is in reference to law enforcement and the justice system. I would hope that our law enforcement and justice systems are operating on real world data, rather than a hypothetical world's data.
        crooked-v 7y ago
        I would absolutely not want the justice system to use historical statistical information without considering factors like lead exposure and past overpolicing, and even pure boots-on-ground distributions of law enforcement resources should take into account historical over- and under-policing and its effects on the statistics being used to distribute those resources.
gbrown 7y ago
You are correct about the distribution of certain crimes, but you've entirely missed the point.* Problems like violence have multiple causes, but are widely understood to be linked to problems of poverty, inequality, and marginalization. Violence is also self-perpetuating through social networks. When an incautious user of statistical tools fails to investigate the causal story behind such findings, they're going to get the wrong answer. When a policymaker acts on incomplete or misleading findings, they're going to make the problem worse. *Unless you're arguing that it's the fact of being non-white which makes people more violent, in which case I have nothing to say to you.
- kempbellt 7y ago
  > Unless you're arguing that it's the fact of being non-white which makes people more violent This is not how I interpreted OPs comment at all. I just read it as a note that data shows correlation between minority communities and crime. Minority is a flexible, relative term, to my understanding - which, notably in recent years is frequently attributed to "non-white", but over time has been attributed to many different groups of people (caucasian as well). Interpreting it as meaning "non-white" in this context is yet another good example of our predisposed biases and relativistic understandings at play, I suppose. Not to harp on you - just noting the differences in our perceptions.
  - gbrown 7y ago
    I totally understand, and maybe I shouldn't have included that, but I do see phrases like "It's unpopular to talk about" get used as dog whistles all the time in other forums (and in real life). There's a whole subculture dedicated to dressing up racism in pseudo-scientific language (race realism etc.) At the end of the day, there's enough real, vile bigotry out there in our society, I think it's important to be extra clear when discussing topics like this.
    - kempbellt 7y ago
      I'm glad you did. It's a great example of the problem we are discussing. I agree, there is are many negative biases in the real world. More so, I'd argue, on the internet, where people feel safer at a distance to state disagreeable opinions. It's worth noting, especially when we are discussing the nature of how an AI learns.
dmwallin 7y ago
The problem is that these systems are fundamentally unable to distinguish causation from correlation. Which is admittedly a hard prob even for humans, but at least we have some capacity to tease these out. In this case, the increased crime in these communities are not caused by their minority status, but rather from a multitude of other factors with historical and societal origins.
- whatshisface 7y ago
  Cynically, does a bank care why zipcode is correlated with failure to repay loans? Forcing the bank to act as if the probability distributions are different from how they really are sounds like a very awkward way to redistribute wealth. Maybe the right answer would be reparations for whatever society did to everyone in those communities and then total freedom of association for banks and others to give loans as they see fit. Another option would be to give a bad credit subsidy to people in the cases where their low estimated trustworthiness is judged to be due to someone else's error. For example if a factory gives me lead poisoning, my interest rates go up because people with high blood lead content are less likely to keep good credit, but the factory has to pay the difference because they were liable. This reminds me of the argument for the UBI. Instead of having thousands of tiny chairty programs sprinkled all over society, why not let the economy make itself as efficient as possible and then hand out the charity in units of dollars?
  - empath75 7y ago
    Redlining: https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining
    - deleted 7y ago
      [deleted]
- hr5eaqhera 7y ago
  > unable to distinguish causation from correlation More specifically, the current breed of machine learning is correlation engine, the only strength these networks have is finding correlations in the absence of context or explanation.
empath75 7y ago
There is more crime in poor communities everywhere, and minority communities are more likely to be poor. Moreover, the sorts of crimes that poor people commit are more likely to be prosecuted, and minorities are more likely to be prosecuted for them.

manfredo 7y ago

Whether or not the differences in society are the product of bias is it's own can of worms. But what you're writing is demonstrative of what I'm saying. People, unlike machines, tend to regard differences between groups with suspicion and are biased against acknowledging these differences. Machines, on the other hand, do not harbor these biases and build a model that reflects the reality we live in. As I wrote in my original comment, there are potential justifications as to why we might want to introduce bias in our machine learning models to prevent the from identifying things we don't want. But the point remains: building a model that, for example, associates technology with men and women at equal rates despite the disparity that exists in the reality is not the elimination of bias. It's the deliberate introduction of bias to make the model produce a result that is in line with what we consider a more ideal worldview.

crooked-v 7y ago
> building a model that, for example, associates technology with men and women at equal rates despite the disparity that exists in the reality is not the elimination of bias You're assuming that that disparity is not itself a result of bias in any way.
- manfredo 7y ago
  Maybe it is, maybe it isn't. But this is not relevant to what I'm saying. The goal of the AI is not to associate words in a hypothetical world where there is no discrimination or bias. The goal is to build a model that accurately reflects the real world. Again, there may be justifications to deliberately introduce bias in the model to produce higher rates of association between women and technology than would normally occur if the creators of the AI so desire. But this is not elimination of bias, this is deliberate introduction of bias to make th model match the creators' worldviews.
  - crooked-v 7y ago
    AI does not, in and of itself, have a goal. AI exists to be used for some human's goal. If the human's goal is to understand the level of association between women and technology without confounding factors like decades of marketing video games and computers to boys more than girls, the historical bias of such factors in the data fed to the AI becomes extremely important.

xamuel 7y ago

Hypothetical question. Suppose no-one ever noticed the lead in the Flint water supply. People put test scores into an AI and the AI emotionlessly concludes that people from Flint perform worse than average. Should you hack the AI to force it to stop saying something like that, because that's obviously racist? Or should you begin to wonder why people from Flint perform worse, and what is causing that (the undiagnosed lead in the water), and ultimately how to correct the problem (fix the water)?

knzhou 7y ago
Incidentally, some actual number crunching [0] shows that the lead levels in Flint were never that high, being less than the average level in many states, and like everywhere else in America had been declining significantly over decades. What's more, in the era of leaded gasoline, almost all children had blood lead levels about 10x what was seen in Flint. 0: https://www.nytimes.com/2018/07/22/opinion/flint-lead-poisoning-water.html https://www.nytimes.com/2018/07/22/opinion/flint-lead-poison...
gbrown 7y ago
As always, it depends entirely on what you're doing with the (predictions/estimates/inferences). This is an entirely false dilemma.
- xamuel 7y ago
  When would you want to hack the AI? Not a rhetorical question. Reading through the comment thread I'm slowly shifting more towards being on the fence, as opposed to having a "don't hack the AI" position. But rather than hacking the AI, why not just get rid of the AI? What is the point of the AI in the first place if you're going to hack it to get the results you want anyway?
  - gbrown 7y ago
    > When would you want to hack the AI? This is not a meaningful phrase - it adds literally nothing to the conversation but confusion. To bend over backwards to give you a reasonable answer: when you're interested in conditional effects. Let's say you're interested in the risk of cancer associated with alcohol consumption. People who drink some are often found to have lower cancer rates than people who don't drink at all. Reasonable models adjust for wealth/income - estimating the risk of cancer association with alcohol CONDITIONING ON wealth/income changes the picture; the positive association is clear. Adjusting for confounding is "removing bias", changing the effect estimates. In a predictive context, using ensembles, NNs etc, the problems don't go away, they're just sometimes harder to detect (and they're dressed up in sexy marketing-speak like "AI"). Repeat after me three times: "AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle"