4 ms·

In any case, you shouldn't neglect the subtle but important sources of bias those pre-crime models can have. Here's an interesting talk about it: https://www.y

In any case, you shouldn't neglect the subtle but important sources of bias those pre-crime models can have. Here's an interesting talk about it: https://www.youtube.com/watch?v=MfThopD7L1Y https://www.youtube.com/watch?v=MfThopD7L1Y Basically, one instance of bias is the fact that many crime-prediction models are trained on police data, which means they will predict crime in places more often targeted by the police anyway. Then the model predictions even amplify that effect, since more training data may be generated from the places now more often policed, etc. There's lots of resources out there on AI fairness these days. I think everyone who tries stuff like crime prediction should read up on that topic.

chronic193 9y ago
We must ensure police are continuously allocated to crime-ridden areas, with or without AI, fair or not. If you are hinting at sending fewer police to black neighborhoods, you can bet that will never happen.
- darpa_escapee 9y ago
  I lived in a town that neighbored another where police presence was about 4 fold compared to the former. You would be stopped while walking down the street in the neighboring town, meanwhile I could look a out my window and watch a few drug deals take place in an afternoon. The difference? The neighboring town had a couple decades long reputation of being a hotbed for crime. That reputation would be cemented in an AI system that was trained off of police data.
- pfarnsworth 9y ago
  You must be talking from your white privilege. It's a common known fact that crime-ridden, mostly minority neighborhoods have a complete dearth of services such as police, firefighters, and ambulance. Even back in the 80s, Public Enemy released a song called "911 is a Joke" because if you called 911 from a black neighborhood, they won't respond. It's routinely known that cops will rather stay in rich neighborhoods and let the poorer neighborhoods fester. This happened during the LA riots where Koreantown burned because most of the cops went to Beverly Hill, etc, to protect the rich people. I saw this first hand in Detroit about 8 years ago. I was driving through a bad neighborhood in Detroit with my friend, and we entered the neighborhood of Grosse Point, a rich area. Cops followed us until we left, which is their way of saying "you don't belong here". Meanwhile, among the burnt down houses and broken windows of Detroit proper, you couldn't see a single cop.
mc32 9y ago
So if the AI identifies insider trading at trading firms, banks, etc., we should beware that this would create a feedback loop to look more into the investment and banking sectors and will ignore the mom and pop insider traders? That they go where crime is rampant over where it isn't and that could be a bad thing?
- bcyn 9y ago
  I think it's not inherently bad to have a bias, but it's bad if you don't recognize it. In your example: If you train an AI on data that solely consists of trading firms & banks, you should recognize that it's an AI biased towards detecting activity at trading firms & banks, and that it might be lacking in other areas. It becomes dangerous when such an AI is marketed and assumed as unbiased and the source of truth for detecting all insider trading activity.
  - mc32 9y ago
    If we don't have infinite resources (prosecutorial, for example), it makes sense to concentrate on the most salient loci of crime. If after enough resources are devoted to the most egregious locus, a different locus becomes the focus of crime and in turn gets the attention, that's not a bad thing... In other words if done decently well, they'd obtain data from all places but focus on the problematic areas till they reach equilibrium and then you redirect to the next hotspot, no?
    - evgen 9y ago
      Is it a loci because that is where the crime happens, or is it a loci because that is where all of the crime prevention/detection/prosecution is directed due to external circumstances? You have to be able to correct for this sort of bias in your training data or else you are just baking the external circumstances into the model and pretending that they reflect reality. Start by figuring out how you would obtain data from all places first.
- otakucode 9y ago
  Mom and pop insider trading? How would that work, exactly? Or was it intended as a joke?
murtali 9y ago
Cathy O'Neil wrote a book called "Weapons of Math Destruction" -- interesting read. You can listen to an interview she does on econtalk -- interesting to learn more about the hidden biases. http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.html http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...
- harrumph 9y ago
  >Cathy O'Neil wrote a book called "Weapons of Math Destruction" -- interesting read +1000. That book should be required reading for anyone working in machine learning. Written by a former Wall Street quant who has the math down cold. What she knows about rampant bias in allegedly politically agnostic machine learning circles is that the formulation and production of answers is trivial when compared to the formulation and production of questions. Super-relevant to this thread is her work on recidivism risk scoring algos run on prisoners and defendants. The feedback loops that these algos spur are seriously damaging the lives of huge numbers of persons in the criminal justice system far beyond proportionality for the offenses that brought them there.

chronic193 9y ago

We must ensure police are continuously allocated to crime-ridden areas, with or without AI, fair or not. If you are hinting at sending fewer police to black neighborhoods, you can bet that will never happen.

darpa_escapee 9y ago
I lived in a town that neighbored another where police presence was about 4 fold compared to the former. You would be stopped while walking down the street in the neighboring town, meanwhile I could look a out my window and watch a few drug deals take place in an afternoon. The difference? The neighboring town had a couple decades long reputation of being a hotbed for crime. That reputation would be cemented in an AI system that was trained off of police data.
pfarnsworth 9y ago
You must be talking from your white privilege. It's a common known fact that crime-ridden, mostly minority neighborhoods have a complete dearth of services such as police, firefighters, and ambulance. Even back in the 80s, Public Enemy released a song called "911 is a Joke" because if you called 911 from a black neighborhood, they won't respond. It's routinely known that cops will rather stay in rich neighborhoods and let the poorer neighborhoods fester. This happened during the LA riots where Koreantown burned because most of the cops went to Beverly Hill, etc, to protect the rich people. I saw this first hand in Detroit about 8 years ago. I was driving through a bad neighborhood in Detroit with my friend, and we entered the neighborhood of Grosse Point, a rich area. Cops followed us until we left, which is their way of saying "you don't belong here". Meanwhile, among the burnt down houses and broken windows of Detroit proper, you couldn't see a single cop.

mc32 9y ago

So if the AI identifies insider trading at trading firms, banks, etc., we should beware that this would create a feedback loop to look more into the investment and banking sectors and will ignore the mom and pop insider traders? That they go where crime is rampant over where it isn't and that could be a bad thing?

bcyn 9y ago
I think it's not inherently bad to have a bias, but it's bad if you don't recognize it. In your example: If you train an AI on data that solely consists of trading firms & banks, you should recognize that it's an AI biased towards detecting activity at trading firms & banks, and that it might be lacking in other areas. It becomes dangerous when such an AI is marketed and assumed as unbiased and the source of truth for detecting all insider trading activity.
- mc32 9y ago
  If we don't have infinite resources (prosecutorial, for example), it makes sense to concentrate on the most salient loci of crime. If after enough resources are devoted to the most egregious locus, a different locus becomes the focus of crime and in turn gets the attention, that's not a bad thing... In other words if done decently well, they'd obtain data from all places but focus on the problematic areas till they reach equilibrium and then you redirect to the next hotspot, no?
  - evgen 9y ago
    Is it a loci because that is where the crime happens, or is it a loci because that is where all of the crime prevention/detection/prosecution is directed due to external circumstances? You have to be able to correct for this sort of bias in your training data or else you are just baking the external circumstances into the model and pretending that they reflect reality. Start by figuring out how you would obtain data from all places first.
otakucode 9y ago
Mom and pop insider trading? How would that work, exactly? Or was it intended as a joke?

murtali 9y ago

Cathy O'Neil wrote a book called "Weapons of Math Destruction" -- interesting read. You can listen to an interview she does on econtalk -- interesting to learn more about the hidden biases. http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.html http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

harrumph 9y ago
>Cathy O'Neil wrote a book called "Weapons of Math Destruction" -- interesting read +1000. That book should be required reading for anyone working in machine learning. Written by a former Wall Street quant who has the math down cold. What she knows about rampant bias in allegedly politically agnostic machine learning circles is that the formulation and production of answers is trivial when compared to the formulation and production of questions. Super-relevant to this thread is her work on recidivism risk scoring algos run on prisoners and defendants. The feedback loops that these algos spur are seriously damaging the lives of huge numbers of persons in the criminal justice system far beyond proportionality for the offenses that brought them there.