4 ms·

I think it's a much more subtle point than that, and he does make it rather directly: * If police departments are racially biased * and the Heat List algorithm

I think it's a much more subtle point than that, and he does make it rather directly: * If police departments are racially biased * and the Heat List algorithm heavily factors in associations * and most people associate with others from their own race * then won't the Heat List disproportionately output people from a certain race? * Won't this then result in increased policing & suspicion of these communities? In other words, the point is: human biases can be a seed issue that machine learning then positive-feedback-loops out of control

sdenton4 11y ago
It depends a lot on the reporting rate for the kind of crime being monitored; things like drug use - where enforcement is almost entirely dependent on police action - will absolutely be subject to this kind of positive bias. On the other hand, violent crimes, like shootings, are events that are (usually) observed regardless of police involvement. There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing-your-mind?act=2#play http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.
blfr 11y ago
Won't this then result in increased policing & suspicion of these communities? Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?
sdenton4 11y ago
I also felt the article could have tried to get to the point more directly. My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race. Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.
yummyfajitas 11y ago
then won't the Heat List disproportionately output people from a certain race? This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step. Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).
- iofj 11y ago
  Sadly, different races commit crimes at a ... different rate. I find this to be pretty logical to be honest. e.g. For instance, certain physical properties (running fast comes to mind) , are correlated with crime. These physical properties aren't distributed randomly across races.
- 21echoes 11y ago
  I think you're missing the first point in both my summary and in the article: It does not need to be the case that "a certain race commits more crimes". It can, instead, just be the case that a certain race is arrested for committing more crimes, despite the equal rates across races of the actual criminal behavior. For instance: it's a well recognized fact[1] that blacks and whites use and deal marijuana at the same rate, but blacks are arrested for it in far larger volume. So, if this data and other similar data sets are the seed in a machine learning algorithm, then algorithms like the Heat List will output racially biased data. [1] https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel2.pdf https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel...
  - yummyfajitas 11y ago
    I didn't miss that. As I said: Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. There is nothing fundamental about machine learning that says seed data like this will give biased outputs - many algorithms do have this problem (it's a difficult one to deal with), but it's not fundamental. I certainly didn't get the impression from the article that it was advocating for algorithms which are less sensitive to these errors. Among other things, that's far less of a conversation that "we have to talk about", but far more of a conversation that some stats geeks have to talk about. These are also far less of an "ethical" problem (as the article asserts) and far more of a technical one. But maybe I misread.

sdenton4 11y ago

It depends a lot on the reporting rate for the kind of crime being monitored; things like drug use - where enforcement is almost entirely dependent on police action - will absolutely be subject to this kind of positive bias. On the other hand, violent crimes, like shootings, are events that are (usually) observed regardless of police involvement. There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing-your-mind?act=2#play http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.

blfr 11y ago

Won't this then result in increased policing & suspicion of these communities? Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?

sdenton4 11y ago

I also felt the article could have tried to get to the point more directly. My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race. Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.

yummyfajitas 11y ago

then won't the Heat List disproportionately output people from a certain race? This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step. Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).

iofj 11y ago
Sadly, different races commit crimes at a ... different rate. I find this to be pretty logical to be honest. e.g. For instance, certain physical properties (running fast comes to mind) , are correlated with crime. These physical properties aren't distributed randomly across races.
21echoes 11y ago
I think you're missing the first point in both my summary and in the article: It does not need to be the case that "a certain race commits more crimes". It can, instead, just be the case that a certain race is arrested for committing more crimes, despite the equal rates across races of the actual criminal behavior. For instance: it's a well recognized fact[1] that blacks and whites use and deal marijuana at the same rate, but blacks are arrested for it in far larger volume. So, if this data and other similar data sets are the seed in a machine learning algorithm, then algorithms like the Heat List will output racially biased data. [1] https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel2.pdf https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel...
- yummyfajitas 11y ago
  I didn't miss that. As I said: Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. There is nothing fundamental about machine learning that says seed data like this will give biased outputs - many algorithms do have this problem (it's a difficult one to deal with), but it's not fundamental. I certainly didn't get the impression from the article that it was advocating for algorithms which are less sensitive to these errors. Among other things, that's far less of a conversation that "we have to talk about", but far more of a conversation that some stats geeks have to talk about. These are also far less of an "ethical" problem (as the article asserts) and far more of a technical one. But maybe I misread.