3 ms·

Let me rephrase the questions that the reporter raises, but in a manner that's too direct for any journalist. What if machine learning systems come up with res

Let me rephrase the questions that the reporter raises, but in a manner that's too direct for any journalist. What if machine learning systems come up with results that agree with "evil" stereotypes and "biases"? What if machine learning systems discover that socially unpleasant stereotypes are actually accurate predictors of reality? Modern moral philosophy has taken an easy copout in the past. It asserts, without proof, that various positive claims are false, and therefore will yield bad decisions if used. We also believe it would be evil to use them to make decisions. The question we need to address is what moral claims can we make which are independent of positive claims?

Frenchgeek 11y ago
What if the data used to teach the machine is severely biased?
lqdc13 11y ago
I think using machine learning for this purpose is equivalent to using stereotypes or intuition to make such decisions. The reason people don't want to use stereotypes is that we might end up in a positive feedback loop when stereotyped people get arrested for the things they were expected to do, not because stereotypes aren't effective. If it is illegal to do stop and frisk, we should not be doing machine learning where features come from intuition in the first place.
21echoes 11y ago
I think it's a much more subtle point than that, and he does make it rather directly: * If police departments are racially biased * and the Heat List algorithm heavily factors in associations * and most people associate with others from their own race * then won't the Heat List disproportionately output people from a certain race? * Won't this then result in increased policing & suspicion of these communities? In other words, the point is: human biases can be a seed issue that machine learning then positive-feedback-loops out of control
- sdenton4 11y ago
  It depends a lot on the reporting rate for the kind of crime being monitored; things like drug use - where enforcement is almost entirely dependent on police action - will absolutely be subject to this kind of positive bias. On the other hand, violent crimes, like shootings, are events that are (usually) observed regardless of police involvement. There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing-your-mind?act=2#play http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.
- blfr 11y ago
  Won't this then result in increased policing & suspicion of these communities? Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?
- sdenton4 11y ago
  I also felt the article could have tried to get to the point more directly. My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race. Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.
- yummyfajitas 11y ago
  then won't the Heat List disproportionately output people from a certain race? This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step. Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).
j2kun 11y ago
> What if machine learning systems come up with results that agree with "evil" stereotypes and "biases"? This is already the case, and shouldn't be surprising. That is why the legal system can veto popular opinion, to protect minorities from the prejudice of the majority. The problem is that many people still don't realize that algorithms reinforce popular prejudice and institutionalized bias against the uneducated and the poor.
- yummyfajitas 11y ago
  If computer systems are getting the wrong conclusion, that's a boring philosophical question. Find better SVM params/neural network architecture/etc and you'll fix it. But what if the only way to eliminate "institutionalized bias" from the system is to add statistical bias? What happens if "popular prejudice" isn't actually prejudice but merely a useful prior, and one which non-racist [1] machine learning systems faithfully reproduce? [1] I'm using "racist" in the classical sense of f(x.copy(race=A)) != f.copy(x.copy(race=B)), not in the modern sense of "any probabilistic cause for unequal statistical outcomes".

Frenchgeek 11y ago

What if the data used to teach the machine is severely biased?

lqdc13 11y ago

I think using machine learning for this purpose is equivalent to using stereotypes or intuition to make such decisions. The reason people don't want to use stereotypes is that we might end up in a positive feedback loop when stereotyped people get arrested for the things they were expected to do, not because stereotypes aren't effective. If it is illegal to do stop and frisk, we should not be doing machine learning where features come from intuition in the first place.

21echoes 11y ago

I think it's a much more subtle point than that, and he does make it rather directly: * If police departments are racially biased * and the Heat List algorithm heavily factors in associations * and most people associate with others from their own race * then won't the Heat List disproportionately output people from a certain race? * Won't this then result in increased policing & suspicion of these communities? In other words, the point is: human biases can be a seed issue that machine learning then positive-feedback-loops out of control

sdenton4 11y ago
It depends a lot on the reporting rate for the kind of crime being monitored; things like drug use - where enforcement is almost entirely dependent on police action - will absolutely be subject to this kind of positive bias. On the other hand, violent crimes, like shootings, are events that are (usually) observed regardless of police involvement. There's been some pretty interesting work on this in Richmond, CA, up north of Berkeley: http://www.thisamericanlife.org/radio-archives/episode/555/the-incredible-rarity-of-changing-your-mind?act=2#play http://www.thisamericanlife.org/radio-archives/episode/555/t... Basically, it was noticed that a small cluster of people were involved in most of the violent crime, and started interventions to make that cluster less likely to engage in violent crime.
blfr 11y ago
Won't this then result in increased policing & suspicion of these communities? Increased policing should lead to lower crime levels in these areas and communities reversing the association. Unless you don't believe that policing works but then why bother with heat lists at all, no matter how accurate?
sdenton4 11y ago
I also felt the article could have tried to get to the point more directly. My initial reading of the article was that they were hand-wringing about whether the feature-set predicting crime could double as a feature-set for predicting race, and whether we should discount the algorithms as a result. To which my reaction was 'meh.' Just don't explicitly use race as a feature, and make sure your feature set will find suspicious people regardless of their race. Feedback loops, on the other hand, are a completely legitimate concern, and something that should be asked about and controlled for in these kinds of analyses.
yummyfajitas 11y ago
then won't the Heat List disproportionately output people from a certain race? This will only happen if that certain race commits more crimes (in the training data). If you take race out of a statistical predictor designed to learn crime, but race is a good predictor of crime, then the predictor might learn race at an intermediate step. Now there are statistical issues one might run into - e.g., early overfitting of what is essentially a bandit algorithm, and unaccounted for feedback between training data and system outputs. But at least the way I'm reading the article, it isn't calling for more and better math (which would be the solution to the problems you describe).

j2kun 11y ago

> What if machine learning systems come up with results that agree with "evil" stereotypes and "biases"? This is already the case, and shouldn't be surprising. That is why the legal system can veto popular opinion, to protect minorities from the prejudice of the majority. The problem is that many people still don't realize that algorithms reinforce popular prejudice and institutionalized bias against the uneducated and the poor.

yummyfajitas 11y ago
If computer systems are getting the wrong conclusion, that's a boring philosophical question. Find better SVM params/neural network architecture/etc and you'll fix it. But what if the only way to eliminate "institutionalized bias" from the system is to add statistical bias? What happens if "popular prejudice" isn't actually prejudice but merely a useful prior, and one which non-racist [1] machine learning systems faithfully reproduce? [1] I'm using "racist" in the classical sense of f(x.copy(race=A)) != f.copy(x.copy(race=B)), not in the modern sense of "any probabilistic cause for unequal statistical outcomes".