3 ms·

Note that there's not only a potential selection bias problem, but a feedback issue as well. For instance, if the algorithm is biased toward assigning higher cr

Note that there's not only a potential selection bias problem, but a feedback issue as well. For instance, if the algorithm is biased toward assigning higher criminal activity risk to black people, the black people will be more likely to be checked and, as a consequence, the future versions of such algorithms will be even more biased in the same direction. Debiasing in such situations is a very tough endeavour.

marcoperaza 8y ago
A lot hinges on how you define fairness. Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality? No matter how accurate your metric is—nay, because of its accuracy—people will be mad at you because it reflects an underlying reality they are in denial of or believe is itself unfair, and want to rectify by requiring fictions elsewhere. I don’t envy people working on this because you by definition can’t win. Some of the powerful political forces, on the one hand, and the demands for accuracy and effectiveness, on the other, are irreconcilable.
- yorwba 8y ago
  > Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality? The article is about the unfairness of a higher percentage of <racial group> getting incorrectly flagged as “criminal activity risks”. The fairness of the expected output is essentially out of scope for this kind of research, it's all about the distribution of correct vs. erroneous decisions. The underlying reality is an imbalance in data quality or similar, not whatever you were thinking about.
- jakelazaroff 8y ago
  That's why the grandparent mentioned feedback. Even if a disparity "aligns with reality", if the algorithm is used in a way that reinforces the disparity, then it's discriminatory. When designing algorithms as parts of systems, we need to be careful to not ossify statistics we're ostensibly just reflecting.
Symmetry 8y ago
This means it's a lot more dangerous to use machine learning for crimes like jaywalking where punishment is discretionary than crimes like murder or jumping bail where the crime will pretty much always be noted and investigated.
- Bartweiss 8y ago
  Seeing it applied to the chance of violating probation or reoffending after a minor crime alarms me for exactly this reason - lots of people commit minor probation violations (e.g. associating with criminals) or crimes (e.g. marijuana possession) and are never caught or charged. Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.

marcoperaza 8y ago

A lot hinges on how you define fairness. Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality? No matter how accurate your metric is—nay, because of its accuracy—people will be mad at you because it reflects an underlying reality they are in denial of or believe is itself unfair, and want to rectify by requiring fictions elsewhere. I don’t envy people working on this because you by definition can’t win. Some of the powerful political forces, on the one hand, and the demands for accuracy and effectiveness, on the other, are irreconcilable.

yorwba 8y ago
> Is it unfair if a higher percentage of <racial group> is flagged as “criminal activity risks”, to use your example, than of <other racial group>? What if it aligns with reality? The article is about the unfairness of a higher percentage of <racial group> getting incorrectly flagged as “criminal activity risks”. The fairness of the expected output is essentially out of scope for this kind of research, it's all about the distribution of correct vs. erroneous decisions. The underlying reality is an imbalance in data quality or similar, not whatever you were thinking about.
jakelazaroff 8y ago
That's why the grandparent mentioned feedback. Even if a disparity "aligns with reality", if the algorithm is used in a way that reinforces the disparity, then it's discriminatory. When designing algorithms as parts of systems, we need to be careful to not ossify statistics we're ostensibly just reflecting.

Symmetry 8y ago

This means it's a lot more dangerous to use machine learning for crimes like jaywalking where punishment is discretionary than crimes like murder or jumping bail where the crime will pretty much always be noted and investigated.

Bartweiss 8y ago
Seeing it applied to the chance of violating probation or reoffending after a minor crime alarms me for exactly this reason - lots of people commit minor probation violations (e.g. associating with criminals) or crimes (e.g. marijuana possession) and are never caught or charged. Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.