3 ms·

I disagree with this statement, > However, just from a cursory review of the mugshots, people of color were disproportionately represented in the mugshot datab

I disagree with this statement, > However, just from a cursory review of the mugshots, people of color were disproportionately represented in the mugshot database. So it’s not entirely fair to criticize the facial recognition technology for matching more people of color. No, the point is the disproportionate representation, and it is a fair criticism, because it is a fundamental limiting factor in the use of the technology. You seem to be distinguishing the "technology" from the data source, but that is not possible. The quality of the technology depends on the quality of the data provided to it. > I believe a curated list of mugshots with certain characteristics would result in a similar representation of mismatches. Nothing I have read about the technology suggests that there are inherit [sic, is inherent] bias. This bias is in the data, not the algorithm per se. This is quite accepted in many, even most, criticisms of machine learning applications in the social sphere. It comes up a lot for example in NLP models, for example with assumptions on gender for professions. It is a technology problem because it demands a technological solution, because removing any and all bias by hand-curating datasets is just not a scalable approach.

netcan 6y ago
I feel like we're missing the popular vocabulary to describe some of these recurring problems. The algorithms reflect and amplify biases in the data. They also go into feedback loops once they affect the reality being represented in data. Imagine that an algorithm selects individuals at an airport for search. Contraband smugglers are caught. The dataset now includes them, evolving the bias. We have already seen this at play on social media, recommendation engines, fraud detection, etc. The future is worrying. There are ratchets on technologies such as these. Unless there are extraordinary counter-reactions, they will be "affecting their own datasets" en masse very soon.
- curveship 6y ago
  The term I've heard for this phenomenon is "mathwashing."
  - bobthepanda 6y ago
    The old chestnut for this is "lies, damned lies, and statistics."
bko 6y ago
The problem of disproportionate representation is a problem whether you're using technology or not. It's not a new problem or specific to tech. If a large percentage of dangerous interactions a police officer has with an individual fits a certain profile (e.g. the third party was a man), the police officer will likely exhibit a different set of behaviors when dealing with that party. Sometimes it's appropriate, many times it's not. Same is true for judges and others in the system and you'll see stricter sentencing for certain groups. This results can lead to a self-reinforcing cycle of oppression. The difference is with technology these biases can at least try to be addressed directly and create an objective more fair system.
- bobthepanda 6y ago
  The major issue is that the tech is not ready. If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%. With the punitive American justice system, that is 14% more people who will be detained unjustly, and then when they leave they will be denied most opportunities for housing, jobs, and other things needed to participate in society. Technology can address biases but we need to make sure that they are addressed sufficiently, so that we do not end up causing more problems than we solve.
  - radarsat1 6y ago
    > If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%. This is a really interesting point that I'll keep in the back of my mind, thanks for that! I hadn't even thought it through in such a concrete manner but I think this really hits the nail on the head.

netcan 6y ago

I feel like we're missing the popular vocabulary to describe some of these recurring problems. The algorithms reflect and amplify biases in the data. They also go into feedback loops once they affect the reality being represented in data. Imagine that an algorithm selects individuals at an airport for search. Contraband smugglers are caught. The dataset now includes them, evolving the bias. We have already seen this at play on social media, recommendation engines, fraud detection, etc. The future is worrying. There are ratchets on technologies such as these. Unless there are extraordinary counter-reactions, they will be "affecting their own datasets" en masse very soon.

curveship 6y ago
The term I've heard for this phenomenon is "mathwashing."
- bobthepanda 6y ago
  The old chestnut for this is "lies, damned lies, and statistics."

bko 6y ago

The problem of disproportionate representation is a problem whether you're using technology or not. It's not a new problem or specific to tech. If a large percentage of dangerous interactions a police officer has with an individual fits a certain profile (e.g. the third party was a man), the police officer will likely exhibit a different set of behaviors when dealing with that party. Sometimes it's appropriate, many times it's not. Same is true for judges and others in the system and you'll see stricter sentencing for certain groups. This results can lead to a self-reinforcing cycle of oppression. The difference is with technology these biases can at least try to be addressed directly and create an objective more fair system.

bobthepanda 6y ago
The major issue is that the tech is not ready. If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%. With the punitive American justice system, that is 14% more people who will be detained unjustly, and then when they leave they will be denied most opportunities for housing, jobs, and other things needed to participate in society. Technology can address biases but we need to make sure that they are addressed sufficiently, so that we do not end up causing more problems than we solve.
- radarsat1 6y ago
  > If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%. This is a really interesting point that I'll keep in the back of my mind, thanks for that! I hadn't even thought it through in such a concrete manner but I think this really hits the nail on the head.