3 ms·
They fed it a large number of mugshots. They didn't ask "is this person a criminal". They asked it "does this face match one of the faces you've seen before".
by Kalium 8y ago
They fed it a large number of mugshots. They didn't ask "is this person a criminal". They asked it "does this face match one of the faces you've seen before".
You're right that systems have previously learned to infer black -> criminal. It's absolutley possible that could have happened here! Given that not every one flagged is a person of color, there's clearly more at work.
Systems like Rekognition let a user specify what level of certainty the system has that it has found a match. In this instance, the system was told to use 80%. Rekognition can be told to use any number from 0 to 100. Your number of 99.999% could be used without any difficulty whatsoever! Since the ACLU doesn't seem to be sharing enough to reproduce the test, we can't tell what would happen.
Regulation based on accuracy is a wonderful idea! One well worth pursuing. With that said, "accuracy" is perhaps difficult to define here. How would you determine that? How accuracy is measured is highly sensitive to what you use as a dataset, and different test datasets can easily produce very different impressions of how common false positives and false negatives are.
- StanislavPetrov 8y agoThey didn't tell it to use 80%, 80% is the default setting. Anyone who thinks that many law enforcement agencies aren't going to use the default setting ought to go look at their routers and other hardware and see how they are configured.
- Kalium 8y agoI understand where you're coming from. They didn't choose that number! It was chosen for them by Amazon! It's on Amazon for not actively forcing users to consider what level of confidence they want to use. Amazon can, should, and I think is perhaps even morally obligated to make customers actively choose a confidence level. Yet, might there not be other factors to consider? Bear in mind that this isn't pushbutton easy off-the-shelf. Some actual work is required here. There's probably a batch of Python scripts sitting around that did the work. The most charitable assumption is that whoever did this at the ACLU didn't look at how the tool under examination should be used at all and missed everything about confidence levels in whatever few docs they did read. Under the ACM code of ethics, I am comfortable calling that an unethical degree of incompetence and carelessness. A less charitable interpretation is that they knew and didn't care because it produced the result they wanted. This is similarly unethical, in my mind on a level with the sort of p-hacking prevalent in social sciences. You're completely right! They just used the default. It's just perhaps possible that there might be other factors worth considering.