7 ms·

> I'm not only disappointed in the ACLU for misinterpreting the results... Did they? This blog post describes some implementation choices which could make thei

> I'm not only disappointed in the ACLU for misinterpreting the results... Did they? This blog post describes some implementation choices which could make their false positive rate lower. It reveals no information about how Rekognition is being deployed by LEAs, and there’s no meaningful regulation or oversight about how that happens. We can’t tell what the assumptions and implementation choices of local police departments are, because that’s top secret information. Most people acting on the predictions have no idea what they are, let alone the implications. It’s fine to ding the ACLU study on methods —- you can, because they published some. No-one would argue that a toy model based on members of Congress is representative of the public at large. But it’s more representative that a press release from a company that are trying to normalize and monetize mass surveillance via facial recognition.

l9k 8y ago
The original ACLU title is: Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. It was spread on the news with the same conclusion. It puts directly the blame on Amazon and its technology and doesn't mention any configuration they used.
- confounded 8y ago
  Why would a title mention API/service configuration? This is not common practice, even in academic applied ML papers. They used the default settings, and a training set which seems extremely likely to be used in ‘LEA-production’ Any user, including LEAs, is free to use which ever configuration options they like. The criticism with the most statistical implications is the confidence level used. The ACLU were clear they they used the defaults. I can take at face value that moving from 80% to 99% confidence on a sample of 500 faces could produce 0 false positives. However, on the faces of the tens/hundreds of thousands of people that might move through the center of town on any given day, the implications for causing ordinary people serious harm are large. I am personally shocked that Amazon are willing to “recommend” law enforcement actions at any level of confidence, from an unintrospectable machine learning system.
  - cptskippy 8y ago
    The title is akin to saying "We bought <gun maker>'s hunting rifle, pointed it at Congress, pulled the trigger, and it shot 22 members of Congress". It's a tool, you're responsible for how it is used, not the manufacturer. I would imagine the confidence level would be tweaked by Law Enforcement or anyone else based on the results. If you're monitoring cameras for missing persons or shooting suspects and getting 0 hits, you might lower it because it's better to dismiss false positives than the alternative. Conversely you might increase it if you're overwhelmed by the number of false positives.
    - d4l3k 8y ago
      Since you're comparing facial recognition to guns and guns are regulated by law and have strict rules for use, perhaps facial recognition should as well? I think there should be a discussion about laws/regulations about facial recognition use. Even if Amazon removes their service there will always be companies that don't care about the ethics as much. Strict rules instead of public shaming seems like the better approach
      - RhodesianHunter 8y ago
        If you screw up with a gun someone dies. If you screw up with facial recognition a human double-checks and nothing happens. Scissors are dangerous but we don't regulate those either.
        syshum 8y ago
        ummm no... First of it is not a given if you "screw up with a gun someone dies" that is not the only result Further it is laughable to believe that with facial recognition a "human double-checks" we have seen time and time again that new technology entering the realm of criminal justice can be used to wrongfully convict people at an alarming rate. See Hair and Bite mark controversy or the number of Labs that have been compromised in recent years The fact is that a "humans" that are suppose to be "double checking" do not in reality, if the computer spits out "Well the computer says it is person X" that is what the Jury will hear, and the Jury will not question the technology and the person will end up in prison until such time as someone like the Innocent project comes along to invalidate either the technology on the whole or the application of the technology
ryanmonroe 8y ago
Taking results of 5% false positives coming from a tool with a 80% confidence level as anything other than "it worked better than expected" is misinterpreting the results.
- sgt101 8y ago
  How many police forces would (without this debate) have ever done more than use the default settings? BTW . Why is the default 80% Does it deliver 4/5 TP in the wild? Has anyone done an evaluation? What about vs. the kind of corruptions that are common in the wild - how does it handle odd light, reflections, shadows?
mful 8y ago
Yes, they did. It's great that they explained their config choices, so that folks with domain knowledge can evaluate reliability of their test, and report back. For those of us without domain knowledge, however, what the ACLU published is wildly misleading, to the point of being actively disingenuous. What are folks without baseline STEM education, much less AI education, supposed to take from the ACLU article other than "Amazon AI is racist and dangerously ineffective"? Do you think they are opening up the Rekognition docs to evaluate if the default config is appropriate for this test? (By the way, I wouldn't be surprised if Rekognition is actually racially biased. I also feel a bit icky criticizing the ACLU here — they do great work, and I encourage everyone to donate.)

l9k 8y ago

The original ACLU title is: Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. It was spread on the news with the same conclusion. It puts directly the blame on Amazon and its technology and doesn't mention any configuration they used.

confounded 8y ago
Why would a title mention API/service configuration? This is not common practice, even in academic applied ML papers. They used the default settings, and a training set which seems extremely likely to be used in ‘LEA-production’ Any user, including LEAs, is free to use which ever configuration options they like. The criticism with the most statistical implications is the confidence level used. The ACLU were clear they they used the defaults. I can take at face value that moving from 80% to 99% confidence on a sample of 500 faces could produce 0 false positives. However, on the faces of the tens/hundreds of thousands of people that might move through the center of town on any given day, the implications for causing ordinary people serious harm are large. I am personally shocked that Amazon are willing to “recommend” law enforcement actions at any level of confidence, from an unintrospectable machine learning system.
- cptskippy 8y ago
  The title is akin to saying "We bought <gun maker>'s hunting rifle, pointed it at Congress, pulled the trigger, and it shot 22 members of Congress". It's a tool, you're responsible for how it is used, not the manufacturer. I would imagine the confidence level would be tweaked by Law Enforcement or anyone else based on the results. If you're monitoring cameras for missing persons or shooting suspects and getting 0 hits, you might lower it because it's better to dismiss false positives than the alternative. Conversely you might increase it if you're overwhelmed by the number of false positives.
  - d4l3k 8y ago
    Since you're comparing facial recognition to guns and guns are regulated by law and have strict rules for use, perhaps facial recognition should as well? I think there should be a discussion about laws/regulations about facial recognition use. Even if Amazon removes their service there will always be companies that don't care about the ethics as much. Strict rules instead of public shaming seems like the better approach
    - RhodesianHunter 8y ago
      If you screw up with a gun someone dies. If you screw up with facial recognition a human double-checks and nothing happens. Scissors are dangerous but we don't regulate those either.
      - syshum 8y ago
        ummm no... First of it is not a given if you "screw up with a gun someone dies" that is not the only result Further it is laughable to believe that with facial recognition a "human double-checks" we have seen time and time again that new technology entering the realm of criminal justice can be used to wrongfully convict people at an alarming rate. See Hair and Bite mark controversy or the number of Labs that have been compromised in recent years The fact is that a "humans" that are suppose to be "double checking" do not in reality, if the computer spits out "Well the computer says it is person X" that is what the Jury will hear, and the Jury will not question the technology and the person will end up in prison until such time as someone like the Innocent project comes along to invalidate either the technology on the whole or the application of the technology

ryanmonroe 8y ago

Taking results of 5% false positives coming from a tool with a 80% confidence level as anything other than "it worked better than expected" is misinterpreting the results.

sgt101 8y ago
How many police forces would (without this debate) have ever done more than use the default settings? BTW . Why is the default 80% Does it deliver 4/5 TP in the wild? Has anyone done an evaluation? What about vs. the kind of corruptions that are common in the wild - how does it handle odd light, reflections, shadows?

mful 8y ago

Yes, they did. It's great that they explained their config choices, so that folks with domain knowledge can evaluate reliability of their test, and report back. For those of us without domain knowledge, however, what the ACLU published is wildly misleading, to the point of being actively disingenuous. What are folks without baseline STEM education, much less AI education, supposed to take from the ACLU article other than "Amazon AI is racist and dangerously ineffective"? Do you think they are opening up the Rekognition docs to evaluate if the default config is appropriate for this test? (By the way, I wouldn't be surprised if Rekognition is actually racially biased. I also feel a bit icky criticizing the ACLU here — they do great work, and I encourage everyone to donate.)