4 ms·

>It has also been well-documented that hostile surveillance technologies are disproportionately used to target marginalized groups, including migrants and black

>It has also been well-documented that hostile surveillance technologies are disproportionately used to target marginalized groups, including migrants and black and brown people. Isn't this where AI could really shine? By being colorblind? I would imagine the AI would be used to sort through the vast amounts of data generated and could be used to remove human bias.

fwlr 4y ago
I’ve worked on and with SOTA facial recognition, there’s largely two separate claims about the intersection of facial recognition and racial discrimination. One is that it’s inaccurate on minorities, and the other is that it’s used to target minorities. First claim is false, and as far as we could tell rests largely on one experiment a researcher did with a database of a few hundred faces and the default settings on Amazon’s Rekognition. That test demonstrated worse accuracy on minorities and therefore “facial recognition is inaccurate on minorities” became true enough to become a reportable fact - which it was, at the time and scoped to that instantiation of that technology, but from there it got included hundreds or thousands of future articles on facial recognition and racial discrimination, which probably creates an incorrect impression of the overall technology. Reality is that SOTA face rec uses large enough training sets to be “colorblind”, at least in the object level sense of not being more likely to confuse two different people for each other just because they are both minorities of the same ethnicity (I fondly recall an Asian-American CEO saying “hey, it can tell apart Asians better than I can!”). Second claim is true, the high-impact pointy end of facial recognition usage is tackling crime, so it necessarily inherits all the racial discrimination issues in law enforcement.
- dragonwriter 4y ago
  > I’ve worked on and with SOTA facial recognition, there’s largely two separate claims about the intersection of facial recognition and racial discrimination. One is that it’s inaccurate on minorities, and the other is that it’s used to target minorities. > First claim is false, Maybe for some definition of “state of the art”, but, in terms of practical applications that matter even today, it is not. [0] > and as far as we could tell rests largely on one experiment a researcher did with a database of a few hundred faces and the default settings on Amazon’s Rekognition. Then…you haven’t tried very hard to tell what the basis for it is. [1][2] [0] https://www.theguardian.com/us-news/2023/feb/08/us-immigration-cbp-one-app-facial-recognition-bias https://www.theguardian.com/us-news/2023/feb/08/us-immigrati... [1] Here’s a 2016 article that covers what was already a substantial history of studies showing such bias, going back to 2011. https://apexart.org/images/breiner/articles/FacialRecognitionSoftwareMight.pdf https://apexart.org/images/breiner/articles/FacialRecognitio... [2] But the bigger, broader, and more recent study was in 2019 from NIST. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software https://www.nist.gov/news-events/news/2019/12/nist-study-eva... : “While it is usually incorrect to make statements across algorithms, we found empirical evidence for the existence of demographic differentials in the majority of the face recognition algorithms we studied,” said Patrick Grother, a NIST computer scientist and the report’s primary author. […] The NIST study evaluated 189 software algorithms from 99 developers — a majority of the industry.
  - fwlr 4y ago
    Note I’m not claiming that one Amazon Rekognition test is the only demonstration of inaccuracy on minorities; there is obviously a lot more evidence, as you’ve capably provided. I was talking about the provenance of the belief of inaccuracy, which in our research at the time (term used loosely, we weren’t publishing a paper or anything, just trying to understand the market) overwhelmingly went like so: “I read it in the media. Which article? This one, here’s a link”, and the linked article has a statement that facial recognition fails on minorities, and that statement links to the Rekognition result. There may well have been many more studies with stronger groundings! As it happened, that wasn’t what the media tended to cite, and so it wasn’t the basis for what people tended to end up believing. (Disclaimer for the next section: my ego compels me to push back on the claim that I didn’t try very hard to understand, this is probably nitpicking, etc.) I am familiar with the first article, I read it when it was published, and it doesn’t really support your claim of a substantial history of studies - “Unfortunately, the two studies discussed here are among the only works on racial bias published in the past decade—far too little review for a technology with the power to implicate people as suspects in a criminal investigation” is a direct quote (I recall agreeing strongly with it at the time). The examples it gives are also illustrative: “One algorithm, which failed to identify the right person in 1 out of 10 encounters with Caucasian subjects, failed nearly twice as often when the photo was of an African American” and “The 2012 study on mug shots found that an algorithm trained exclusively on either African American or Caucasian faces recognized members of the race in its training set more readily than members of any other race”. The historical context at the time of that article is that NIST really wanted to invest more heavily in surveying and analyzing facial recognition racial bias, and this piece was arguing in support of that. The 2019 NIST report wasn’t out at the time I did the research, but it was in the works. We had actually discussed submitting to NIST’s call for algorithms, but the broad perception in the industry was that was a silly thing to do with your valuable IP, and I personally suspect that as a result most of the submissions to the NIST were not cutting edge or commercially viable. With that said, the NIST report also agrees with me: “those that are the most equitable also rank among the most accurate”. As for the CBP One app, it should really use a better facial recognition system! The description in that article made it sound like the app was failing to even register Black faces as faces, which is ridiculously bad for an app with that much responsibility. There are many better systems out there and they should upgrade as a matter of urgency. (I’m not unsympathetic to this issue; a large part of the reason I’m no longer working on face rec is that I was refused entry by US immigration. We actually used to joke* that we were building the tech to keep me out of the country.) * Note this is a joke, to my knowledge we weren’t actually working on border control applications.

fwlr 4y ago

I’ve worked on and with SOTA facial recognition, there’s largely two separate claims about the intersection of facial recognition and racial discrimination. One is that it’s inaccurate on minorities, and the other is that it’s used to target minorities. First claim is false, and as far as we could tell rests largely on one experiment a researcher did with a database of a few hundred faces and the default settings on Amazon’s Rekognition. That test demonstrated worse accuracy on minorities and therefore “facial recognition is inaccurate on minorities” became true enough to become a reportable fact - which it was, at the time and scoped to that instantiation of that technology, but from there it got included hundreds or thousands of future articles on facial recognition and racial discrimination, which probably creates an incorrect impression of the overall technology. Reality is that SOTA face rec uses large enough training sets to be “colorblind”, at least in the object level sense of not being more likely to confuse two different people for each other just because they are both minorities of the same ethnicity (I fondly recall an Asian-American CEO saying “hey, it can tell apart Asians better than I can!”). Second claim is true, the high-impact pointy end of facial recognition usage is tackling crime, so it necessarily inherits all the racial discrimination issues in law enforcement.

dragonwriter 4y ago
> I’ve worked on and with SOTA facial recognition, there’s largely two separate claims about the intersection of facial recognition and racial discrimination. One is that it’s inaccurate on minorities, and the other is that it’s used to target minorities. > First claim is false, Maybe for some definition of “state of the art”, but, in terms of practical applications that matter even today, it is not. [0] > and as far as we could tell rests largely on one experiment a researcher did with a database of a few hundred faces and the default settings on Amazon’s Rekognition. Then…you haven’t tried very hard to tell what the basis for it is. [1][2] [0] https://www.theguardian.com/us-news/2023/feb/08/us-immigration-cbp-one-app-facial-recognition-bias https://www.theguardian.com/us-news/2023/feb/08/us-immigrati... [1] Here’s a 2016 article that covers what was already a substantial history of studies showing such bias, going back to 2011. https://apexart.org/images/breiner/articles/FacialRecognitionSoftwareMight.pdf https://apexart.org/images/breiner/articles/FacialRecognitio... [2] But the bigger, broader, and more recent study was in 2019 from NIST. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software https://www.nist.gov/news-events/news/2019/12/nist-study-eva... : “While it is usually incorrect to make statements across algorithms, we found empirical evidence for the existence of demographic differentials in the majority of the face recognition algorithms we studied,” said Patrick Grother, a NIST computer scientist and the report’s primary author. […] The NIST study evaluated 189 software algorithms from 99 developers — a majority of the industry.
- fwlr 4y ago
  Note I’m not claiming that one Amazon Rekognition test is the only demonstration of inaccuracy on minorities; there is obviously a lot more evidence, as you’ve capably provided. I was talking about the provenance of the belief of inaccuracy, which in our research at the time (term used loosely, we weren’t publishing a paper or anything, just trying to understand the market) overwhelmingly went like so: “I read it in the media. Which article? This one, here’s a link”, and the linked article has a statement that facial recognition fails on minorities, and that statement links to the Rekognition result. There may well have been many more studies with stronger groundings! As it happened, that wasn’t what the media tended to cite, and so it wasn’t the basis for what people tended to end up believing. (Disclaimer for the next section: my ego compels me to push back on the claim that I didn’t try very hard to understand, this is probably nitpicking, etc.) I am familiar with the first article, I read it when it was published, and it doesn’t really support your claim of a substantial history of studies - “Unfortunately, the two studies discussed here are among the only works on racial bias published in the past decade—far too little review for a technology with the power to implicate people as suspects in a criminal investigation” is a direct quote (I recall agreeing strongly with it at the time). The examples it gives are also illustrative: “One algorithm, which failed to identify the right person in 1 out of 10 encounters with Caucasian subjects, failed nearly twice as often when the photo was of an African American” and “The 2012 study on mug shots found that an algorithm trained exclusively on either African American or Caucasian faces recognized members of the race in its training set more readily than members of any other race”. The historical context at the time of that article is that NIST really wanted to invest more heavily in surveying and analyzing facial recognition racial bias, and this piece was arguing in support of that. The 2019 NIST report wasn’t out at the time I did the research, but it was in the works. We had actually discussed submitting to NIST’s call for algorithms, but the broad perception in the industry was that was a silly thing to do with your valuable IP, and I personally suspect that as a result most of the submissions to the NIST were not cutting edge or commercially viable. With that said, the NIST report also agrees with me: “those that are the most equitable also rank among the most accurate”. As for the CBP One app, it should really use a better facial recognition system! The description in that article made it sound like the app was failing to even register Black faces as faces, which is ridiculously bad for an app with that much responsibility. There are many better systems out there and they should upgrade as a matter of urgency. (I’m not unsympathetic to this issue; a large part of the reason I’m no longer working on face rec is that I was refused entry by US immigration. We actually used to joke* that we were building the tech to keep me out of the country.) * Note this is a joke, to my knowledge we weren’t actually working on border control applications.