9 ms·
In every reported mistaken arrest using facial recognition, person was Black
- RugnirViking 3y agoWhile I have no doubt of the potential for facial recongition systems to have NUMEROUS inbuilt biases, including even IF every actor involved in their creation and deployment held no ill intent. There have only been six such cases apparently. It seems a bit premature for this kind of hedline. That also being said, this kind of software is just bad in general, even outside of the biases thing. One more thing. A quote from the article: "Detroit's police chief said their facial recognition technology, when used alone, fails 96% of the time, Insider previously reported." - What?? That's insane. Surely those numbers can't be right? the source on that he goes into more detail with similar specific numbers. I really hope it doesn't only exist to give more ammunition in court, and they don't care whether it works at all...
- whaleofatw2022 3y agoAI is the new Polygraph... Except now it can also be used by academia, employers, and more!
- dhbanes 3y agoAI is imperfect but it works a certain percentage of the time in a certain percentage of cases. Polygraphs are just complete pseudo-science bullshit.
- jahewson 3y agoIt is insane. Because it’s completely made-up: > "If we were just to use the technology by itself, to identify someone, I would say 96% of the time it would misidentify” He pulled this statistic out his ass.
- lucasjans 3y ago> "Detroit's police chief said their facial recognition technology, when used alone, fails 96% of the time, Insider previously reported." When it fails 96% of the time, are they given a hundred possible matches to scroll through? If so, that's concerning to the majority of people showing up without any affiliation to a crime.
- MrStonedOne 3y ago[dead]
- cltby 3y ago> "We believe this results from factors that include the lack of Black faces in the algorithms' training data sets..." the researchers wrote in an op-ed for Scientific American. > The research also demonstrated that Black people are overrepresented in databases of mugshots. The sort of clear-headed thinking that makes the AI bias field as respected as it is.
- sonofhans 3y agoYeah, that’s terrifying. It’s the same reasoning as, “Of course more Black people are convicted of crimes — we arrest more of them!”
- wuiheerfoj 3y agoBut presumably the AI is just identifying faces, not judging who is and is not a criminal
- vkou 3y agoIf the training set is biased, the AI becomes biased. GIGO.
- Tozen 3y agoWhen this software is being sold to these departments, it's amazing that people in the chain don't seem to be talking enough about the training set used or performance on certain populations. If you are going to arrest or build a case on facial recognition, you would think that they would be prepared to defend its accuracy against a broad range of demographics. Embarrassing failures and mistaken arrests, hurts their program, not to mention the money the city losses in lawsuits.
- vkou 3y agoThe answer to this conundrum might be that neither the departments nor the vendors are particularly interested in avoiding bias. Paying lip service is generally sufficient.
- deleted 3y ago[deleted]
- jacknobody 3y agoIn northern Cape York (Australia) I have several friends who cannot be successfully photographed because their skin is so dark that cameras are unable to resolve their facial features. I don't expect facial recognition to be much use on Cape York.
- anaganisk 3y agoWhat if the AI is designed to flag them as potential bad actors.
- ipaddr 3y agoFlag the person who cannot be photographed? What are we flagging again?
- ithkuil 3y agoClearly all such people that cannot be photographed are all the same person
- afiori 3y agoYou need to look at it from the perspective of the system. The system has a database of individuals and is asked "does this photo matches any of the individuals in your database?" If someone whose photo is basically a dark circle with eyes is entered in the system then the next time you ask the system to identify a photo that is basically a dark circle with eyes the system will respond to with the previous match. Like a bloom filter with a bitvector of size 1.
- staticautomatic 3y agoPretty sure wedding photographers solved this problem a long time ago, though I hope none of them let the facial recognition vampires in on the mysterious secrets of basic lighting and exposure.
- quicklime 3y ago
- tedunangst 3y agoWould be interesting to see when false matches are double checked by police and when police simply proceed to the arrest.
- cratermoon 3y agoAs far back as 1985 Brian Winston wrote[1], "It is one such expression – color film that more readily photographs Caucasians than other human types – that is our concern in this piece." Richard Dyer wrote in 1997, "The aesthetic technology of photography, as it has been invented, refined, and elaborated, and the dominant uses of that technology, as they have become fixed and naturalised, assume and privilege the white subject."[2]. Key findings in the bias of facial recognition come from a 2018 paper in Proceedings of Machine Learning Research which found, "We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%. The substantial disparities in the accuracy of classifying darker females, lighter females, darker males, and lighter males in gender classification systems require urgent attention if commercial companies are to build genuinely fair, transparent and accountable facial analysis algorithms." 1 "A Whole Technology of Dyeing: A Note on Ideology and the Apparatus of the Chromatic Moving Image" <https://www.jstor.org/stable/20025012 https://www.jstor.org/stable/20025012> 2 Dyer, Richard. 1997. White : Essays on Race and Culture. London: Routledge. 3 "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification" <https://proceedings.mlr.press/v81/buolamwini18a.html https://proceedings.mlr.press/v81/buolamwini18a.html>
- marsven_422 3y ago[dead]
- rootusrootus 3y agoIs it plausible that cameras have more trouble with dark skin because of reduced contrast of facial features? Or is that already studied and discarded, and this is just a training set problem?
- jxf 3y agoThat's very plausible. But then that means there is an inherent bias against people with darker skin. So the more interesting question is: why wasn't that bias anticipated and caught, and if it was known, why was it allowed to go to production where it affected the lives and liberty of one group of people disproportionately?
- jurimasa 3y agoThat's easy: racism.
- vuln 3y agoBecause the people selling the software reported the false positive rate a x per y but just accidentally forget to report that ALL of the false positives are of one single group. I would love to be in a sales meeting with a company that develops this type of software and the police that buy it.
- mapt 3y agoBecause nobody cared. No consequences, profit-driven company, metric-driven agencies, nobody cared. Constructing a modern-day criminal justice system from the ashes of the Bad Old Days is going to be a project for our generation or the next or the next after that, but it probably requires burning things to the ground. In the meantime: Most of the liberal democratic ideals you and I were raised on with regard to criminal justice were from academia musing on what should be, or from copaganda television shows lying about what is. They live in our heads - if you want them to be real, you have to make them real.
- 6D794163636F756 3y agoThere's also the chance that they didn't care to build their training set to match the expected population. Another industry dealing with this is automotives. Until recently tests were done exclusively on male crash test dummies. This resulted in higher mortality rates in women as things like seatbelts and airbags were tested on models not reflecting their bodies. This would all be because either they didn't care or they didn't realize they should care. Which is to say, I agree with you and the problem you touched on is far more pervasive and effects more people than just the legal system and we need to start doing a better job addressing it.
- NotSuspicious 3y ago> All six known reports of false arrests due to facial recognition technology were made by Black people. It's fascinating how sample size differs based on who wants what policy to be enacted. What are the false arrest rates due to people just looking at a picture and thinking "That's the one"?
- andirk 3y agoWithout comparing to more common forms of ID, I don't see how this data proves much at all. Furthermore, with a large black population like Baltimore, if they mostly ran the software on black people, then yeah the false IDs will also be mostly black people.
- gopher_space 3y agoIt means their software is garbage and lawsuits will follow. Using this software moving forward is probably a bad idea since we know that it’s broken.
- fastaccount123 3y agoDoes it? I don't want to let the cops off the hook here. The previous commenter suggested a mechanically plausible way: "decent software used in a racist way" would have the above problem
- 6D794163636F756 3y agoUnfortunately in America it is very difficult to sue or otherwise obtain redress of grievance when the perpetrator of an act is a police officer. It is almost certainly easier to sue the company to stop the software's use than sue the police to stop them from using it.
- shkkmo 3y agoNot sure why this is being downvoted. This is a factually accurate description of an issue highly relevant to the topic. If you aren't aware of the specifics, research "qualified immunity" , a legal doctrine that was not passed into law but invented by the courts to shield police from the consequences of violating your civil rights.
- Animats 3y agoThere's work going on to look a little further into the infrared to fix this.[1] See the image set "Sample database images in all three spectra". High dynamic range images help, a lot. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10320199/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10320199/
- bandrami 3y agoWe can't even get automatic hand dryers to work for nonwhite people so I think usable face recognition is still a ways off
- tasogare 3y agoGiven that hand dryers are ubiquitous in Japan, they're certainly working on a large part of nonwhite people.
- golemiprague 3y agoIn America "white" is not a colour, Italian or Arabs with dark skin are also "white", unless they are from south America, then they are latinos. The only true meaning of white in the US is "not black", so those things working in Japan is not a valid counter point.
- jadamson 3y ago> The only true meaning of white in the US is "not black", That doesn't even follow from your own previous sentence - it would have to mean "not black or latino". Have you asked any South East Asians if they consider themselves white?
- felipelemos 3y agoI think he is referring to what others considers them, not themselves.
- jadamson 3y agoHe's trying to generate a post-hoc justification for an inaccurate statement, there's nothing more to it than that. Still, self-identification matters and I wanted to see if he'd cross the Rubicon and try to force the white identity on groups that don't want it. Bearing in mind his linchpin example - Arabs - are neither perceived as nor self-identify as white [1], it's bizarre to suggest that it follows that the Japanese, a separate case anyway, must also be white. Add to that his conflation of ethnicity with race [2], and it might be the single most stupid comment I've ever seen on HN. Just say "OP meant to say black", there's no need to construct an alternate reality in which non-white and black are synonyms. [1] https://www.pbs.org/newshour/nation/why-arab-americans-are-pushing-for-a-middle-east-or-north-african-category-on-the-census https://www.pbs.org/newshour/nation/why-arab-americans-are-p... "many in the MENA community do not share the same lived experience as White people with European ancestry, do not identify as White, and are not perceived as White by others." [2] https://www.census.gov/quickfacts/fact/note/US/RHI625222 https://www.census.gov/quickfacts/fact/note/US/RHI625222 "People who identify their origin as Hispanic, Latino, or Spanish may be of any race."
- henearkr 3y agoThe facial ID standards should move towards adding a 3D scan of the face (especially now that biometrics are digitalized and stored on chips on ID cards and passports). That would solve this kind of problems. Neuronal networks could then just add that in their training and be accurate again, and surveillance systems could use a complementary lidar.
- michaelmrose 3y agoHow do you intend on getting 3D scans of the crimes in progress?
- deleted 3y ago[deleted]
- 10g1k 3y agoLight = contours = facial recognition...
- commonlisp94 3y agoContrast helps. But running a convolution model on a single video frame, or aggregate of frames, is not how human vision works either. Give a human a glob of pixels with no context and they will struggle to identify it too. Identification involves motion and knowledge of the scene.
- xqcgrek2 3y agoN=6
- wonderwonder 3y agoI hope each of these facial recognition matches is reviewed by someone of the same race / skin color as the alleged perp. Often times to someone of a different race it can be hard to see differences that are obvious to those of the same race. I have heard both black and white people say "They all look the same to me" regarding people of the other race. I am sure the same applies to Asians.
- xwdv 3y agoSome races really do look more similar to each other than others. In America there is a high degree of mixed genes, that you don’t find in a more homogenous genetic pool such as Japan or places in Africa.
- LeoNatan25 3y agoIt’s just to your untrained eyes that they look “more similar”. No, all Asians do not look alike. Also, contrary to popular belief, they do age. And blacks do “crack”. They just have different patterns of progressions, and your brain is not trained to notice the nuances.
- stormfather 3y agoGo to Korea and look around on the subway. There is objectively less phenotypical variety there.
- LeoNatan25 3y agoThat's a silly argument. Go to Norway (or elsewhere in Scandinavia)—there is "objectively less phenotypical variety there" too; would you extrapolate that all caucasians look alike?
- xwdv 3y agoI am almost certain Asians (not Indians etc) look more alike than other races. Sure, there may be differences that let you identify an individual, but at a distance of say 20 feet, can you resolve those differences and identify who you’re looking at? Clothes and hair styles might help, but if not available can you simply identify a face? You’d need a close look and a lot of practice. What about if they wear a mask? When an Asian wears a mask, they could almost be anybody. Asians are even more androgynous than other races too so it’s possible to not even know if they are male or female without further context, typically hair and body shape. I’m not even sure what a “typical” white or black person looks like, but the typical Asian always seems to be straight black hair and certain shapes of eyes and light skin tone. Am I crazy or is this accurate?
- Eumenes 3y agoFacial recognition in policing is unethical, authoritarian, prone to abuse in the first place - don't need to bring race into it.
- mikrotikker 3y agoExactly there was a Netflix doco a while back showing the struggle of black people to get recognised by facial recognition. I couldn't help but feel that they were advocating for their own oppression and the conspiracy theorist in me wondered if it was a ploy by some three letter agency.
- stormfather 3y agoYeah I'm pretty sure this is actually racism against pale people. I would love to be camouflaged against the surveillance state!
- RecycledEle 3y agoGiven the The Media only report on mistaken arrests when the victim is black, it makes sense that every victim The Media reported on was black. It's called a self selecting sample.
- polygamous_bat 3y ago> The Media only report on mistaken arrests when the victim is black Strong claims have a strong burden for a proof.
- benced 3y agoIt feels like this issue could be fixed with human processes to cross check the facial recognition results.
- Dwedit 3y agoWhat if there was a localized adaptive contrast enhancing step performed before anything hits any neural networks?
- not_enoch_wise 3y agoIf only someone had warned- Lol jk, years of warnings, software vendors don’t care
- stevage 3y agoI wonder if problems like this would be drastically reduced if there was an instant, automatic compensation offered in any case of mistaken arrest, like $5,000 or something.
- brightlancer 3y ago"Porcha Woodruff — a mother from Detroit, Michigan — became the first woman to report that police falsely identified her as a suspect using faulty facial recognition, The New York Times reported." As explained in the NYTimes article (yet omitted by BI), the victim picked her photo out of a line-up. "Five days after the carjacking, the police report said, the detective assigned to the case asked the victim to look at the mug shots of six Black women, commonly called a “six-pack photo lineup.” Ms. Woodruff’s photo was among them. He identified Ms. Woodruff as the woman he had been with. That was the basis for her arrest, according to the police report." https://www.nytimes.com/2023/08/06/technology/facial-recognition-false-arrest.html https://www.nytimes.com/2023/08/06/technology/facial-recogni... To put this on facial recognition is dishonest.
- prawn 3y agoIt was facial recognition tech of some sort that put her into the six-pack: "The ordeal started with an automated facial recognition search, according to an investigator’s report from the Detroit Police Department." "A detective with the police department’s commercial auto theft unit got the surveillance video from the BP gas station, the police report said, and asked a crime analyst at the department to run a facial recognition search on the woman."
- steve76 3y ago[dead]
- puffyengineer 3y ago[dead]
- shrubble 3y ago"the Baltimore Police Department ran nearly 800 facial recognition searches in 2022. The Detroit Police Department makes about 125 facial recognition searches each year," So, 6 total that we know about, over 925 searches? How many human-initiated IDs (that is, searches by cops themselves, however you want to define that) led to false IDs?
- _8j50 3y agoWhy do you and others on the thread keep asking that. If a human made a mistake he or she is held accountable individually and punished. The software cannot be punished, it does not have the ability to fear punishment and be perfectly careful. The point you are missing is that the software isn't augmenting human responsibility, it is largley replacing it. Even if not on paper, humans now have an excuse to lay responsibility on the software. This is the same problem I have with ML and my work, if I have to triple check its output anyways then what good is it? If I don't, who will take responsibility for the mistake? Even if humans made more errors, you can tune humans by adjusting their incentives. For example, if they faced a couple of years of jail time would humans keep the same amount of error rate? The proper use of such software is to narrow down the list of suspects, humans then have to do much more vetting and verification, assuming the software's results are false before concluding the identification. The time saved scouring through a large list of suspects should be spent painstakingly vetting and independently corroborating software identified hits. There is no replacement for humans doing the actual/final identification and assuming the responsibility if their mistakes.
- golergka 3y ago> If a human made a mistake he or she is held accountable individually and punished. Is she?
- _8j50 3y agoYes, in a properly run police department, rare as they may be.
- whatshisface 3y ago
- drc500free 3y agoThis is one of the few topics on hn that I actually have some expertise in, after a ten year career consulting for the three big biometric systems; involved in standards work, accuracy testing, and R&D program management. Note that I have been out of the game for 7-8 years, but things do not change fast. There are several issues, some of which the biometrics community is aware of and some of which are outside their worldview. The first is that the face matching algorithms are inherently less accurate on people of color given the sensors and algorithms in use. This has been an issue going back to color film being developed for Caucasian consumers - the range on a lot of sensors simply doesn't provide enough contrasting tones for brown and black people. The face detection, extraction, and matching algorithms are often not developed using datasets that match the demographics they will be used on, and at least 7-8 years ago there wasn't much focus on providing ethnicity-specific error rates. The biometrics community largely knows about this and understands it. The second issue is that even the flagship programs don't measure Precision and Recall. They operate purely off metrics like False Match Rate and False Positive Identification Rate. Those are closed-set (i.e. "everyone is enrolled"), tech-only metrics that can be measured by NIST, that don't consider system usage or CONOPs. Program management asks for Precision and Recall metrics (not by name, because they don't know what they are), and tech responds with True Match Rate and True Positive Identification Rate. This is an entire worldview where whole-system accuracy has been placed fully on algorithmic error rates. Leadership generally DOES NOT KNOW that error rates are just as dependent on usage - i.e. that if you search for someone who isn't in the database, 100% of the matches will be false. They do not understand which part of the confusion matrix they are being told about. The third issue is that, while these massive federal programs are run by ethical and mission-oriented professionals who have hill-climbed their way to a pretty accurate system with stable usage, the talent and professionalism isn't there across the state and local level (there are a couple bright spots where at least the tech is well understood). Face matching systems are generally not lights-out like fingerprint or iris systems (fingerprint systems do generally require a manual verification step, but that's driven more by policy and history than the accuracy metrics require); FR requires a second step where a trained face matching expert looks at the proposed matches in the consideration set and uses several techniques to assess if they are correct - it's a notable difference compared to fingerprint that candidate faces are not supposed to be presumed matches when they come out of the automated system, and even after the human expert they are generally supposed to be evidence/leads that need to be corroborated through other means. As far as I know (again, several years out of date now), no one has checked whether the accuracy of the human experts varies by ethnicity of the subject. Even the FBI's professionalism broke down with the Madrid Train Bombing, resulting in the arrest of a random Brown lawyer in Seattle (again, no one considered that looking for an international political terrorist in a domestic criminal database meant a much lower prior probability than IAFIS generally operates with, before you even start talking about stitching together multiple partial fingerprints, and you shouldn't be considering anything in the person's file while making an objective matching decision). A local police department is going to be much worse in terms of policy and system usage. And finally, there is the reality that once you are enrolled in a booking database, you are accused of every crime that happens in any linked jurisdiction until the system goes offline. The probability that you will have an encounter with the police is not uniformly distributed across the population, nor is the probability that such an encounter will lead to a booking where your photo is taken and enrolled in a database forever. This is not something the biometrics community really discusses at all, but every enrollment in one of these systems is hundreds of thousands of opportunities for a false match per year, hanging over your head like bad debt. The end result is that it's not surprising that just under one in six people in America are Black, but all six of these false arrests were Black. It's also not easily fixed in one place by "giving everyone implicit bias training."
- archsurface 3y agoAnd the others haven't been reported because....?
- spicymaki 3y agoThis illustrates a serious problem with AI. We (Humans) are ceding our autonomy to AI without any evidence of the validity of the outcomes. The industry puts up a front saying they will be careful, but the reality is that they are faking it until they make it. Techies want to fix the problem by adding more data points or tweaking the algos, but the real fix will take actual structural societal changes which is time consuming and expensive. Technology alone will not fix this.
- AndrewKemendo 3y agoThis is the microcosm of the biggest problem facing humanity Of course this is the result, because our society can only create data that is embedded with our personal and social structural biases and fears. There is no way to possibly align artificial intelligence because we are only demonstrating anti-social biased actions via the data we create, it’s just a simple fact, there’s no way around it, there’s no pathway to fix it; As long as we are a selfish antisocial society, creating antisocial data, we’re going to make antisocial biased machines
- balderdash 3y agoWait Baltimore and Detroit are like ~60-80% black, and people are drawing inferences about set of six where the people were black - instead of say 4-5 black and 1-2 white?! [this doesn't excuse the police actually arresting the people]
- beej71 3y agoHow many black people in a row would it take?
- balderdash 3y agoNo idea - my whole point is that without more data you can’t really draw any inferences here, (e.g. it’d be nice to know how many people were sent for facial recognition, and the ethnic composition of that group, the. It would be nice to know the misidentification rate in that BEFORE the police went out arresting people), one answer here may be that facial recognition’s failure rate is quite high, but the police don’t do the requisite follow up work when it’s a black suspect vs a white suspect. Which brings me back to my original point, we need more data, but what limited data that is supplied doesn’t YET give us a smoking gun. [separately I’m hopeful that those wrongly arrested enjoy success in the civil courts]
- n8henrie 3y agoI think you'd need to know the distribution of criminal suspects, which may or may not reflect the general population. For example if 50% of the suspects are white, then 80% of the population being black wouldn't matter. Right? EDIT: On second thought perhaps not, I think some are arguing for a biased training dataset, which would presumable be the "population."
- imchillyb 3y ago"Hello, motion sensors, I'm motioning. Please sense me." > https://www.youtube.com/watch?v=XyXNmiTIupg https://www.youtube.com/watch?v=XyXNmiTIupg
- talldatethrow 3y ago[flagged]
- chii 3y ago> African Americans commit about half the murders in the US not disagreeing, but you've also committed the statistical cardinal sin of survivorship bias here as well. It's not half the murders, it's half the _prosecuted_ murders. You don't know how many are not caught or known.
- tdehnel 3y agoSo it could very well be more than half.
- flangola7 3y agoDoubtful. Whites have more wealth and better lawyers.
- talldatethrow 3y agoOh yes, thousands of murders a year dismissed by clever lawyers! Thousands!
- shadowgovt 3y agoThe scarier possibility is "thousands of wrongful convictions due to ineffective representation." ... scarier and hard to discount.
- chii 3y agoAnd it is imho better for 9 murderers to walk free and prevent 1 wrongful conviction, than to have 1 wrongful conviction to keep 9 murderers off the streets.
- talldatethrow 3y ago
- HWR_14 3y ago"We used facial recognition to identify people to arrest" is a statement that belongs in dystopian sci-fi, not reality. False positives, while the worst part, aren't the only issue.
- sharts 3y agoPolicy makers have very particular intentions when deploying these things.
- archo 3y agohttps://archive.is/p204S https://archive.is/p204S
- wiseowise 3y ago[ragebait]
- tbrownaw 3y ago> The research also demonstrated that Black people are overrepresented in databases of mugshots, and that skews AI. > "Consequently AI is more likely to mark Black faces as criminal, leading to the targeting and arresting of innocent Black people," they wrote. Um, no. This is not a system that turns a picture into a criminality score. This is a system that takes one picture as input and tries to search its database for other pictures of the same person.