22 ms·

Facial Recognition Leads To False Arrest Of Black Man In Detroit

661 points by vermontdevil 6y ago

VWWHFSfQ 6y ago
sounds like this guy is about to get a big payday.
- avs733 6y ago
  at tax payer expense.
  - dfxm12 6y ago
    It must be frustrating to be a taxpayer, demanding to defund the police while simultaneously having to pay for their mistakes. If only their elected officials would listen to them...
    - avs733 6y ago
      For starters, you would think that * Revenue from tickets/fines/etc. shouldn't go to into police pockets or it just incentives rent seeking behavior by people with law enforcement powers * Settlements should come out of their budget not city insurance.
- ncallaway 6y ago
  That and we might get some kind of judicial ruling that current incarnations of facial recognition software are racially based. It would be a great result if a court declared that the use of racially biased facial recognition software is a violation of the 14th amendment violation, and enjoined PDs from using such software unless it can be demonstrated to be free of racial bias.
  - ChrisMarshallNY 6y ago
    Might have something to do with the...monochromatic...makeup of most software company C-suites. Fortunately, that is changing, but not that quickly.
  - raxxorrax 6y ago
    Facial recognition is a solution to a problem we don't have. It is the smartwatch of ML.
- whatshisface 6y ago
  Will he? I thought it was pretty hard to win cases against the police. What does the evidence about the practicality of pursuing police for torts say here, and for that matter what kind of evidence should we be looking for?
  - VWWHFSfQ 6y ago
    most cases are settled by the city/county before it goes to court
  - dfxm12 6y ago
    One can file a civil case for false arrest. However, thanks to the qualified immunity cops enjoy coupled with the newness of the facial recognition technology, it will be very hard to prove that they obviously violated clearly established law. Maybe the cops will even point their fingers at the software. Maybe he can go after the makers of the facial recognition software, but they can probably point their finger at the cops for using it wrong. So, in any case, the guy will be left with a big legal bill.
danso 6y ago
Since the NPR is a 3 minute listen without a transcript, here's the ACLU's text/image article: https://www.aclu.org/news/privacy-technology/wrongfully-arrested-because-face-recognition-cant-tell-black-people-apart/ https://www.aclu.org/news/privacy-technology/wrongfully-arre... And here's a 1st-person account from the arrested man: https://www.washingtonpost.com/opinions/2020/06/24/i-was-wrongfully-arrested-because-facial-recognition-why-are-police-allowed-use-this-technology/ https://www.washingtonpost.com/opinions/2020/06/24/i-was-wro...
- TedDoesntTalk 6y ago
  As soon as I saw it was audio only, i left the site. Why do sites do this? How many people actually stick to the page and listen to that?
  - asveikau 6y ago
    > How many people actually stick to the page and listen to that? I just did. 3 minutes wasn't that bad and I wasn't somewhere where it would be a problem. > Why do sites do this? NPR is a radio network. I have seen that often they do transcribe their clips. I am not sure what the process they have for that looks like, but it seems this particular clip didn't get transcribed. Edit: looks like they do have a transcription mentioned elsewhere in the thread. So seems like some kind of UI fail.
  - danso 6y ago
    NPR does transcribe (many, most?) its audio stories, but usually there's a delay of a day or so – the published timestamp for this story is 5:06AM (ET) today. edit: looks like there's a text version of the article. I'm assuming this is a CMS issue: there's an audio story and a "print story", but the former hadn't been linked to the latter: https://news.ycombinator.com/item?id=23628790 https://news.ycombinator.com/item?id=23628790
    - dhosek 6y ago
      They transcribe all their stories. Back before the web was widespread, you could call or write NPR and have them mail a transcript to you.
ChrisMarshallNY 6y ago
Sadly, there's plenty more where that came from.
hpoe 6y ago
I don't think using the facial recognition is necessarily wrong to help identify probable suspects, but arresting someone based on a facial match algorithm is definitely going too far. Of course really I blame the AI/ML hucksters for part of this mess who have sold us the idea of machines replacing rather than augmenting human decision making.
- jordanpg 6y ago
  Those hucksters should be worried about the Supreme Court swatting away their business model, because that's where I see this headed.
  - hnlmorg 6y ago
    I don't think they'll worry about that. Even if that did happen there are foreign markets who would still invest in this.
- jacquesm 6y ago
  I think it is very wrong. Faces are anything but unique. Having a particular face should not result in you being a suspect. Only once actual policing results in you becoming a suspect then this might be a low quality extra signal.
  - gridlockd 6y ago
    > Having a particular face should not result in you being a suspect. Only once actual policing results in you becoming a suspect then this might be a low quality extra signal. Having a picture or just a description of the face is one of the most important pieces of information the police has in order to do actual policing. You can be arrested for just broadly matching the description if you happen to be in the vicinity. Had the guy been convicted of anything just based on that evidence, this would be a scandal. As it is, a suspect is just a suspect and this kind of thing happens all the time, because humans are just as fallible. It's just not news when there's no AI involved.
    - 6y ago
gentleman11 6y ago
The discussion about this tech revolves around accuracy and racism, but the real threat is in global unlimited surveillance. China is installing 200 million of facial recognition cameras right now to keep the population under control. It might be the death of human freedom as this technology spreads Edit: one source says it is 400 million new cameras: https://www.cbc.ca/passionateeye/m_features/in-xinjiang-china-surveillance-technology-is-used-to-help-the-state-control https://www.cbc.ca/passionateeye/m_features/in-xinjiang-chin...
whatshisface 6y ago
How does computerized facial recognition compare in terms of racial bias and accuracy to human-brain facial recognition? Police are not exactly perfect in either regard.
- suizi 6y ago
  Face recognition widens the scope of how many people can be harassed.
  - _underfl0w_ 6y ago
    While also enabling finger-pointing, e.g. the police can say "We aren't racist or aren't at fault. The system is just faulty." while the engineers behind the facial recognition tech can say that they, "Were just doing their job. The police should've heeded their disclaimers, etc."
danso 6y ago
This story is really alarming because as described, the police ran a face recognition tool based on a frame of grainy security footage and got a positive hit. Does this tool give any indication of a confidence value? Does it return a list (sorted by confidence) of possible suspects, or any other kind of feedback that would indicate even to a layperson how much uncertainty there is? The issue of face recognition algorithms performing worse on dark faces is a major problem. But the other side of it is: would police be more hesitant to act on such fuzzy evidence if the top match appeared to be a middle-class Caucasian (i.e. someone who is more likely to take legal recourse)?
- Pxtl 6y ago
  Intresting and related, a team made a neat "face depixelizer" that takes a pixelated image and uses machine learning to generate a face that should match the pixelated image. What's hilarious is that it makes faces that look nothing like the original high-resolution images. https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801
  - mywittyname 6y ago
    I wonder if this is trained on the same, or similar, datasets.
    - jcims 6y ago
      One of the underlying models, PULSE, was trained on CelebAHQ, which is likely what the results are mostly white-looking. StyleGAN, which was trained on the much more diverse (but sparse) FFHQ dataset does come up with a much more diverse set of faces[1]...but PULSE couldn't get them to converge very closely on the pixelated subjects...so they went with CelebA [2]. [1] https://github.com/NVlabs/stylegan https://github.com/NVlabs/stylegan [2] https://arxiv.org/pdf/2003.03808.pdf https://arxiv.org/pdf/2003.03808.pdf (ctrl+f ffhq)
  - jacquesm 6y ago
    That should be called a face generator, not a depixelizer.
MikusR 6y ago
Is that different from somebody getting arrested based on mistaken eyewitness.
- Enginerrrd 6y ago
  The difference is that is a known problem, but with ML, a large fraction of the population thinks it's infallible. Worse, its reported confidence for an individual face may be grossly overstated, since that is based on all the data it was trained on, rather than the particular subset you may be dealing with.
  - raxxorrax 6y ago
    large fraction of the population and ML marketing both believe that. I still think it insane. We have falling crime rates and we still arm ourselves as fast as we can. Humanity could live without face recognition and we wouldn't even suffer any penalties. Nope, people need to sell their evidently shitty ML work.
    - treis 6y ago
      (1) We still have extreme levels of crime compared to other first world countries even if it is in decline (2) Your argument strikes me as somewhat similar to "I feel fine why should I keep taking my medicine?". It's not exactly the same as the medicine is scientifically proven to cure disease while it's impossible to measure the impact of police on crime. But "things are getting better so we should change what we're doing" is not a particularly sound logical argument.
      - raxxorrax 6y ago
        Crimes rates dropped even faster in countries with more rehabilitative approaches and long before some countries began to upgrade their police forces because of unrelated fears. It was more about giving people a second chance in all that. Criminologists aren't certain about surveillance having a positive or negative effects on crime. We have more than 40 studies with mixed results. What is certain with that this kind of surveillance isn't responsible for the falling crime rates described. Most data is from the UK. Currently I don't think countries without surveillance fair worse on crime. Maybe quite to the contrary. "what we're doing" is not equivalent to increasing video surveillance or generally increasing armament in civil spaces. It may be sound logic if you extend the benefit of the doubt but it may also just be a false statement. Since surveillance is actually constitutionally forbidden in many countries, on could argue that deployment would "increase crime". In some other sound logic it might just be a self reinforcing private prison industry with economic interests to keep a steady supply of criminals. Would also be completely sound. But all these discussions are quite dishonest, don't you think? I just don't want your fucking camera in my face.
rusty__ 6y ago
any defence lawyer with more than 3 brain cells would have an absolute field day deconstructing a case brought solely on the basis of a facial recognition. What happened to the idea that police need to gather a variety of evidence confirming their suspicions before an arrest is issued. Even a state prosecutor wouldn't authorize a warrant based on such flimsy methods.
- ARandomerDude 6y ago
  True but the defendant is still financially, and in many cases professionally, ruined.
- failuser 6y ago
  Getting a layer that would advise beyond pleading guilty for a reduced sentence is not a default option.
zro 6y ago
NPR article about the same, if you prefer to read instead of listen: https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig https://www.npr.org/2020/06/24/882683463/the-computer-got-it... I'll be watching this case with great interest
crazygringo 6y ago
In this particular case, computerized facial recognition is not the problem. Facial recognition produces potential matches. It's still up to humans to look at footage themselves and use their judgment as to whether it's actually the same person or not, as well as to judge whether other elements fit the suspect or not. The problem here is 100% on the cop(s) who made that call for themselves, or intentionally ignored obvious differences. (Of course, without us seeing the actual images in question, it's hard to judge.) There are plenty of dangers with facial recognition (like using it at scale, or to track people without accountability), but this one doesn't seem to be it.
- ncallaway 6y ago
  > The problem here is 100% on the cop(s) who made that call for themselves I disagree. There is plenty of blame on the cops who made that call for themselves, true. But there doesn't have to be a single party who is at fault. The facial recognition software is badly flawed in this dimension. It's well established that the current technologies are racially biased. So there's at least some fault in the developer of that technology, and the purchasing officer at the police department, and a criminal justice system that allows it to be used that way. Reducing a complex problem to a single at-fault person produces an analysis that will often let other issues continue to fester. Consider if the FAA always stopped the analysis of air-crashes at: "the pilot made an error, so we won't take any other corrective actions other than punishing the pilot". Air travel wouldn't nearly as safe as it is today. While we should hold these officers responsible for their mistake (abolish QI so that these officers could be sued civilly for the wrongful arrest!), we should also fix the other parts of the system that are obviously broken.
  - dfxm12 6y ago
    The facial recognition software is badly flawed in this dimension. It's well established that the current technologies are racially biased. Who decided to use this software for this purpose, despite these bad flaws and well established bias? The buck stops with the cops.
    - 6y ago
vermontdevil 6y ago
From ACLU article: Third, Robert’s arrest demonstrates why claims that face recognition isn’t dangerous are far-removed from reality. Law enforcement has claimed that face recognition technology is only used as an investigative lead and not as the sole basis for arrest. But once the technology falsely identified Robert, there was no real investigation. I fear this is going to be the norm among police investigations.
vmception 6y ago
> Federal studies have shown that facial-recognition systems misidentify Asian and black people up to 100 times more often than white people. The idea behind inclusion is that this product would have never made it to production if the engineering teams, product team, executive team and board members represented the population. But enough representation so that there is a countering voice is even better. Would have just been "this edge case is not an edge case at all, axe it." Accurately addressing a market is the point of the corporation more than an illusion of meritocracy amongst the employees.
- JangoSteve 6y ago
  This is so incredibly common, it's embarrassing. I was on an expert panel about "AI and Machine Learning in Healthcare and Life Sciences" back in January, and I made it a point throughout my discussions to keep emphasizing the amount of bias inherent in our current systems, which ends up getting amplified and codified in machine learning systems. Worse yet, it ends up justifying the bias based on the false pretense that the systems built are objective and the data doesn't lie. Afterward, a couple people asked me to put together a list of the examples I cited in my talk. I'll be adding this to my list of examples: * A hospital AI algorithm discriminating against black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 * The dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abstract/2688587 https://jamanetwork.com/journals/jamadermatology/article-abs... And some other relevant, but not healthcare examples as well: * When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/ https://fortune.com/2019/08/16/google-jigsaw-perspective-rac... * When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio... * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though: * A hospital used AI to enable care and cut costs (though the reporting seems to over simplify and gloss over enough to make the actual analysis of the results a little suspect). https://www.healthcareitnews.com/news/flagler-hospital-uses-ai-create-clinical-pathways-enhance-care-and-slash-costs https://www.healthcareitnews.com/news/flagler-hospital-uses-...
at_a_remove 6y ago
I think that your prints, DNA, and so forth must be, in the interests of fairness, utterly erased from all systems in the case of false arrest. With some kind of enormous, ruinous financial penalty in place for the organizations for non-compliance, as well as automatic jail times for involved personnel. These things need teeth to happen.
ghostpepper 6y ago
He wasn't arrested until the shop owner had also "identified" him. The cops used a single frame of grainy video to pull his driver's license photo, and then put that photo in a lineup and showed the store clerk. The store clerk (who hadn't witnessed the crime and was going off the same frame of video fed into the facial recognition software) said the driver's license photo was a match. There are several problems with the conduct of the police in this story but IMHO the use of facial recognition is not the most egregious.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- bsenftner 6y ago
  Yes, this is a story of police misconduct. The regulation of facial recognition that is required is regulation against police/authority stupidity. The FR system aids in throwing away misses, leaving investigative leads. But if a criminal is not in the FR database to begin with, any results of the FR are wastes of time.
- businesslucas 6y ago
  It is not clear to me that the person who identified him was shop owner or clerk. From the nyt article: https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html https://www.nytimes.com/2020/06/24/technology/facial-recogni... "The Shinola shoplifting occurred in October 2018. Katherine Johnston, an investigator at Mackinac Partners, a loss prevention firm, reviewed the store’s surveillance video and sent a copy to the Detroit police" "In this case, however, according to the Detroit police report, investigators simply included Mr. Williams’s picture in a “6-pack photo lineup” they created and showed to Ms. Johnston, Shinola’s loss-prevention contractor, and she identified him. (Ms. Johnston declined to comment.)"
  - ghostpepper 6y ago
    I think you're correct that the person was not an owner or clerk. IMHO the salient point is that the person was not any sort of eyewitness but merely comparing the same grainy photo as the algorithm.
sneak 6y ago
Another reason that it's absolutely insane that the state demands to know where you sleep at night in a free society. These clowns were able to just show up at his house and kidnap him. The practice of disclosing one's residence address to the state (for sale to data brokers[1] and accessible by stalkers and the like) when these kinds of abuses are happening is something that needs to stop. There's absolutely no reason that an ID should be gated on the state knowing your residence. It's none of their business. (It's not on a passport. Why is it on a driver's license?) [1]: https://www.newsweek.com/dmv-drivers-license-data-database-integrity-department-motor-vehicles-1458141 https://www.newsweek.com/dmv-drivers-license-data-database-i...
deleted 6y ago
[deleted]
TedDoesntTalk 6y ago
> Even if this technology does become accurate (at the expense of people like me), I don’t want my daughters’ faces to be part of some government database. Stop using Amazon Ring and similar doorbell products.
FpUser 6y ago
And then in some states employers are allowed to ask have you eve been arrested (never mind convicted of any crime) on employment application. Sure, keep putting people down. One day it might catch up with China's social scoring policies.
paulorlando 6y ago
I've been thinking this sort of event has become inevitable. Tech development and business models support extending the environments in which we collect images and analyze them. Confidence values lead to statistical guilt. I wrote about it here if interested: https://unintendedconsequenc.es/inevitable-surveillance/ https://unintendedconsequenc.es/inevitable-surveillance/
mistercool 6y ago
relevant: https://www.theregister.com/2020/06/24/face_criminal_ai/ https://www.theregister.com/2020/06/24/face_criminal_ai/
linuxftw 6y ago
Wait until you hear about how garbage and unscientific fingerprint identification is.
- _underfl0w_ 6y ago
  Speaking of pseudoscience, didn't most police forces just start phasing out polygraphs in the last decade?
  - linuxftw 6y ago
    Unlikely unless they were compelled by law or found something else to replace it, and I think it's the latter. Something about machine learning and such.
js2 6y ago
> "I picked it up and held it to my face and told him, 'I hope you don't think all Black people look alike,' " Williams said. I'm white. I grew up around a sea of white faces. Often when watching a movie filled with a cast of non-white faces, I will have trouble distinguishing one actor from another, especially if they are dressed similarly. This sometimes happens in movies with faces similar to the kinds I grew up surrounded by, but less so. So unfortunately, yes, I probably do have more trouble distinguishing one black face from another vs one white face from another. This is known as the cross-race effect and it's only something I became aware of in the last 5-10 years. Add to that the fallibility of human memory, and I can't believe we still even use line ups. Are there any studies about how often line ups identify the wrong person? https://en.wikipedia.org/wiki/Cross-race_effect https://en.wikipedia.org/wiki/Cross-race_effect
- SauciestGNU 6y ago
  I lived in South Africa for a while and heard many times, with various degrees of irony, "you white people all look the same" from black South Africans. So yeah it's definitely a cross-racial recognition problem, and it's probably also a problem with distinguishing between members of visible minorities using traits beyond the most noticable othering characteristic.
Anthony-G 6y ago
There is just so much wrong with this story. For starters: The shoplifting incident occurred in October 2018 but it wasn’t until March 2019 that the police uploaded the security camera images to the state image-recognition system but the police still waited until the following January to arrest Williams. Unless there was something special about that date in October, there is no way for anyone to remember what they might have been doing on a particular day 15 months previously. Though, as it turns out, the NPR report states that the police did not even try to ascertain whether or not he had an alibi. Also, after 15 months, there is virtually no chance that any eye-witness (such as the security guard who picked Williams out of a line-up) would be able to recall what the suspect looked like with any degree of certainty or accuracy. This WUSF article [1] includes a photo of the actual “Investigative Lead Report” and the original image is far too dark for a anyone (human or algorithm) to recognise the person. It’s possible that the original is better quality and better detail can be discerned by applying image-processing filters – but it still looks like a very noisy source. That same “Investigative Lead Report” also clearly states that “This document is not a positive identification … and is not probable cause to arrest. Further investigation is needed to develop probable cause of arrest”. The New York Times article [2] states that this facial recognition technology that the Michigan tax-payer has paid millions of dollars for is known to be biased and that the vendors do “not formally measure the systems’ accuracy or bias”. Finally, the original NPR article states that > "Most of the time, people who are arrested using face recognition are not told face recognition was used to arrest them," said Jameson Spivack [1] https://www.wusf.org/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michigan/ https://www.wusf.org/the-computer-got-it-wrong-how-facial-re... [2] https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html https://www.nytimes.com/2020/06/24/technology/facial-recogni...
- cwkoss 6y ago
  Really seems like most police departments in our country are incompetent, negligent, ineffective and systemically racist. Many of these cops are earning $200k plus annually! Our law enforcement system is ridiculous and needs an overhaul.
  - 6y ago
mnw21cam 6y ago
This is a classic example of the false positive rate fallacy. Let's say that there are a million people, and the police have photos of 100,000 of them. A crime is committed, and they pull the surveillance of it, and match against their database. They have a funky image matching system that has a false positive rate of 1 in 100,000 people, which is way more accurate than I think facial recognition systems are right now, but let's just roll with it. Of course, on average, this system will produce one positive hit per search. So, the police roll up to that person's home and arrest them. Then, in court, they get to argue that their system has a 1 in 100,000 false positive rate, so there is a chance of 1 in 100,000 that this person is innocent. Wrong! There are ten people in the population of 1 million that the software would comfortably produce a positive hit for. They can't all be the culprit. The chance isn't 1 in 100,000 that the person is innocent - it is in fact at least 9 out of 10 that they are innocent. This person just happens to be the one person out of the ten that would match that had the bad luck to be stored in the police database. Nothing more.
- Buttons840 6y ago
  There's a good book called "The Drunkards Walk", that describes a woman who was jailed after having 2 children die from SIDS. They argued that the odds of this happening is 1 in a million (or something like that), so probably the woman is a baby killer. The prosecution had statisticians argue this. The woman was found guilty. She later won on appeal in part because the defense showed that the testimony and argument of the original statisticians were wrong. This stuff is so easy to get wrong. A little knowledge of statistics can be dangerous.
  - Polylactic_acid 6y ago
    And even if the original stats were right. A 1 in a million event happens to about 100 people per day in the US.
    - microcolonel 6y ago
      > A 1 in a million event happens to about 100 people per day in the US. This is a meaningless statement, you could choose literally any number for this statement, because you are missing the denominator.
seebetter 6y ago
Reminds me of this- Facial recognition technology flagged 26 California lawmakers as criminals. (August 2019) https://www.mercurynews.com/2019/08/14/facial-recognition-technology-flagged-26-california-lawmakers-as-criminals-this-bill-to-ban-the-tech-is-headed-to-the-senate/ https://www.mercurynews.com/2019/08/14/facial-recognition-te...
throwawaysea 6y ago
A human still confirmed the match right? That makes this not a facial recognition issue but something else.
- aaronmdjones 6y ago
  A human who had only seen the same grainy security footage that the algorithm saw.
tantalor 6y ago
No mention of whether a judge signed a warrant for the arrest. In what world can cops just show up and arrest you on your front lawn based on their hunch?
cpeterso 6y ago
What is a unique use case for facial recognition that cannot be abused and has no other alternative solution? Even the "good" use cases like unlocking your phone have security problems because malicious people can use photos or videos of your face and you can't change your face like you would a breached username and password.
aritraghosh007 6y ago
The pandemic has accelerated the use of no-touch surfaces specially at places like an airport that are more inclined to now use a face recognition security kiosk. What's not clear is the vetting process for these (albeit controversial) technologies. What if Google thinks person A is an offender but Amazon thinks otherwise. Can they be used as counter evidence? What is the gold standard for surveillance?
jandrewrogers 6y ago
It isn't just facial recognition, license plate readers can have the same indefensibly Kafka-esque outcomes where no one is held accountable for verifying computer-generated "evidence". Systems like in the article make it so cheap for the government to make a mistake, since there are few consequences, that they simply accept mistakes as a cost of doing business. Someone I know received vehicular fines from San Francisco on an almost weekly basis solely from license plate reader hits. The documentary evidence sent with the fines clearly showed her car had been misidentified but no one ever bothered to check. She was forced to fight each and every fine because they come with a presumption of guilt, but as soon as she cleared one they would send her a new one. The experience became extremely upsetting for her, the entire bureaucracy simply didn't care. It took threats of legal action against the city for them to set a flag that apparently causes violations attributed to her car to be manually reviewed. The city itself claimed the system was only 80-90% accurate, but they didn't believe that to be a problem.
- black_puppydog 6y ago
  I agree that's bad, and license plate readers come with their own set of problems. But being biased by the skin color of the driver is (AFAIK) not one of them. Which is exactly the problem with vision systems applied to humans, at least the ones we've seen deployed so far. If a system discriminates against a specific population, that's very different from (indiscriminately) being unreliable.
- vmception 6y ago
  That suddenly reminded me why I feel so privileged to not own a car, a distinct contrast from when I was a teenager and felt it was a rite of passage! I had forgotten about the routine of fighting traffic tickets multiple times a year as a fact of life. Let alone fender benders. I had only been reveling in the lack of a frustrating commute. Last decade I did get a car for 3 months, and the insurance company was so thrilled that I was "such a good driver" because of my "spotless record" for many years. Little do they know I just don't drive and perhaps have now less experience than others. Although tangentially, their risk matrix actually might be correct, if I can afford to live in dense desirable areas then maybe it is less likely that I would be going fast and getting into circumstances that pull from their insurance pool at larger amounts. They probably thought "one of the largest companies in the world probably chauffeurs him down the highway in a bus anyway"
neonate 6y ago
The prosecutor and the police chief should personally apologize to his daughters, assuming that would be age appropriate.
datavirtue 6y ago
All that rigamaroll for $3800 worth of crap? They should just switch it up and start entrapping people like the FBI does. Then at least they would have perhaps one leg to stand on.
joyj2nd 6y ago
Understandable, all black men look the same.
bosswipe 6y ago
Boston just banned facial recognition, as have San Francisco, Oakland and a bunch of other cities. You can join this movement by urging your local government officials to follow suit.
renewiltord 6y ago
I've got to be honest: I'm getting the picture the police here aren't very competent. I know I know, POSIWID and maybe they're very competently aiming at the current outcome. But don't they just look like a bunch of idiots?
ibudiallo 6y ago
Here is a part that I personally have to wrestle with: > "They never even asked him any questions before arresting him. They never asked him if he had an alibi. They never asked if he had a red Cardinals hat. They never asked him where he was that day," said lawyer Phil Mayor with the ACLU of Michigan. When I was fired by an automated system, no one asked if I had done something wrong. They asked me to leave. If they had just checked his alibi, he would have been cleared. But the machine said it was him, so case closed. Not too long ago, I wrote a comment here about this [1]: > The trouble is not that the AI can be wrong, it's that we will rely on its answers to make decisions. > When the facial recognition software combines your facial expression and your name, while you are walking under the bridge late at night, in an unfamiliar neighborhood, and you are black; your terrorist score is at 52%. A police car is dispatched. Most of us here can be excited about Facial Recognition technology but still know that it's not something to be deployed in the field. It's by no means ready. We might even consider the moral ethics before building it as a toy. But that's not how it is being sold to law enforcement or other entities. It's _Reduce crime in your cities. Catch criminals in ways never thought possible. Catch terrorists before they blow up anything._ It is sold as an ultimate decision maker. [1]:https://news.ycombinator.com/item?id=21339530 https://news.ycombinator.com/item?id=21339530
- nimbius 6y ago
  This shit right here. This is why I don't stop for the inventory control alarms at department store doorways if they go off. I know I've paid, and little sirens are just a nuisance at this point. This is why I've never stopped for receipt checks, because it's my receipt, and I've paid. The security theatre is just bad comedy. Just because the machine says I've done a no no, doesn't mean I can't come back and win a lawsuit later. It doesn't abdicate cops from their jobs. I have a winning complexion, so I'll never enjoy a false positive, but if I do, I'll make sure it bankrupts whatever startup incubator garbage decided to shill a replacement for real law enforcement.
  - demadog 6y ago
    So out of curiosity, you just roll out of Costco with a cart full of food and gear while the receipt checker tries to stop you?
deleted 6y ago
[deleted]
jackklika 6y ago
The company that developed this software is Dataworks plus, according to the article. Name and shame.
redorb 6y ago
If it's statistically proven to not work with black people then I think the only options are 1) Make it avoid black people, i.e. they aren't stored in the database and aren't processed when scanned. 2) Put a 5 year hiatus on commercial / public use. Either of these things are more acceptable than too many false positives. #1 is really interesting to me as a thought experiment because it makes everyone think twice.
- fatso784 6y ago
  The issue is that it's hard to determine who is considered "black" and who is not since race is a made-up fiction constructed by white supremacists that is now so engrained in American society we find it difficult if not impossible to reflect on it. Are we using the Fitzpatrick skin color scale? Are we continuing the hypodescent rule (an algorithm, btw)? Maybe we just outlaw face recognition in criminal justice entirely.
  - redorb 6y ago
    It definitely shouldn't be allowed as the main evidence - perhaps the answer to skin tone is to use the data and see where the algo starts to get too many false positives.. The largest - over arching idea - is to get everyone to think twice by making the majority think twice - If white people think 'its only for us!?' it'll make them really study the effects.. (I'm white.)
ineedasername 6y ago
It's supposed to be a cornerstone of "innocent until proven guilty" legal systems that it is better to have 10 guilty people go free than to deprive a single innocent person of their freedom. It seems like the needle has been moving in the wrong direction on that. I'm not sure it that's just my impression on things, or if it's because there's more awareness with the internet/social networking of issues...
atum47 6y ago
well, I'm going to take something out of my chest. every time I shared a project here using machine learning people always gave me crap. saying my models were simplistic, or I did something wrong or the solution didn't work 100% of the time. well, I studied ML back in college. the basics, the algorithms that started all, linear regression, perceptron, adaline, knn, kmeans... and guess what? ML doesn't work 100% of the time. I always wanted to see how people would react when a car driven by ml hits something or when they based an important decision based on the classification of an nn. ML should be used along side human intelligence not by itself. you don't blindly trusts a black box.
aussieguy1234 6y ago
In alot of police departments around the world, the photo database used is the drivers license database. There is clothing available that can confuse facial recognition systems. What would happen if, next time you go for your drivers license photo, you wore a T shirt designed to confuse facial recognition, for example like this one? https://www.redbubble.com/i/t-shirt/Anti-Surveillance-Clothing-by-Naamiko/24714049.1YYVU?u https://www.redbubble.com/i/t-shirt/Anti-Surveillance-Clothi...
nebulous1 6y ago
> In Williams' case, police had asked the store security guard, who had not witnessed the robbery, to pick the suspect out of a photo lineup based on the footage, and the security guard selected Williams. Great job police
czbond 6y ago
A few things I just don't have the stomach for as an engineer, writing software that: - impacts someones health - impacts someones finances - impacts someones freedoms Call me weak, but I think about the "what ifs" a bit too much in those cases. What if my bug keeps them from selling their stock and they lose their savings? What if the wrong person is arrested, etc?
- Barrin92 6y ago
  Why would anyone call you weak, that's principled and it's the correct attitude. It's the people who don't think about the consequences of the products they help build that are the problem.
  - czbond 6y ago
    Thank you for that really helpful reply. I re-framed my thoughts based on it.
alex_young 6y ago
This technology will never be ready to use like this. Similarly we shouldn’t collect vast databases of fingerprints or DNA and search them for every crime. Why? Because error rates are unavoidable. There is some uncertainty, and in large enough numbers you will find false matches with perfect DNA matching. We must keep our senses and use these technologies to help us rather than find the hidden bad guy.
kwonkicker 6y ago
Due process should not be abandoned in favour of automation. This was police negligence as much as it was a software mismatch. One more thing, the article was being to dramatic about the whole incident.
w_t_payne 6y ago
Perhaps we, as technologists, are going about this the wrong way. Maybe, instead of trying to reduce the false alarm rate to an arbitrarily low number, we instead develop CFAR (Constant false alarm rate) systems, so that users of the system know that they will get some false alarms, and develop procedures for responding appropriately. In that way, we could get the benefit of the technology, whilst also ensuring that the system as a whole (man and machine together) are designed to be robust and have appropriate checks and balances.
- gwd 6y ago
  If you follow the link, you'll see that the computer report had this message right at the top in massive letters: THIS DOCUMENT IS NOT A POSITIVE IDENTIFICATION. IT IS AN INVESTIGATIVE LEAD ONLY AND IS _NOT_ PROBABLE CAUSE TO ARREST. FURTHER INVESTIGATION IS NEEDED TO DEVELOP PROBABLE CAUSE TO ARREST. I mean, what else could the technologists have done?
  - Mangalor 6y ago
    I wonder if the problem here is just the way traditional policing works, not even the technology.
    - noisy_boy 6y ago
      It is indeed the way traditional policing works. The average police officer things facial recognition is the visual equivalent of Google and since he/she can similarly rely on search results - they either have no idea about false positive biases based on race or worse, probably are too lazy to dig deeper. Though the suffering of the victims of such wrong matches is real, one consolation is that more of such cases will hopefully bring about the much needed scepticism in the results so that some old-fashioned validation/investigation is done.
  - w_t_payne 6y ago
    Well .. it looks like it isn't enough, and additional human-factors oriented requirements need to be documented - like deliberately engineering the system to produce a certain number of false alarms so the police always have to do additional work to discard them - as opposed to only sometimes.
MertsA 6y ago
I would love to see police trying to take a crack at this from the other side of things. Instead of matching against a database, Set up a style GAN and come up with a mask of the original photo or video to isolate just the face and have the discriminator try to match the face. Then at the end you can see the generated face with a decent pose and more importantly look through the range of generated faces that result in a reasonable match to give a somewhat decent idea of how confident you should be about any identification.
loup-vaillant 6y ago
And now the poor guy has an arrest record. Which wouldn't be a problem in reasonable legislations, where it's nobody's business whether you've been arrested or not, as long as you've not been convicted. But in the US, I've heard that it can make it harder to get a job. I believe I'm starting to get a feel for how the school to prison pipeline may work.
hkai 6y ago
While this case is bad enough, mistakes like this are not the biggest concern. Mistakenly arrested people are (hopefully) eventually released, even though they have to go through quite a bit of trouble. The consequence that is much worse would be mass incarceration of certain groups, because the AI is too good at catching people who actually did something. This second wave of mass incarceration will lead to even more single parent families and poor households, and will reinforce the current situation.
d--b 6y ago
The worst part is they use facial recognition which finds someone that looks like the suspect, and then they put the guy in a lineup and have him identified by the victim. Wtf?
- jl2718 6y ago
  That’s a good point. The selection from a lineup then gives less information. So the real issue is poorly understood priors. I’m pretty sure that this can be used fairly with a rigorous Bayesian treatment.
blackrock 6y ago
TOTAL FAIL.

661 points by vermontdevil 6y ago

VWWHFSfQ 6y ago

sounds like this guy is about to get a big payday.

avs733 6y ago
at tax payer expense.
- dfxm12 6y ago
  It must be frustrating to be a taxpayer, demanding to defund the police while simultaneously having to pay for their mistakes. If only their elected officials would listen to them...
  - avs733 6y ago
    For starters, you would think that * Revenue from tickets/fines/etc. shouldn't go to into police pockets or it just incentives rent seeking behavior by people with law enforcement powers * Settlements should come out of their budget not city insurance.
ncallaway 6y ago
That and we might get some kind of judicial ruling that current incarnations of facial recognition software are racially based. It would be a great result if a court declared that the use of racially biased facial recognition software is a violation of the 14th amendment violation, and enjoined PDs from using such software unless it can be demonstrated to be free of racial bias.
- ChrisMarshallNY 6y ago
  Might have something to do with the...monochromatic...makeup of most software company C-suites. Fortunately, that is changing, but not that quickly.
- raxxorrax 6y ago
  Facial recognition is a solution to a problem we don't have. It is the smartwatch of ML.
whatshisface 6y ago
Will he? I thought it was pretty hard to win cases against the police. What does the evidence about the practicality of pursuing police for torts say here, and for that matter what kind of evidence should we be looking for?
- VWWHFSfQ 6y ago
  most cases are settled by the city/county before it goes to court
- dfxm12 6y ago
  One can file a civil case for false arrest. However, thanks to the qualified immunity cops enjoy coupled with the newness of the facial recognition technology, it will be very hard to prove that they obviously violated clearly established law. Maybe the cops will even point their fingers at the software. Maybe he can go after the makers of the facial recognition software, but they can probably point their finger at the cops for using it wrong. So, in any case, the guy will be left with a big legal bill.

danso 6y ago

Since the NPR is a 3 minute listen without a transcript, here's the ACLU's text/image article: https://www.aclu.org/news/privacy-technology/wrongfully-arrested-because-face-recognition-cant-tell-black-people-apart/ https://www.aclu.org/news/privacy-technology/wrongfully-arre... And here's a 1st-person account from the arrested man: https://www.washingtonpost.com/opinions/2020/06/24/i-was-wrongfully-arrested-because-facial-recognition-why-are-police-allowed-use-this-technology/ https://www.washingtonpost.com/opinions/2020/06/24/i-was-wro...

TedDoesntTalk 6y ago
As soon as I saw it was audio only, i left the site. Why do sites do this? How many people actually stick to the page and listen to that?
- asveikau 6y ago
  > How many people actually stick to the page and listen to that? I just did. 3 minutes wasn't that bad and I wasn't somewhere where it would be a problem. > Why do sites do this? NPR is a radio network. I have seen that often they do transcribe their clips. I am not sure what the process they have for that looks like, but it seems this particular clip didn't get transcribed. Edit: looks like they do have a transcription mentioned elsewhere in the thread. So seems like some kind of UI fail.
- danso 6y ago
  NPR does transcribe (many, most?) its audio stories, but usually there's a delay of a day or so – the published timestamp for this story is 5:06AM (ET) today. edit: looks like there's a text version of the article. I'm assuming this is a CMS issue: there's an audio story and a "print story", but the former hadn't been linked to the latter: https://news.ycombinator.com/item?id=23628790 https://news.ycombinator.com/item?id=23628790
  - dhosek 6y ago
    They transcribe all their stories. Back before the web was widespread, you could call or write NPR and have them mail a transcript to you.

ChrisMarshallNY 6y ago

Sadly, there's plenty more where that came from.

hpoe 6y ago

I don't think using the facial recognition is necessarily wrong to help identify probable suspects, but arresting someone based on a facial match algorithm is definitely going too far. Of course really I blame the AI/ML hucksters for part of this mess who have sold us the idea of machines replacing rather than augmenting human decision making.

jordanpg 6y ago
Those hucksters should be worried about the Supreme Court swatting away their business model, because that's where I see this headed.
- hnlmorg 6y ago
  I don't think they'll worry about that. Even if that did happen there are foreign markets who would still invest in this.
jacquesm 6y ago
I think it is very wrong. Faces are anything but unique. Having a particular face should not result in you being a suspect. Only once actual policing results in you becoming a suspect then this might be a low quality extra signal.
- gridlockd 6y ago
  > Having a particular face should not result in you being a suspect. Only once actual policing results in you becoming a suspect then this might be a low quality extra signal. Having a picture or just a description of the face is one of the most important pieces of information the police has in order to do actual policing. You can be arrested for just broadly matching the description if you happen to be in the vicinity. Had the guy been convicted of anything just based on that evidence, this would be a scandal. As it is, a suspect is just a suspect and this kind of thing happens all the time, because humans are just as fallible. It's just not news when there's no AI involved.
  - 6y ago

gentleman11 6y ago

The discussion about this tech revolves around accuracy and racism, but the real threat is in global unlimited surveillance. China is installing 200 million of facial recognition cameras right now to keep the population under control. It might be the death of human freedom as this technology spreads Edit: one source says it is 400 million new cameras: https://www.cbc.ca/passionateeye/m_features/in-xinjiang-china-surveillance-technology-is-used-to-help-the-state-control https://www.cbc.ca/passionateeye/m_features/in-xinjiang-chin...

whatshisface 6y ago

How does computerized facial recognition compare in terms of racial bias and accuracy to human-brain facial recognition? Police are not exactly perfect in either regard.

suizi 6y ago
Face recognition widens the scope of how many people can be harassed.
- _underfl0w_ 6y ago
  While also enabling finger-pointing, e.g. the police can say "We aren't racist or aren't at fault. The system is just faulty." while the engineers behind the facial recognition tech can say that they, "Were just doing their job. The police should've heeded their disclaimers, etc."

danso 6y ago

This story is really alarming because as described, the police ran a face recognition tool based on a frame of grainy security footage and got a positive hit. Does this tool give any indication of a confidence value? Does it return a list (sorted by confidence) of possible suspects, or any other kind of feedback that would indicate even to a layperson how much uncertainty there is? The issue of face recognition algorithms performing worse on dark faces is a major problem. But the other side of it is: would police be more hesitant to act on such fuzzy evidence if the top match appeared to be a middle-class Caucasian (i.e. someone who is more likely to take legal recourse)?

Pxtl 6y ago
Intresting and related, a team made a neat "face depixelizer" that takes a pixelated image and uses machine learning to generate a face that should match the pixelated image. What's hilarious is that it makes faces that look nothing like the original high-resolution images. https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801
- mywittyname 6y ago
  I wonder if this is trained on the same, or similar, datasets.
  - jcims 6y ago
    One of the underlying models, PULSE, was trained on CelebAHQ, which is likely what the results are mostly white-looking. StyleGAN, which was trained on the much more diverse (but sparse) FFHQ dataset does come up with a much more diverse set of faces[1]...but PULSE couldn't get them to converge very closely on the pixelated subjects...so they went with CelebA [2]. [1] https://github.com/NVlabs/stylegan https://github.com/NVlabs/stylegan [2] https://arxiv.org/pdf/2003.03808.pdf https://arxiv.org/pdf/2003.03808.pdf (ctrl+f ffhq)
- jacquesm 6y ago
  That should be called a face generator, not a depixelizer.

MikusR 6y ago

Is that different from somebody getting arrested based on mistaken eyewitness.

Enginerrrd 6y ago
The difference is that is a known problem, but with ML, a large fraction of the population thinks it's infallible. Worse, its reported confidence for an individual face may be grossly overstated, since that is based on all the data it was trained on, rather than the particular subset you may be dealing with.
- raxxorrax 6y ago
  large fraction of the population and ML marketing both believe that. I still think it insane. We have falling crime rates and we still arm ourselves as fast as we can. Humanity could live without face recognition and we wouldn't even suffer any penalties. Nope, people need to sell their evidently shitty ML work.
  - treis 6y ago
    (1) We still have extreme levels of crime compared to other first world countries even if it is in decline (2) Your argument strikes me as somewhat similar to "I feel fine why should I keep taking my medicine?". It's not exactly the same as the medicine is scientifically proven to cure disease while it's impossible to measure the impact of police on crime. But "things are getting better so we should change what we're doing" is not a particularly sound logical argument.
    - raxxorrax 6y ago
      Crimes rates dropped even faster in countries with more rehabilitative approaches and long before some countries began to upgrade their police forces because of unrelated fears. It was more about giving people a second chance in all that. Criminologists aren't certain about surveillance having a positive or negative effects on crime. We have more than 40 studies with mixed results. What is certain with that this kind of surveillance isn't responsible for the falling crime rates described. Most data is from the UK. Currently I don't think countries without surveillance fair worse on crime. Maybe quite to the contrary. "what we're doing" is not equivalent to increasing video surveillance or generally increasing armament in civil spaces. It may be sound logic if you extend the benefit of the doubt but it may also just be a false statement. Since surveillance is actually constitutionally forbidden in many countries, on could argue that deployment would "increase crime". In some other sound logic it might just be a self reinforcing private prison industry with economic interests to keep a steady supply of criminals. Would also be completely sound. But all these discussions are quite dishonest, don't you think? I just don't want your fucking camera in my face.

rusty__ 6y ago

any defence lawyer with more than 3 brain cells would have an absolute field day deconstructing a case brought solely on the basis of a facial recognition. What happened to the idea that police need to gather a variety of evidence confirming their suspicions before an arrest is issued. Even a state prosecutor wouldn't authorize a warrant based on such flimsy methods.

ARandomerDude 6y ago
True but the defendant is still financially, and in many cases professionally, ruined.
failuser 6y ago
Getting a layer that would advise beyond pleading guilty for a reduced sentence is not a default option.

zro 6y ago

NPR article about the same, if you prefer to read instead of listen: https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig https://www.npr.org/2020/06/24/882683463/the-computer-got-it... I'll be watching this case with great interest

crazygringo 6y ago

In this particular case, computerized facial recognition is not the problem. Facial recognition produces potential matches. It's still up to humans to look at footage themselves and use their judgment as to whether it's actually the same person or not, as well as to judge whether other elements fit the suspect or not. The problem here is 100% on the cop(s) who made that call for themselves, or intentionally ignored obvious differences. (Of course, without us seeing the actual images in question, it's hard to judge.) There are plenty of dangers with facial recognition (like using it at scale, or to track people without accountability), but this one doesn't seem to be it.

ncallaway 6y ago
> The problem here is 100% on the cop(s) who made that call for themselves I disagree. There is plenty of blame on the cops who made that call for themselves, true. But there doesn't have to be a single party who is at fault. The facial recognition software is badly flawed in this dimension. It's well established that the current technologies are racially biased. So there's at least some fault in the developer of that technology, and the purchasing officer at the police department, and a criminal justice system that allows it to be used that way. Reducing a complex problem to a single at-fault person produces an analysis that will often let other issues continue to fester. Consider if the FAA always stopped the analysis of air-crashes at: "the pilot made an error, so we won't take any other corrective actions other than punishing the pilot". Air travel wouldn't nearly as safe as it is today. While we should hold these officers responsible for their mistake (abolish QI so that these officers could be sued civilly for the wrongful arrest!), we should also fix the other parts of the system that are obviously broken.
- dfxm12 6y ago
  The facial recognition software is badly flawed in this dimension. It's well established that the current technologies are racially biased. Who decided to use this software for this purpose, despite these bad flaws and well established bias? The buck stops with the cops.
  - 6y ago

vermontdevil 6y ago

From ACLU article: Third, Robert’s arrest demonstrates why claims that face recognition isn’t dangerous are far-removed from reality. Law enforcement has claimed that face recognition technology is only used as an investigative lead and not as the sole basis for arrest. But once the technology falsely identified Robert, there was no real investigation. I fear this is going to be the norm among police investigations.

vmception 6y ago

> Federal studies have shown that facial-recognition systems misidentify Asian and black people up to 100 times more often than white people. The idea behind inclusion is that this product would have never made it to production if the engineering teams, product team, executive team and board members represented the population. But enough representation so that there is a countering voice is even better. Would have just been "this edge case is not an edge case at all, axe it." Accurately addressing a market is the point of the corporation more than an illusion of meritocracy amongst the employees.

JangoSteve 6y ago
This is so incredibly common, it's embarrassing. I was on an expert panel about "AI and Machine Learning in Healthcare and Life Sciences" back in January, and I made it a point throughout my discussions to keep emphasizing the amount of bias inherent in our current systems, which ends up getting amplified and codified in machine learning systems. Worse yet, it ends up justifying the bias based on the false pretense that the systems built are objective and the data doesn't lie. Afterward, a couple people asked me to put together a list of the examples I cited in my talk. I'll be adding this to my list of examples: * A hospital AI algorithm discriminating against black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 * The dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abstract/2688587 https://jamanetwork.com/journals/jamadermatology/article-abs... And some other relevant, but not healthcare examples as well: * When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/ https://fortune.com/2019/08/16/google-jigsaw-perspective-rac... * When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio... * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though: * A hospital used AI to enable care and cut costs (though the reporting seems to over simplify and gloss over enough to make the actual analysis of the results a little suspect). https://www.healthcareitnews.com/news/flagler-hospital-uses-ai-create-clinical-pathways-enhance-care-and-slash-costs https://www.healthcareitnews.com/news/flagler-hospital-uses-...

at_a_remove 6y ago

I think that your prints, DNA, and so forth must be, in the interests of fairness, utterly erased from all systems in the case of false arrest. With some kind of enormous, ruinous financial penalty in place for the organizations for non-compliance, as well as automatic jail times for involved personnel. These things need teeth to happen.

ghostpepper 6y ago

He wasn't arrested until the shop owner had also "identified" him. The cops used a single frame of grainy video to pull his driver's license photo, and then put that photo in a lineup and showed the store clerk. The store clerk (who hadn't witnessed the crime and was going off the same frame of video fed into the facial recognition software) said the driver's license photo was a match. There are several problems with the conduct of the police in this story but IMHO the use of facial recognition is not the most egregious.

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
bsenftner 6y ago
Yes, this is a story of police misconduct. The regulation of facial recognition that is required is regulation against police/authority stupidity. The FR system aids in throwing away misses, leaving investigative leads. But if a criminal is not in the FR database to begin with, any results of the FR are wastes of time.
businesslucas 6y ago
It is not clear to me that the person who identified him was shop owner or clerk. From the nyt article: https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html https://www.nytimes.com/2020/06/24/technology/facial-recogni... "The Shinola shoplifting occurred in October 2018. Katherine Johnston, an investigator at Mackinac Partners, a loss prevention firm, reviewed the store’s surveillance video and sent a copy to the Detroit police" "In this case, however, according to the Detroit police report, investigators simply included Mr. Williams’s picture in a “6-pack photo lineup” they created and showed to Ms. Johnston, Shinola’s loss-prevention contractor, and she identified him. (Ms. Johnston declined to comment.)"
- ghostpepper 6y ago
  I think you're correct that the person was not an owner or clerk. IMHO the salient point is that the person was not any sort of eyewitness but merely comparing the same grainy photo as the algorithm.

sneak 6y ago

Another reason that it's absolutely insane that the state demands to know where you sleep at night in a free society. These clowns were able to just show up at his house and kidnap him. The practice of disclosing one's residence address to the state (for sale to data brokers[1] and accessible by stalkers and the like) when these kinds of abuses are happening is something that needs to stop. There's absolutely no reason that an ID should be gated on the state knowing your residence. It's none of their business. (It's not on a passport. Why is it on a driver's license?) [1]: https://www.newsweek.com/dmv-drivers-license-data-database-integrity-department-motor-vehicles-1458141 https://www.newsweek.com/dmv-drivers-license-data-database-i...

deleted 6y ago

[deleted]

TedDoesntTalk 6y ago

> Even if this technology does become accurate (at the expense of people like me), I don’t want my daughters’ faces to be part of some government database. Stop using Amazon Ring and similar doorbell products.

FpUser 6y ago

And then in some states employers are allowed to ask have you eve been arrested (never mind convicted of any crime) on employment application. Sure, keep putting people down. One day it might catch up with China's social scoring policies.

paulorlando 6y ago

I've been thinking this sort of event has become inevitable. Tech development and business models support extending the environments in which we collect images and analyze them. Confidence values lead to statistical guilt. I wrote about it here if interested: https://unintendedconsequenc.es/inevitable-surveillance/ https://unintendedconsequenc.es/inevitable-surveillance/

mistercool 6y ago

relevant: https://www.theregister.com/2020/06/24/face_criminal_ai/ https://www.theregister.com/2020/06/24/face_criminal_ai/

linuxftw 6y ago

Wait until you hear about how garbage and unscientific fingerprint identification is.

_underfl0w_ 6y ago
Speaking of pseudoscience, didn't most police forces just start phasing out polygraphs in the last decade?
- linuxftw 6y ago
  Unlikely unless they were compelled by law or found something else to replace it, and I think it's the latter. Something about machine learning and such.

js2 6y ago

> "I picked it up and held it to my face and told him, 'I hope you don't think all Black people look alike,' " Williams said. I'm white. I grew up around a sea of white faces. Often when watching a movie filled with a cast of non-white faces, I will have trouble distinguishing one actor from another, especially if they are dressed similarly. This sometimes happens in movies with faces similar to the kinds I grew up surrounded by, but less so. So unfortunately, yes, I probably do have more trouble distinguishing one black face from another vs one white face from another. This is known as the cross-race effect and it's only something I became aware of in the last 5-10 years. Add to that the fallibility of human memory, and I can't believe we still even use line ups. Are there any studies about how often line ups identify the wrong person? https://en.wikipedia.org/wiki/Cross-race_effect https://en.wikipedia.org/wiki/Cross-race_effect

SauciestGNU 6y ago
I lived in South Africa for a while and heard many times, with various degrees of irony, "you white people all look the same" from black South Africans. So yeah it's definitely a cross-racial recognition problem, and it's probably also a problem with distinguishing between members of visible minorities using traits beyond the most noticable othering characteristic.

Anthony-G 6y ago

There is just so much wrong with this story. For starters: The shoplifting incident occurred in October 2018 but it wasn’t until March 2019 that the police uploaded the security camera images to the state image-recognition system but the police still waited until the following January to arrest Williams. Unless there was something special about that date in October, there is no way for anyone to remember what they might have been doing on a particular day 15 months previously. Though, as it turns out, the NPR report states that the police did not even try to ascertain whether or not he had an alibi. Also, after 15 months, there is virtually no chance that any eye-witness (such as the security guard who picked Williams out of a line-up) would be able to recall what the suspect looked like with any degree of certainty or accuracy. This WUSF article [1] includes a photo of the actual “Investigative Lead Report” and the original image is far too dark for a anyone (human or algorithm) to recognise the person. It’s possible that the original is better quality and better detail can be discerned by applying image-processing filters – but it still looks like a very noisy source. That same “Investigative Lead Report” also clearly states that “This document is not a positive identification … and is not probable cause to arrest. Further investigation is needed to develop probable cause of arrest”. The New York Times article [2] states that this facial recognition technology that the Michigan tax-payer has paid millions of dollars for is known to be biased and that the vendors do “not formally measure the systems’ accuracy or bias”. Finally, the original NPR article states that > "Most of the time, people who are arrested using face recognition are not told face recognition was used to arrest them," said Jameson Spivack [1] https://www.wusf.org/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michigan/ https://www.wusf.org/the-computer-got-it-wrong-how-facial-re... [2] https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html https://www.nytimes.com/2020/06/24/technology/facial-recogni...

cwkoss 6y ago
Really seems like most police departments in our country are incompetent, negligent, ineffective and systemically racist. Many of these cops are earning $200k plus annually! Our law enforcement system is ridiculous and needs an overhaul.
- 6y ago

mnw21cam 6y ago

This is a classic example of the false positive rate fallacy. Let's say that there are a million people, and the police have photos of 100,000 of them. A crime is committed, and they pull the surveillance of it, and match against their database. They have a funky image matching system that has a false positive rate of 1 in 100,000 people, which is way more accurate than I think facial recognition systems are right now, but let's just roll with it. Of course, on average, this system will produce one positive hit per search. So, the police roll up to that person's home and arrest them. Then, in court, they get to argue that their system has a 1 in 100,000 false positive rate, so there is a chance of 1 in 100,000 that this person is innocent. Wrong! There are ten people in the population of 1 million that the software would comfortably produce a positive hit for. They can't all be the culprit. The chance isn't 1 in 100,000 that the person is innocent - it is in fact at least 9 out of 10 that they are innocent. This person just happens to be the one person out of the ten that would match that had the bad luck to be stored in the police database. Nothing more.

Buttons840 6y ago
There's a good book called "The Drunkards Walk", that describes a woman who was jailed after having 2 children die from SIDS. They argued that the odds of this happening is 1 in a million (or something like that), so probably the woman is a baby killer. The prosecution had statisticians argue this. The woman was found guilty. She later won on appeal in part because the defense showed that the testimony and argument of the original statisticians were wrong. This stuff is so easy to get wrong. A little knowledge of statistics can be dangerous.
- Polylactic_acid 6y ago
  And even if the original stats were right. A 1 in a million event happens to about 100 people per day in the US.
  - microcolonel 6y ago
    > A 1 in a million event happens to about 100 people per day in the US. This is a meaningless statement, you could choose literally any number for this statement, because you are missing the denominator.

seebetter 6y ago

Reminds me of this- Facial recognition technology flagged 26 California lawmakers as criminals. (August 2019) https://www.mercurynews.com/2019/08/14/facial-recognition-technology-flagged-26-california-lawmakers-as-criminals-this-bill-to-ban-the-tech-is-headed-to-the-senate/ https://www.mercurynews.com/2019/08/14/facial-recognition-te...

throwawaysea 6y ago

A human still confirmed the match right? That makes this not a facial recognition issue but something else.

aaronmdjones 6y ago
A human who had only seen the same grainy security footage that the algorithm saw.

tantalor 6y ago

No mention of whether a judge signed a warrant for the arrest. In what world can cops just show up and arrest you on your front lawn based on their hunch?

cpeterso 6y ago

What is a unique use case for facial recognition that cannot be abused and has no other alternative solution? Even the "good" use cases like unlocking your phone have security problems because malicious people can use photos or videos of your face and you can't change your face like you would a breached username and password.

aritraghosh007 6y ago

The pandemic has accelerated the use of no-touch surfaces specially at places like an airport that are more inclined to now use a face recognition security kiosk. What's not clear is the vetting process for these (albeit controversial) technologies. What if Google thinks person A is an offender but Amazon thinks otherwise. Can they be used as counter evidence? What is the gold standard for surveillance?

jandrewrogers 6y ago

It isn't just facial recognition, license plate readers can have the same indefensibly Kafka-esque outcomes where no one is held accountable for verifying computer-generated "evidence". Systems like in the article make it so cheap for the government to make a mistake, since there are few consequences, that they simply accept mistakes as a cost of doing business. Someone I know received vehicular fines from San Francisco on an almost weekly basis solely from license plate reader hits. The documentary evidence sent with the fines clearly showed her car had been misidentified but no one ever bothered to check. She was forced to fight each and every fine because they come with a presumption of guilt, but as soon as she cleared one they would send her a new one. The experience became extremely upsetting for her, the entire bureaucracy simply didn't care. It took threats of legal action against the city for them to set a flag that apparently causes violations attributed to her car to be manually reviewed. The city itself claimed the system was only 80-90% accurate, but they didn't believe that to be a problem.

black_puppydog 6y ago
I agree that's bad, and license plate readers come with their own set of problems. But being biased by the skin color of the driver is (AFAIK) not one of them. Which is exactly the problem with vision systems applied to humans, at least the ones we've seen deployed so far. If a system discriminates against a specific population, that's very different from (indiscriminately) being unreliable.
vmception 6y ago
That suddenly reminded me why I feel so privileged to not own a car, a distinct contrast from when I was a teenager and felt it was a rite of passage! I had forgotten about the routine of fighting traffic tickets multiple times a year as a fact of life. Let alone fender benders. I had only been reveling in the lack of a frustrating commute. Last decade I did get a car for 3 months, and the insurance company was so thrilled that I was "such a good driver" because of my "spotless record" for many years. Little do they know I just don't drive and perhaps have now less experience than others. Although tangentially, their risk matrix actually might be correct, if I can afford to live in dense desirable areas then maybe it is less likely that I would be going fast and getting into circumstances that pull from their insurance pool at larger amounts. They probably thought "one of the largest companies in the world probably chauffeurs him down the highway in a bus anyway"

neonate 6y ago

The prosecutor and the police chief should personally apologize to his daughters, assuming that would be age appropriate.

datavirtue 6y ago

All that rigamaroll for $3800 worth of crap? They should just switch it up and start entrapping people like the FBI does. Then at least they would have perhaps one leg to stand on.

joyj2nd 6y ago

Understandable, all black men look the same.

bosswipe 6y ago

Boston just banned facial recognition, as have San Francisco, Oakland and a bunch of other cities. You can join this movement by urging your local government officials to follow suit.

renewiltord 6y ago

I've got to be honest: I'm getting the picture the police here aren't very competent. I know I know, POSIWID and maybe they're very competently aiming at the current outcome. But don't they just look like a bunch of idiots?

ibudiallo 6y ago

Here is a part that I personally have to wrestle with: > "They never even asked him any questions before arresting him. They never asked him if he had an alibi. They never asked if he had a red Cardinals hat. They never asked him where he was that day," said lawyer Phil Mayor with the ACLU of Michigan. When I was fired by an automated system, no one asked if I had done something wrong. They asked me to leave. If they had just checked his alibi, he would have been cleared. But the machine said it was him, so case closed. Not too long ago, I wrote a comment here about this [1]: > The trouble is not that the AI can be wrong, it's that we will rely on its answers to make decisions. > When the facial recognition software combines your facial expression and your name, while you are walking under the bridge late at night, in an unfamiliar neighborhood, and you are black; your terrorist score is at 52%. A police car is dispatched. Most of us here can be excited about Facial Recognition technology but still know that it's not something to be deployed in the field. It's by no means ready. We might even consider the moral ethics before building it as a toy. But that's not how it is being sold to law enforcement or other entities. It's _Reduce crime in your cities. Catch criminals in ways never thought possible. Catch terrorists before they blow up anything._ It is sold as an ultimate decision maker. [1]:https://news.ycombinator.com/item?id=21339530 https://news.ycombinator.com/item?id=21339530

nimbius 6y ago
This shit right here. This is why I don't stop for the inventory control alarms at department store doorways if they go off. I know I've paid, and little sirens are just a nuisance at this point. This is why I've never stopped for receipt checks, because it's my receipt, and I've paid. The security theatre is just bad comedy. Just because the machine says I've done a no no, doesn't mean I can't come back and win a lawsuit later. It doesn't abdicate cops from their jobs. I have a winning complexion, so I'll never enjoy a false positive, but if I do, I'll make sure it bankrupts whatever startup incubator garbage decided to shill a replacement for real law enforcement.
- demadog 6y ago
  So out of curiosity, you just roll out of Costco with a cart full of food and gear while the receipt checker tries to stop you?

deleted 6y ago

[deleted]

jackklika 6y ago

The company that developed this software is Dataworks plus, according to the article. Name and shame.

redorb 6y ago

If it's statistically proven to not work with black people then I think the only options are 1) Make it avoid black people, i.e. they aren't stored in the database and aren't processed when scanned. 2) Put a 5 year hiatus on commercial / public use. Either of these things are more acceptable than too many false positives. #1 is really interesting to me as a thought experiment because it makes everyone think twice.

fatso784 6y ago
The issue is that it's hard to determine who is considered "black" and who is not since race is a made-up fiction constructed by white supremacists that is now so engrained in American society we find it difficult if not impossible to reflect on it. Are we using the Fitzpatrick skin color scale? Are we continuing the hypodescent rule (an algorithm, btw)? Maybe we just outlaw face recognition in criminal justice entirely.
- redorb 6y ago
  It definitely shouldn't be allowed as the main evidence - perhaps the answer to skin tone is to use the data and see where the algo starts to get too many false positives.. The largest - over arching idea - is to get everyone to think twice by making the majority think twice - If white people think 'its only for us!?' it'll make them really study the effects.. (I'm white.)

ineedasername 6y ago

It's supposed to be a cornerstone of "innocent until proven guilty" legal systems that it is better to have 10 guilty people go free than to deprive a single innocent person of their freedom. It seems like the needle has been moving in the wrong direction on that. I'm not sure it that's just my impression on things, or if it's because there's more awareness with the internet/social networking of issues...

atum47 6y ago

well, I'm going to take something out of my chest. every time I shared a project here using machine learning people always gave me crap. saying my models were simplistic, or I did something wrong or the solution didn't work 100% of the time. well, I studied ML back in college. the basics, the algorithms that started all, linear regression, perceptron, adaline, knn, kmeans... and guess what? ML doesn't work 100% of the time. I always wanted to see how people would react when a car driven by ml hits something or when they based an important decision based on the classification of an nn. ML should be used along side human intelligence not by itself. you don't blindly trusts a black box.

aussieguy1234 6y ago

In alot of police departments around the world, the photo database used is the drivers license database. There is clothing available that can confuse facial recognition systems. What would happen if, next time you go for your drivers license photo, you wore a T shirt designed to confuse facial recognition, for example like this one? https://www.redbubble.com/i/t-shirt/Anti-Surveillance-Clothing-by-Naamiko/24714049.1YYVU?u https://www.redbubble.com/i/t-shirt/Anti-Surveillance-Clothi...

nebulous1 6y ago

> In Williams' case, police had asked the store security guard, who had not witnessed the robbery, to pick the suspect out of a photo lineup based on the footage, and the security guard selected Williams. Great job police

czbond 6y ago

A few things I just don't have the stomach for as an engineer, writing software that: - impacts someones health - impacts someones finances - impacts someones freedoms Call me weak, but I think about the "what ifs" a bit too much in those cases. What if my bug keeps them from selling their stock and they lose their savings? What if the wrong person is arrested, etc?

Barrin92 6y ago
Why would anyone call you weak, that's principled and it's the correct attitude. It's the people who don't think about the consequences of the products they help build that are the problem.
- czbond 6y ago
  Thank you for that really helpful reply. I re-framed my thoughts based on it.

alex_young 6y ago

This technology will never be ready to use like this. Similarly we shouldn’t collect vast databases of fingerprints or DNA and search them for every crime. Why? Because error rates are unavoidable. There is some uncertainty, and in large enough numbers you will find false matches with perfect DNA matching. We must keep our senses and use these technologies to help us rather than find the hidden bad guy.

kwonkicker 6y ago

Due process should not be abandoned in favour of automation. This was police negligence as much as it was a software mismatch. One more thing, the article was being to dramatic about the whole incident.

w_t_payne 6y ago

Perhaps we, as technologists, are going about this the wrong way. Maybe, instead of trying to reduce the false alarm rate to an arbitrarily low number, we instead develop CFAR (Constant false alarm rate) systems, so that users of the system know that they will get some false alarms, and develop procedures for responding appropriately. In that way, we could get the benefit of the technology, whilst also ensuring that the system as a whole (man and machine together) are designed to be robust and have appropriate checks and balances.

gwd 6y ago
If you follow the link, you'll see that the computer report had this message right at the top in massive letters: THIS DOCUMENT IS NOT A POSITIVE IDENTIFICATION. IT IS AN INVESTIGATIVE LEAD ONLY AND IS _NOT_ PROBABLE CAUSE TO ARREST. FURTHER INVESTIGATION IS NEEDED TO DEVELOP PROBABLE CAUSE TO ARREST. I mean, what else could the technologists have done?
- Mangalor 6y ago
  I wonder if the problem here is just the way traditional policing works, not even the technology.
  - noisy_boy 6y ago
    It is indeed the way traditional policing works. The average police officer things facial recognition is the visual equivalent of Google and since he/she can similarly rely on search results - they either have no idea about false positive biases based on race or worse, probably are too lazy to dig deeper. Though the suffering of the victims of such wrong matches is real, one consolation is that more of such cases will hopefully bring about the much needed scepticism in the results so that some old-fashioned validation/investigation is done.
- w_t_payne 6y ago
  Well .. it looks like it isn't enough, and additional human-factors oriented requirements need to be documented - like deliberately engineering the system to produce a certain number of false alarms so the police always have to do additional work to discard them - as opposed to only sometimes.

MertsA 6y ago

I would love to see police trying to take a crack at this from the other side of things. Instead of matching against a database, Set up a style GAN and come up with a mask of the original photo or video to isolate just the face and have the discriminator try to match the face. Then at the end you can see the generated face with a decent pose and more importantly look through the range of generated faces that result in a reasonable match to give a somewhat decent idea of how confident you should be about any identification.

loup-vaillant 6y ago

And now the poor guy has an arrest record. Which wouldn't be a problem in reasonable legislations, where it's nobody's business whether you've been arrested or not, as long as you've not been convicted. But in the US, I've heard that it can make it harder to get a job. I believe I'm starting to get a feel for how the school to prison pipeline may work.

hkai 6y ago

While this case is bad enough, mistakes like this are not the biggest concern. Mistakenly arrested people are (hopefully) eventually released, even though they have to go through quite a bit of trouble. The consequence that is much worse would be mass incarceration of certain groups, because the AI is too good at catching people who actually did something. This second wave of mass incarceration will lead to even more single parent families and poor households, and will reinforce the current situation.

d--b 6y ago

The worst part is they use facial recognition which finds someone that looks like the suspect, and then they put the guy in a lineup and have him identified by the victim. Wtf?

jl2718 6y ago
That’s a good point. The selection from a lineup then gives less information. So the real issue is poorly understood priors. I’m pretty sure that this can be used fairly with a rigorous Bayesian treatment.

blackrock 6y ago

TOTAL FAIL.