9 ms·
Police used AI facial recognition to wrongly arrest TN woman for crimes in ND
- jqpabc123 7mo agoAI is a liability issue waiting to happen. And this is just another example.
- garyfirestorm 7mo agoIt’s a tool. Used incorrectly will lead to errors. Just like a hammer, used incorrectly could hit the users finger.
- happytoexplain 7mo agoThere is enormous variability in how hard a tool is to use correctly, how likely it is to go wrong, and how severe the consequences are. AI has a wide range on all those variables because its use cases vary so widely compared to a hammer. The use case here is police facial recognition. Not hitting nails. The parent wasn't saying "AI is a liability" with no context.
- mikkupikku 7mo agoWhen somebody uses a tool to hurt somebody, they need to be held accountable. If I smack you with a hammer, that needs to be prosecuted. Using AI is no different. The problem here is incidental to the tool; it was done by the cops and therefore nobody will be held accountable.
- tovej 7mo agoSystems are also a tool. Whoever institutes and helps build the system that systematically results in harm is also responsible. That would be the vendors, the system planners, and the institutions that greenlit this. It would also include the larger financial tech circle that is trying to drive large scale AI adoption. Like Peter Thiel, who sees technology as an "alternative to politics". I.e. a way to circumvent democracy [1] [1] https://stavroulapabst.substack.com/p/techxgeopolitics-18-technology-as https://stavroulapabst.substack.com/p/techxgeopolitics-18-te...
- fc417fc802 7mo agoNonsense. The manufacturer, distributor, and vendor of a hammer are not liable for its misuse. We already litigated and then legislated this regarding guns in the US. As much as I detest Clearview and Thiel the fault for this incident falls squarely on the justice system.
- tovej 6mo agoYour first paragraph conflates the system with the tool. Please at least parse what I wrote before you respond. You are also conflating legality and morality. The US gun industry being good at lobbying has no bearing on whether an industry that enables mass school shootings is accountable or not. I mean it clearly is. Just compare gun deaths in the US to any civilized nation and you'll see that gun control is the moral and sane approach. A hammer/gun is not the same as the wider hammer/gun industrial system, and the societal systems it is a part of. The justice system is a part of that. So even though you say you disagree, somehow you still agree?
- fc417fc802 6mo ago> Your first paragraph conflates the system with the tool. No, it responds to the claim you made. You asserted that systems are also tools and attributed fault on that basis. Perhaps at least reread your own comment before condescending. > You are also conflating legality and morality. Also incorrect. I understood you to be claiming tool vendors to bear both legal and moral responsibility; perhaps I misunderstood. Regardless, my position is that tool vendors bear neither of those more or less unconditionally. The only way a tool vendor can become responsible in any sense (IMO) is if they knowingly and intentionally facilitate a particular outcome. The manufacturer of a hammer, gun, or AI facial recognition system is never at fault for what the user does with it unless they actively encouraged that particular use.
- tovej 6mo agoThe AI companies are very obviously facilitating particular outcomes.
- suzzer99 7mo agoDynamite is a tool. But we don't hand it out to anyone who wants to play with it.
- mikkupikku 7mo agoWe used to until quite recently. Anybody could buy dynamite at the hardware store. We had to end this because of criminals using it to hurt people.
- jqpabc123 7mo agoLook for AI to follow a similar trajectory over time.
- GaryBluto 7mo agoImpossible at this point. You cannot download dynamite.
- mikkupikku 7mo agoYes, regulation is inevitable.
- jfengel 7mo agoRegulation is impossible. The AI barons literally control the federal government, so not even state regulations get tried.
- jfengel 7mo agoExcept this time the criminals are police.
- AngryData 7mo agoThey are far more often than anyone wants to admit. That's how we got 25% of the world's prison population.
- skeeter2020 7mo agoAI feels closer to a firearm than a hammer when accessing law enforcement's ability to quickly do massive, unrecoverable harm.
- jqpabc123 7mo agoUsed incorrectly will lead to errors. Only one small little problem --- there is no way to tell if you are using it "correctly". The only way to be sure is to not use it. Using it basically boils down to, "Do you feel lucky?". The Fargo police didn't get lucky in this case. And now the liability kicks in.
- jfengel 7mo agoNow the "qualified" immunity kicks in.
- jqpabc123 7mo agoWe will find out. But relying on AI is likely to cost the city of Fargo in one way or another. They say they have already stopped using AI and returned to good old fashioned human investigation. https://www.lawlegalhub.com/how-much-is-a-wrongful-arrest-lawsuit-worth/ https://www.lawlegalhub.com/how-much-is-a-wrongful-arrest-la...
- nkrisc 7mo agoSome basic investigatory police work (the kind they did before AI) would have revealed the mistake before an innocent woman’s life was destroyed.
- jqpabc123 7mo agoYes. But doing the investigation negates much of the incentive for using AI. Look for similar to play out elsewhere --- using unreliable tools for decision making is not a good, responsible business plan. And lawyers are just waiting to press the point.
- bornfreddy 7mo agoAI can provide leads. Someone still needs to verify them and decide.
- 7mo ago
- tgv 7mo agoThis tool, however, is specifically built for mass surveillance. It serves no other purpose. The tool is broken, and everybody knows it. The tool makers are at least as guilty as those who use it.
- cyanydeez 7mo agoThe tool, like Google search, is likely biased towards returning results regardless of confidence.
- fc417fc802 7mo agoThe tool is unethical, not broken. And unfortunately remains legal for the time being. To that end it's a social or political problem that can be fixed.
- MattDaEskimo 7mo agoWhat kind of outcome results from misuse? Clearly a hammer's misuse has very little in common with a global, hivemind network used in high-stake campaigns. Now, if I misused a hammer and it hurt everyone's thumb in my country, then maybe what you said would have some merit. Otherwise, I'd say it's an extremely lazy argument
- hrimfaxi 7mo agoUnlike hammers people preface things with "claude says", etc. I never see that kind of distancing with tools that aren't AI.
- gtowey 7mo agoIt's the opposite, it's absolution from liability. "The AI did it" is the ultimate excuse to avoid accepting responsibility and consequences.
- jqpabc123 7mo agoCourts are already refusing to accept this excuse. https://pub.towardsai.net/the-air-gapped-chronicles-the-court-asked-for-the-llms-reasoning-48471090eada https://pub.towardsai.net/the-air-gapped-chronicles-the-cour...
- gtowey 7mo agoGood to know it's not a fait accompli yet, but I wont be surprised to see corporations pushing for this hard.
- Hizonner 7mo ago... which is why the institutions that assign responsibility and consequences need to make it really clear that excuse won't fly. With illustrative examples.
- xtajv 6mo agoIt occurs to me that software engineering is just about the only engineering field which is neither licensed nor bonded nor insured. I wonder if AI / shadow IT will change that.
- jqpabc123 6mo agoI wonder if AI / shadow IT will change that. I doubt it. Computing has traditionally been all about math and logic. This is really all that a binary logic computer is capable of. When applied to this purpose, it can offer highly accurate results at very low cost. Current AI is an attempt to branch out from simply calculating into decision making. But it does so in the worst possible way --- using probability and statistics (aka guesswork) instead of logic and reasoning. In other words, AI offers questionable results at high cost. As this article shows, relying on guesswork is a legal liability issue waiting to happen in many (if not most) operating environments.
- xtajv 6mo agoHeh, I wasn't suggesting that AI would actually replace decision-making. Rather, I wonder whether attempts to use AI in this way would result in such publicly-embarassing and catastrophic outcomes that software engineers might decide to organize professional guardrails about it. I fully agree, this seems like a legal liability issue waiting to happen.
- jqpabc123 6mo agosoftware engineers might decide to organize professional guardrails about it. Engineering isn't the one creating the liability issue here --- marketing is. And it starts with using the word "intelligence" to describe what an LLM does.
- mitchbob 7mo agoEarlier discussion (405 comments): https://news.ycombinator.com/item?id=47356968 https://news.ycombinator.com/item?id=47356968
- casey2 7mo ago[flagged]
- jstanley 7mo agoYou think they deliberately chose to do this to a woman? Why?
- cyanydeez 7mo agoProbably just reading the room, with States like texas making abortions illegal and allowing random citizens from enforcing that. Famously, abortions are a woman thing. Anyway, looking through the facts, it's just some random woman. There's better evidence that these facial recognition systems are much worse at minorities rather than genders. Interesting biases are own-gendeR: https://pmc.ncbi.nlm.nih.gov/articles/PMC11841357/ https://pmc.ncbi.nlm.nih.gov/articles/PMC11841357/ Racial bias: https://mitsloan.mit.edu/ideas-made-to-matter/unmasking-bias-facial-recognition-algorithms https://mitsloan.mit.edu/ideas-made-to-matter/unmasking-bias... Miss rates: https://par.nsf.gov/servlets/purl/10358566 https://par.nsf.gov/servlets/purl/10358566 Although you can probably interpret the facts differently, we've seen how any search function gets enshittified: Once people get used to searching for things, they tend to select something that returns results vs something that fails to return results. Rather than the user blaming themselves, they blame the searcher. As such, any search system overtime will bias towards returning search (eg, Outlook), rather than accuracy. So if these systems easily miss certain classes of people, women, minorities, they'll more likely be surfaced as inaccurate matches rather than men who'll have a higher confidence of being screened out. That's how I interpret this 2 second commment.
- deleted 7mo ago[deleted]
- IncreasePosts 7mo agoWhat? Women are much more sympathetic figures when it comes to crime and punishment. And there are 10x more men in prison in america than women. If you were trying to "introduce" some nefarious law enforcement system to the US you would use it on undesirable men first (drug addicts and gang members)
- oopsiremembered 7mo agoMoney quote from someone quoted in the article: "[I]t’s not just a technology problem, it’s a technology and people problem." I can't. I just can't.
- bryanrasmussen 7mo agoI've been hearing "it's not just... it's a" touted as an AI sign recently, personally I think it's an AI sign because it's a human thinking shortcut sign, and AI copies it, but it would be funny if AI wrote the article and then hallucinated this specific money quote.
- oopsiremembered 7mo agoI doubt this happened here, but FWIW, AI does have a habit of "cleaning up" (read: hallucinating) interview transcript quotes if you ask it to go through a transcript and pull quotes. You have to prompt AI very specifically to get it to not "clean up" the quotes when you ask it to do that task.
- b112 7mo agoAnd why not? If you look at examples of people quoting on the internet, lots are out of context, paraphrased, or made up. AI is just mimicking what it has seen.
- bryanrasmussen 6mo agoI often wake up at night from dreams of a crying AI yelling at me "I learned it from watching you, alright?!"
- firefoxd 7mo agoWithout even looking at the AI part, I have a single question: Did anybody investigate? That's it. Whether it's AI that flagged her, or a witness who saw her, or her IP address appeared on the logs. Did anybody bothered to ask her "where were you the morning of july 10th between 3 and 4pm. But that's not what happened, they saw the data and said "we got her". But this is the worst part of the story: > And after her ordeal, she never plans to return to the state: “I’m just glad it’s over,” she told WDAY. “I’ll never go back to North Dakota.” That's the lesson? Never go back to North Dakota. No, challenge the entire system. A few years back it was a kid accused of shoplifting [0]. Then a man dragged while his family was crying [1]. Unless we fight back, we are all guilty until cleared. [0]: https://www.theregister.com/2021/05/29/apple_sis_lawsuit/ https://www.theregister.com/2021/05/29/apple_sis_lawsuit/ [1]: https://news.ycombinator.com/item?id=23628394 https://news.ycombinator.com/item?id=23628394
- bl4ckneon 7mo agoI think you missed many important points. "The trauma, loss of liberty, and reputational damage cannot be easily fixed,” Lipps' lawyers told CNN in an email. That sounds a LOT like a statement you make for before suing for damages, not to mention they literally say "Her lawyers are exploring civil rights claims but have yet to file a lawsuit, they said." This lady probably just wants to go back to normal life and get some money for the hell they put her in. She has never been on a airplane before, I doubt she is going to take on the entire system like you suggest. Easier said than done to "challenge the entire system", what does that even mean exactly?
- 3eb7988a1663 7mo agoIt was worse than that, the reporting from an earlier story[0] ...Unable to pay her bills from jail, she lost her home, her car and even her dog. There is not a jury in the country that will side against the woman. I am not even sure who will make the best pop culture mashup - John Wick or a country song writer? (Also, what happened to journalism - no Oxford comma?) [0] https://news.ycombinator.com/item?id=47356968 https://news.ycombinator.com/item?id=47356968
- tlogan 7mo agoThis is a weak or misleading story about AI. First, the detective used the FaceSketchID system, which has been around since around 2014. It is not new or uniquely tied to modern AI. Second, the system only suggests possible matches. It is still up to the detective to investigate further and decide whether to pursue charges. And then it is up to court to issue the warrant. The real question is why she was held in jail for four months. That is the part that I do not understand. My understanding is that there is 30-day limit (the requesting state must pick up the defendant within 30 day). Regarding the individual involved, Angela Lipps, she has reportedly been arrested before, so it is possible she was on parole. So maybe they were holding her because of that? Can someone clarify how that process works?
- suzzer99 7mo agoIn the US there are no consequences for people in power failing to follow procedures, laws or regulations - except for being told to stop doing whatever illegal thing they're doing, and possibly getting sued way down the line, which gets paid by taxpayers.
- tlogan 7mo agoFrom reading more into the case, it seems the issue may be related to how her lawyer handled the case. They probably did “identity challenge” arguing that she is not the right person. But from Tennessee’s perspective, she was considered the correct person to be arrested, so there was no “mistaken identity” in their system. In other words, North Dakota Wanted person x and here is person x. Once a judge in North Dakota reviewed the full evidence (and found that person they issued warrant for arrest is not one they want), the case was dismissed.
- frankharv 7mo agoYes but a judge issued the warrant in the first place. Cops did not do a proper investigation and the judge green-lighted it. It is all on the JUDGE or possibly a magistrate who approved a faulty warrant. The judge failed the poor woman. FIRE him. Then sue Clearview for big bucks.
- renewiltord 7mo ago[flagged]
- rootusrootus 7mo agoHas it not been fairly common to require police officers to have a bachelor’s degree? Or an associate’s? I think recently that has been relaxed but I’ve lived in places where it was absolutely a requirement. I don’t think they’re as stupid as you suggest.
- llbbdd 7mo agoHaving a degree is a very low bar for intelligence.
- deleted 7mo ago[deleted]
- zephen 7mo ago> I don’t think they’re as stupid as you suggest. I'll just leave this here: https://abcnews.com/US/court-oks-barring-high-iqs-cops/story?id=95836 https://abcnews.com/US/court-oks-barring-high-iqs-cops/story...
- rootusrootus 7mo agoIt says the national average is just slightly above average IQ. Sounds fine.
- zephen 7mo ago> Sounds fine. Really? Maybe your perception of the "average" person is colored by where you live and who you interact with. In any case, the dumber they are, the more lethal they are. https://www.sciencedirect.com/science/article/abs/pii/S0160289618300734 https://www.sciencedirect.com/science/article/abs/pii/S01602...
- 7mo ago
- garethsprice 7mo agoThe vendor they used, Clearview AI, does not allow you to request data deletion unless you live in one of the half-dozen states that legally mandate it. https://www.clearview.ai/privacy-and-requests https://www.clearview.ai/privacy-and-requests I have suddenly becomes very interested in New York's S1422 Biometric Privacy Act.
- dawnerd 7mo agoSadly this is really the only tool we have right now. Just have to keep spamming them with delete requests because once they delete it’ll end up back in their database eventually.
- guelo 7mo agoTo get your data deleted in the states that require it you have to submit a photo of yourself which I really don't want to do for a sketchy company with ties to evil billionaire Peter Thiel.
- csomar 7mo agoWould that data be admissible in a lawsuit if you'd already submitted a deletion request?
- KomoD 7mo agorelated https://noyb.eu/en/criminal-complaint-against-facial-recognition-company-clearview-ai https://noyb.eu/en/criminal-complaint-against-facial-recogni...
- clcaev 7mo agoFor IL residents the policy requires collection and retention of your biomarkers. Presumably there is a law enforcement exclusion implicitly or explicitly, eg search via administrative warrant.
- balderdash 7mo agolol you have to give them your picture to delete your picture
- giardini 7mo agoThis has been posted at least twice before on HN.
- indigodaddy 7mo agoA lot of dumb shit happens in this arena, where if you had just one smart cop, it could have been prevented. Here’s one from 2023: https://youtu.be/lPUBXN2Fd_E https://youtu.be/lPUBXN2Fd_E
- deleted 7mo ago[deleted]
- mememememememo 7mo agoWow thought the bar for probable cause for an arrest warrant would be much higher. Especially to drag soneone from another state.
- cortesoft 7mo agoIt’s a classic example of the base rate fallacy. The judge sees that a system with a seemingly high accuracy rate (like 99.999% accurate) has flagged a person, and they assume that means the person is highly likely to be guilty. However, the system uses a dragnet approach, and is checking against millions of people. If you are checking 300 million people, that 99.999% accuracy check is going to find 3,000 people, and AT LEAST 99.96% of those people are going to be innocent. This is why we can’t have wide, automated surveillance.
- ValveFan6969 7mo ago[dead]
- jeremie_strand 7mo ago[dead]
- internetguy 7mo agoInsane. Not even an apology. And they ask why we should respect the police.
- lucasfin000 7mo agoThe actual scariest part isn't that the AI got it wrong... it's that nobody felt the need to verify the AI. A tip from an anonymous caller can get investigated and found out if its true or not, and a match from a facial recognition system apparently does not. People haven't built better investigative tools they've just built better ways to skip around the investigation.
- deleted 7mo ago[deleted]
- dnaranjo 7mo ago[dead]
- advisedwang 7mo agoFor me the worst thing in this case is that a JUDGE signed off on an arrest warrant with only a clearview match linking Ms Lipps to the crime. A judge and the warrant process are supposed to be the safeguard against police doing shady stuff (like relying on an AI hit to decide who commit a crime). But if the judges can't be bothered...
- shevy-java 7mo agoSo cops used AI to attempt to investigate a crime. But, there was no crime - the arrest was wrong. Why can cops excuse themselves here for delegating their responsibilities (protecting society, allegedly that is) onto software? AI may also be written by some corporations to "tweak" this or that, see this foreign-looking guy being more likely to be AI-investigated. This is like the movie Minority Report - but stupid. IMO the courts should conclude that cops should not be allowed to use AI without having a prior, independently verified objective reasoning for any investigation. This mass sniffing that is currently going on is very clearly illegal. The current orange guy does not care about the law; see flock cameras aka spy cameras employed by the government on all car drivers at all times.
- cortesoft 7mo agoWhat? There was absolutely a crime?
- mike741 7mo agoYou're absolutely right! There was a crime. I appreciate the course correction—it’s a significant oversight on my part. I've updated our previous plan to better reflect that a crime occurred. You're under arrest.
- cindyllm 7mo ago[dead]
- mistM 7mo ago[dead]
- selcuka 7mo agoPrevious discussion: https://news.ycombinator.com/item?id=47356968 https://news.ycombinator.com/item?id=47356968
- repiret 7mo agoThis isn't the first time this month I've read about someone suffering consequences of mistaken identity after their facial recognition said they look like someone who committed a crime. I'm sure this is starting to happen at an alarming rate. The fundamental problem is that among the 350 million people living in the United States, there are a lot of pairs of people who look pretty darn similar. It used to be impractical to ask a question like "who in the US looks like the person in this security footage", and so as a matter of practicality, once you found someone who looks like the suspect, you probably also have other evidence, even if it's pretty weak, linking them to the crime. But with AI, you can ask "who in the US looks like this person", and so we need to re-calibrate what it means if all you know is that someone looks like a suspect. I am of the opinion that "looks like someone," in the absence of any other evidence, is reasonable suspicion, but not probable cause, that you are the person you look like. Reasonable suspicion is enough for the police to stop you on the street and ask for your ID, but not enough to arrest you. There are other data points that alone might not even be reasonable suspicion, but could be combined with "looks like someone" to make probable cause, such as "was near the place at the time the crime happened". AI isn't really the problem, even whether or not the AI's determination that two people look alike is valid or reviewed by a human isn't the problem. The problem is assuming that because two people look alike they must be the same person, even if you have no other evidence of them being the same person.
- chaostheory 6mo agoI’m starting to believe that the internet will become a Dark Forest soon.
- RagnarD 6mo agoShe should sue the city controlling that police department, into oblivion. Or at least to the absolute max she can get.
- ch3 6mo ago[dead]
- pratyushsood 6mo agoa single data point (facial match) would never pass a security assessment. AI-assisted decisions need verification chains.