8 ms·

Here is a part that I personally have to wrestle with: > "They never even asked him any questions before arresting him. They never asked him if he had an alibi

Here is a part that I personally have to wrestle with: > "They never even asked him any questions before arresting him. They never asked him if he had an alibi. They never asked if he had a red Cardinals hat. They never asked him where he was that day," said lawyer Phil Mayor with the ACLU of Michigan. When I was fired by an automated system, no one asked if I had done something wrong. They asked me to leave. If they had just checked his alibi, he would have been cleared. But the machine said it was him, so case closed. Not too long ago, I wrote a comment here about this [1]: > The trouble is not that the AI can be wrong, it's that we will rely on its answers to make decisions. > When the facial recognition software combines your facial expression and your name, while you are walking under the bridge late at night, in an unfamiliar neighborhood, and you are black; your terrorist score is at 52%. A police car is dispatched. Most of us here can be excited about Facial Recognition technology but still know that it's not something to be deployed in the field. It's by no means ready. We might even consider the moral ethics before building it as a toy. But that's not how it is being sold to law enforcement or other entities. It's _Reduce crime in your cities. Catch criminals in ways never thought possible. Catch terrorists before they blow up anything._ It is sold as an ultimate decision maker. [1]:https://news.ycombinator.com/item?id=21339530 https://news.ycombinator.com/item?id=21339530

nimbius 6y ago
This shit right here. This is why I don't stop for the inventory control alarms at department store doorways if they go off. I know I've paid, and little sirens are just a nuisance at this point. This is why I've never stopped for receipt checks, because it's my receipt, and I've paid. The security theatre is just bad comedy. Just because the machine says I've done a no no, doesn't mean I can't come back and win a lawsuit later. It doesn't abdicate cops from their jobs. I have a winning complexion, so I'll never enjoy a false positive, but if I do, I'll make sure it bankrupts whatever startup incubator garbage decided to shill a replacement for real law enforcement.
- demadog 6y ago
  So out of curiosity, you just roll out of Costco with a cart full of food and gear while the receipt checker tries to stop you?
  - justinhj 6y ago
    I believe in a regular store you can just roll by the security without letting them check your stuff. They can make a citizens arrest or call the cops if they think you stole something, but at great risk of a lawsuit if they are wrong. However, Costco is a private club. You agree to their terms and conditions as a member of that club, and you must abide by the receipt check or they can ask you to leave. That was my understanding of the situation a decade ago, things may have changed.
  - raincom 6y ago
    At most Costco can cancel his membership. In other stores, if they find regular abusers, they can get restraining orders. Fry's electronics used to do that against a few customers.
- wilsonrocks 6y ago
  Is this an attitude that is safe for people of all races, though? Also, can everyone afford to pursue lawsuits?
  - stronglikedan 6y ago
    > Is this an attitude that is safe for people of all races, though? Yes, it is. Security cannot stop you for bypassing alarms and receipt checks. They have to have definitive proof that you stole something before they can lay a hand on you. Even in membership stores like Costco, the most they can do is cancel your membership. If they do touch you, there are plenty of lawyers who will take your case and only collect payment if you win.
    - runako 6y ago
      Theory & Law != What Actually Happens > If they do touch you, there are plenty of lawyers who will take your case and only collect payment if you win. This falls squarely into the genre of "yes, you are technically right, but you may have spent a week in jail and thousands to tens of thousands of dollars of time and money to prove it, for which you will not be fully compensated."
      - stronglikedan 6y ago
        My point is that anyone can walk by and nothing will likely happen, but if it does, there is some recourse, and the unlikely event is just as unlikely regardless of race.
        jacobush 6y ago
        That’s just false. Brown people get shot in malls for much less.
        stronglikedan 6y ago
        > Brown people get shot in malls for much les Not by security or police, so my point still stands.
m0zg 6y ago
> The trouble is not that the AI can be wrong Exactly what I thought when I've read about this. It's not like humans are great at matching faces either. In fact machines have been better at facial recognition for over a decade now. I bet there are hundreds of people (of all races) in prison right now who are there simply because they were mis-identified by a human. Human memory, even in the absence of bias and prejudice, is pretty fallible. There is a notion of "mixture of experts" in machine learning. It's when you have two or more models that are not, by themselves, sufficiently good to make a robust prediction, but that make different kinds of mistakes, and you use the consensus estimate. The resulting estimate will be better than any model in isolation. The same should be done here - AI should be merely a signal, it is not a replacement for detective work, and what's described in the article is just bad policing. AI has very little to do with that.
ineedasername 6y ago
Can you add any details about how an automated system was used to fire you? I'm not familiar with systems like that.
- sillysaurusx 6y ago
  Seconded! Fascinating situation.
  - draw_down 6y ago
    Not fascinating really, just incredibly shitty.
- ibudiallo 6y ago
  We had a whole discussion about it here a couple years ago: https://news.ycombinator.com/item?id=17350645 https://news.ycombinator.com/item?id=17350645
  - ineedasername 6y ago
    Thanks, that's insane. Sometimes I look at the HR group where I work and I'm astonished at how much (relatively easy to automate) work is still done manually or semi-manually. For example, today I had to send an email to a specific individual for what should be a simple form in the modern & best-of-breed HR system we use. After reading your story, I am very glad that we probably have in aggregate 2 or 3 full-time employees doing things that might be automated away. It's not like that prevents mindless bureaucracy of all sorts, but something like your situation would certainly never happen.
zamalek 6y ago
52% is little better than a coin flip. If you have a million individuals in your city, your confidence should be in the ballpark of 99.9999% (1 individual in 1 million). That has really been my concern with this, the software will report any facial match above 75% confidence. Apart from the fact that it appalling confidence, no cop will pay attention to that percentage; immediately arresting or killing the individual. Software can kill. This software can kill 50% of black people.
- ssss11 6y ago
  What a great comment. This encapsulates my concerns about the topic eloquently. The technology is not ready for use.
- cortesoft 6y ago
  It would have to be even higher than that level of accuracy, because every person is going to be 'tested' multiple times.... if everyone's face is scanned 100 times a day, the number of false positives is going to be even higher.
  - bigfudge 6y ago
    We shouldn’t assume those tests and errors are independent, they probably aren’t, but you are right that the overall error rate would be inflated.
- dtwest 6y ago
  Software can kill if we put blind trust in it and give it full control over the situation. But we shouldn't do that. Even if it was correct 99% of the time, we need to recognize that software can make mistakes. It is a tool, and people need to be responsible enough to use it correctly. I think I agree with your general idea here, but to put all of the blame on software strikes me as an incomplete assessment. Technically the software isn't killing anyone, irresponsible users of it are.
  - ArtDev 6y ago
    Any software developer will tell how the marketing and sales departments will say or spin ANYTHING they can get away with to sell the product.
99_00 6y ago
>But that's not how it is being sold to law enforcement or other entities. It's _Reduce crime in your cities. Catch criminals in ways never thought possible. Catch terrorists before they blow up anything._ It is sold as an ultimate decision maker. None of those selling points logically lead to the conclusion that it is the ultimate decisions maker.
yread 6y ago
You see it everywhere with AI and other tools. We overly trust them. Even when doctors have a high confidence in their diagnosis, they accept wrong AI-recommended conclusion that contradicts it. https://www.nature.com/articles/s41591-020-0942-0 https://www.nature.com/articles/s41591-020-0942-0 Bit like with self driving cars - if it's not perfect we don't know how to integrate it with people
- koz_ 6y ago
  That's interesting. I can imagine in cases like this it's not necessarily that the doctor doubts their own diagnosis, but rather the AI is essentially offering to relieve them of responsibility for it either way. It's like in human hierarchies - it's often not the person who is more likely to make the best decision who gets to decide, it's the one who is going to bear the consequences of being wrong.
air7 6y ago
The problem is not with the technology, but with how it's used. A medical test is also not 100% error-proof which is why a professional needs to interpret the results, sometimes conducting other tests or disregarding it completely. A cop stopping someone that has a resemblance to a criminal for questioning seems like a good thing to me, as long as the cop knows that there's a reasonable chance it's the wrong guy.
sgt101 6y ago
Worse - an AI decision puts an obligation on the user to follow it. What do I mean? Well - imagine you are a cop, you get an auto flag to arrest someone and use your discretion to overide it. The person goes on to do something completely different; like they are flagged as a murderer but then go and kill someone DUI. You will be flayed, pilloried. So basically safety first, just do the arrest. The secret is that these systems should not be making calls in this kind of context because they just aren't going to be good enough, it's like cancer diagnosis - the oncologist should have the first say, the machine should be a safety net.

nimbius 6y ago

This shit right here. This is why I don't stop for the inventory control alarms at department store doorways if they go off. I know I've paid, and little sirens are just a nuisance at this point. This is why I've never stopped for receipt checks, because it's my receipt, and I've paid. The security theatre is just bad comedy. Just because the machine says I've done a no no, doesn't mean I can't come back and win a lawsuit later. It doesn't abdicate cops from their jobs. I have a winning complexion, so I'll never enjoy a false positive, but if I do, I'll make sure it bankrupts whatever startup incubator garbage decided to shill a replacement for real law enforcement.

demadog 6y ago
So out of curiosity, you just roll out of Costco with a cart full of food and gear while the receipt checker tries to stop you?
- justinhj 6y ago
  I believe in a regular store you can just roll by the security without letting them check your stuff. They can make a citizens arrest or call the cops if they think you stole something, but at great risk of a lawsuit if they are wrong. However, Costco is a private club. You agree to their terms and conditions as a member of that club, and you must abide by the receipt check or they can ask you to leave. That was my understanding of the situation a decade ago, things may have changed.
- raincom 6y ago
  At most Costco can cancel his membership. In other stores, if they find regular abusers, they can get restraining orders. Fry's electronics used to do that against a few customers.
wilsonrocks 6y ago
Is this an attitude that is safe for people of all races, though? Also, can everyone afford to pursue lawsuits?
- stronglikedan 6y ago
  > Is this an attitude that is safe for people of all races, though? Yes, it is. Security cannot stop you for bypassing alarms and receipt checks. They have to have definitive proof that you stole something before they can lay a hand on you. Even in membership stores like Costco, the most they can do is cancel your membership. If they do touch you, there are plenty of lawyers who will take your case and only collect payment if you win.
  - runako 6y ago
    Theory & Law != What Actually Happens > If they do touch you, there are plenty of lawyers who will take your case and only collect payment if you win. This falls squarely into the genre of "yes, you are technically right, but you may have spent a week in jail and thousands to tens of thousands of dollars of time and money to prove it, for which you will not be fully compensated."
    - stronglikedan 6y ago
      My point is that anyone can walk by and nothing will likely happen, but if it does, there is some recourse, and the unlikely event is just as unlikely regardless of race.
      - jacobush 6y ago
        That’s just false. Brown people get shot in malls for much less.
        stronglikedan 6y ago
        > Brown people get shot in malls for much les Not by security or police, so my point still stands.

m0zg 6y ago

> The trouble is not that the AI can be wrong Exactly what I thought when I've read about this. It's not like humans are great at matching faces either. In fact machines have been better at facial recognition for over a decade now. I bet there are hundreds of people (of all races) in prison right now who are there simply because they were mis-identified by a human. Human memory, even in the absence of bias and prejudice, is pretty fallible. There is a notion of "mixture of experts" in machine learning. It's when you have two or more models that are not, by themselves, sufficiently good to make a robust prediction, but that make different kinds of mistakes, and you use the consensus estimate. The resulting estimate will be better than any model in isolation. The same should be done here - AI should be merely a signal, it is not a replacement for detective work, and what's described in the article is just bad policing. AI has very little to do with that.

ineedasername 6y ago

Can you add any details about how an automated system was used to fire you? I'm not familiar with systems like that.

sillysaurusx 6y ago
Seconded! Fascinating situation.
- draw_down 6y ago
  Not fascinating really, just incredibly shitty.
ibudiallo 6y ago
We had a whole discussion about it here a couple years ago: https://news.ycombinator.com/item?id=17350645 https://news.ycombinator.com/item?id=17350645
- ineedasername 6y ago
  Thanks, that's insane. Sometimes I look at the HR group where I work and I'm astonished at how much (relatively easy to automate) work is still done manually or semi-manually. For example, today I had to send an email to a specific individual for what should be a simple form in the modern & best-of-breed HR system we use. After reading your story, I am very glad that we probably have in aggregate 2 or 3 full-time employees doing things that might be automated away. It's not like that prevents mindless bureaucracy of all sorts, but something like your situation would certainly never happen.

zamalek 6y ago

52% is little better than a coin flip. If you have a million individuals in your city, your confidence should be in the ballpark of 99.9999% (1 individual in 1 million). That has really been my concern with this, the software will report any facial match above 75% confidence. Apart from the fact that it appalling confidence, no cop will pay attention to that percentage; immediately arresting or killing the individual. Software can kill. This software can kill 50% of black people.

ssss11 6y ago
What a great comment. This encapsulates my concerns about the topic eloquently. The technology is not ready for use.
cortesoft 6y ago
It would have to be even higher than that level of accuracy, because every person is going to be 'tested' multiple times.... if everyone's face is scanned 100 times a day, the number of false positives is going to be even higher.
- bigfudge 6y ago
  We shouldn’t assume those tests and errors are independent, they probably aren’t, but you are right that the overall error rate would be inflated.
dtwest 6y ago
Software can kill if we put blind trust in it and give it full control over the situation. But we shouldn't do that. Even if it was correct 99% of the time, we need to recognize that software can make mistakes. It is a tool, and people need to be responsible enough to use it correctly. I think I agree with your general idea here, but to put all of the blame on software strikes me as an incomplete assessment. Technically the software isn't killing anyone, irresponsible users of it are.
- ArtDev 6y ago
  Any software developer will tell how the marketing and sales departments will say or spin ANYTHING they can get away with to sell the product.

99_00 6y ago

>But that's not how it is being sold to law enforcement or other entities. It's _Reduce crime in your cities. Catch criminals in ways never thought possible. Catch terrorists before they blow up anything._ It is sold as an ultimate decision maker. None of those selling points logically lead to the conclusion that it is the ultimate decisions maker.

yread 6y ago

You see it everywhere with AI and other tools. We overly trust them. Even when doctors have a high confidence in their diagnosis, they accept wrong AI-recommended conclusion that contradicts it. https://www.nature.com/articles/s41591-020-0942-0 https://www.nature.com/articles/s41591-020-0942-0 Bit like with self driving cars - if it's not perfect we don't know how to integrate it with people

koz_ 6y ago
That's interesting. I can imagine in cases like this it's not necessarily that the doctor doubts their own diagnosis, but rather the AI is essentially offering to relieve them of responsibility for it either way. It's like in human hierarchies - it's often not the person who is more likely to make the best decision who gets to decide, it's the one who is going to bear the consequences of being wrong.

air7 6y ago

The problem is not with the technology, but with how it's used. A medical test is also not 100% error-proof which is why a professional needs to interpret the results, sometimes conducting other tests or disregarding it completely. A cop stopping someone that has a resemblance to a criminal for questioning seems like a good thing to me, as long as the cop knows that there's a reasonable chance it's the wrong guy.

sgt101 6y ago

Worse - an AI decision puts an obligation on the user to follow it. What do I mean? Well - imagine you are a cop, you get an auto flag to arrest someone and use your discretion to overide it. The person goes on to do something completely different; like they are flagged as a murderer but then go and kill someone DUI. You will be flayed, pilloried. So basically safety first, just do the arrest. The secret is that these systems should not be making calls in this kind of context because they just aren't going to be good enough, it's like cancer diagnosis - the oncologist should have the first say, the machine should be a safety net.