8 ms·

Innocent woman arrested in false-positive in facial recognition system in Brazil

93 points by gota 7y ago

An innocent woman was arrested today (while working in an informal job without any personally identifiable documentation) in Rio de Janeiro. The automated facial recognition systems (FRS) identified her as another woman, wanted for murder - it turns out the actual criminal is _already in jail_ and the police organization operating the FRS wasn't informed about it yet due to delays between the systems. This is the second day of operation of the system, according to the press. However, a restricted version of the same FRS was used for 15 days during the Carnaval festivities with no alarming failures and reportedly resulted in a few arrests. It has now been deployed in 25 locations in Copacabana. (Anedoctally, I live here and can't tell you where they are - the equipment, cameras, etc. must be well hidden, or I'm terrible at spotting them). Source, in Portuguese: "Facial recognition fails in its second day and innocent woman confused with criminal is arrested" https://oglobo.globo.com/rio/reconhecimento-facial-falha-em-segundo-dia-mulher-inocente-confundida-com-criminosa-ja-presa-23798913 Related previous submission (with zero comments): https://news.ycombinator.com/item?id=19074434

mikece 7y ago
As long as biometrics alone are the basis for an arrest, we'll see more and more of this. I'm curious what the accuracy score for the biometric source "evidence" was on which she was arrested, and what the match score was between her face and the source evidence.
- gtirloni 7y ago
  She wasn't arrested. She was detained and taken to the police station and her family brought her documents to prove she wasn't the other woman they were after.
  - Retric 7y ago
    Being detained is in the same category as being arrested. She was forcibly taken into custody only to be released without any form of compensation for lost time. In many countries people can be held for significant amounts of time without being formally charged with any crime.
    - gtirloni 7y ago
      Similar category but very different outcomes, rights, etc. https://criminal-law.freeadvice.com/criminal-law/arrests_and_searches/arrest-detention.htm https://criminal-law.freeadvice.com/criminal-law/arrests_and...
      - typenil 7y ago
        You're making a pedantic argument and backing it up with a link specific to the US court system. What the US supreme court says had no bearing in Brazil.
        gtirloni 7y ago
        No, I'm not. The Brazilian supreme court says pretty much the same thing.
        EForEndeavour 7y ago
        I feel you're missing the point entirely: to the person who was forcibly taken into custody by police and likely didn't have a clear and confident understanding of their rights (a fair assumption for the general populace, I'd say), it doesn't matter what you call it.
        gtirloni 7y ago
        Of course it's upsetting. I don't think anyone is making the case that it's just a regular day and you shouldn't complain. The automation system needs to improve, that's blatantly obvious. And the police officers need to learn to work with it. This is what happened (detention): "I was working and the police arrived. They confounded me with someone else they were looking for and I had to go to the police station and prove I wasn't that person. What a day". Not this (arrest): ""I was working and the police arrived. They confounded me with someone else they were looking for and I had to go to the police station and prove I wasn't that person. They still didn't believe me and I'm in a jail cell waiting for a judge to hear my case. I don't think I'll be home for a few days".
  - DanBC 7y ago
    > She wasn't arrested. > She was detained and taken to the police station What's that if it's not an arrest?
    - gtirloni 7y ago
      That's a detention. An arrest is when you're charged with a crime. It's a much more serious situation.
  - brandonmenc 7y ago
    > She wasn't arrested. She was detained Good thing no one is ever harmed while being detained but not yet arrested. /s
    - kabwj 7y ago
      Being detained doesn’t equal being beaten up.
ainar-g 7y ago
I severely dislike public facial recognition systems, but wouldn't it basically be the same if a police officer mistook her for someone else based on a sketch or a composite? The scale is obviously different though.
- mikece 7y ago
  Human cops don't have implicit faith in the accuracy of their human colleagues. But if a COMPUTER lists you as a biometric match for a person of interest/suspect, then clearly you did _something_...
  - jolmg 7y ago
    Right. Many people believe that computers can't make mistakes. It might not be true, but you'll still have a harder time convincing them that despite the computer identifying you, the computer is mistaken.
  - duxup 7y ago
    And humans are inclined to believe CSI type magic.
- dv_dt 7y ago
  With a sketch or composite, the identifying individual is the one with the full responsibility to make the match. With a recognition system, the software externalizes the responsibility, at best giving an individual the responsibility to reject the match. That setup results in the recognition system being inherently given a strong tendency to override individual judgement.
  - droithomme 7y ago
    > the software externalizes the responsibility The arresting officer still has the responsibility though to decide whether or not to use deadly force against the person should they resist the false arrest.
drtillberg 7y ago
And in other news, 25% of all stranger identifications by human eyewitnesses also were found to be erroneous.[1] [1] https://californiainnocenceproject.org/issues-we-face/eyewitness-identification/ https://californiainnocenceproject.org/issues-we-face/eyewit...
- xphilter 7y ago
  What’s the point of bringing that up? They’re not mutually exclusive. The difference though is that the accused is granted the right to test the veracity of the witness. Will the accused in cases like this be allowed to challenge the code? How can a poor person (who will undoubtedly be the one affected by this issue) afford that? And even if they can, the truth will be muddied by “experts” and a judge without any training or knowledge about facial recognition.
  - SiempreViernes 7y ago
    What is there to challenge, than you look like someone else?
    - xphilter 7y ago
      Or, as one example, that the algorithm was never tested on people of color and so the false positive rate is higher?
      - gtirloni 7y ago
        You're assuming the "algorithm" has the same faulty memory as witnesses. There are recorded images that can be triple checked by humans before any verdict is given (in the case of a conviction). A judge is not simply going "oh sorry, the system tells me it was you and I can't verify that. Gotta trust it".
        xphilter 7y ago
        You’re equally assuming plenty. If someone is arrested based on a false positive, they would likely be interrogated by the police. How crazy would it be for that same person to admit to a crime they didn’t do? There are plenty of hypos. My point is that there needs to be strong checks against the use of facial recognition. Citizens should see the algorithm, the training data, and results. The public should test the software rather than a poor criminal defendant.
deleted 7y ago
[deleted]
magwa101 7y ago
There's a first, wrong person apprehended
meruru 7y ago
I'm not concerned with false positives. That's something the legal system can expect and handle properly. My issue with this kind of technology is the potential for abuse by incumbent powers. What happens when the government starts using this to track whistleblowers?
sudoaza 7y ago
In Brazil simply being preto can get you in trouble with military police, can only imagine it has gotten much worse with bolzonaro
srameshc 7y ago
Pardon my ignorance, but isn't this a human rights issue ? Can the UN or other organizations not question the governments who are in such a rush to enforce these system regardless of all the evidence that it isn't reliable yet.
- mateus1 7y ago
  Yes it is. But under the current authoritarian Brazilian president "human rights" has been demonized. Rio de Janeiro's Governor has advocated for snipers to shoot-to-kill people identified as criminals in the favelas. It's horrible.

93 points by gota 7y ago

mikece 7y ago

As long as biometrics alone are the basis for an arrest, we'll see more and more of this. I'm curious what the accuracy score for the biometric source "evidence" was on which she was arrested, and what the match score was between her face and the source evidence.

gtirloni 7y ago
She wasn't arrested. She was detained and taken to the police station and her family brought her documents to prove she wasn't the other woman they were after.
- Retric 7y ago
  Being detained is in the same category as being arrested. She was forcibly taken into custody only to be released without any form of compensation for lost time. In many countries people can be held for significant amounts of time without being formally charged with any crime.
  - gtirloni 7y ago
    Similar category but very different outcomes, rights, etc. https://criminal-law.freeadvice.com/criminal-law/arrests_and_searches/arrest-detention.htm https://criminal-law.freeadvice.com/criminal-law/arrests_and...
    - typenil 7y ago
      You're making a pedantic argument and backing it up with a link specific to the US court system. What the US supreme court says had no bearing in Brazil.
      - gtirloni 7y ago
        No, I'm not. The Brazilian supreme court says pretty much the same thing.
        EForEndeavour 7y ago
        I feel you're missing the point entirely: to the person who was forcibly taken into custody by police and likely didn't have a clear and confident understanding of their rights (a fair assumption for the general populace, I'd say), it doesn't matter what you call it.
        gtirloni 7y ago
        Of course it's upsetting. I don't think anyone is making the case that it's just a regular day and you shouldn't complain. The automation system needs to improve, that's blatantly obvious. And the police officers need to learn to work with it. This is what happened (detention): "I was working and the police arrived. They confounded me with someone else they were looking for and I had to go to the police station and prove I wasn't that person. What a day". Not this (arrest): ""I was working and the police arrived. They confounded me with someone else they were looking for and I had to go to the police station and prove I wasn't that person. They still didn't believe me and I'm in a jail cell waiting for a judge to hear my case. I don't think I'll be home for a few days".
- DanBC 7y ago
  > She wasn't arrested. > She was detained and taken to the police station What's that if it's not an arrest?
  - gtirloni 7y ago
    That's a detention. An arrest is when you're charged with a crime. It's a much more serious situation.
- brandonmenc 7y ago
  > She wasn't arrested. She was detained Good thing no one is ever harmed while being detained but not yet arrested. /s
  - kabwj 7y ago
    Being detained doesn’t equal being beaten up.

ainar-g 7y ago

I severely dislike public facial recognition systems, but wouldn't it basically be the same if a police officer mistook her for someone else based on a sketch or a composite? The scale is obviously different though.

mikece 7y ago
Human cops don't have implicit faith in the accuracy of their human colleagues. But if a COMPUTER lists you as a biometric match for a person of interest/suspect, then clearly you did _something_...
- jolmg 7y ago
  Right. Many people believe that computers can't make mistakes. It might not be true, but you'll still have a harder time convincing them that despite the computer identifying you, the computer is mistaken.
- duxup 7y ago
  And humans are inclined to believe CSI type magic.
dv_dt 7y ago
With a sketch or composite, the identifying individual is the one with the full responsibility to make the match. With a recognition system, the software externalizes the responsibility, at best giving an individual the responsibility to reject the match. That setup results in the recognition system being inherently given a strong tendency to override individual judgement.
- droithomme 7y ago
  > the software externalizes the responsibility The arresting officer still has the responsibility though to decide whether or not to use deadly force against the person should they resist the false arrest.

drtillberg 7y ago

And in other news, 25% of all stranger identifications by human eyewitnesses also were found to be erroneous.[1] [1] https://californiainnocenceproject.org/issues-we-face/eyewitness-identification/ https://californiainnocenceproject.org/issues-we-face/eyewit...

xphilter 7y ago
What’s the point of bringing that up? They’re not mutually exclusive. The difference though is that the accused is granted the right to test the veracity of the witness. Will the accused in cases like this be allowed to challenge the code? How can a poor person (who will undoubtedly be the one affected by this issue) afford that? And even if they can, the truth will be muddied by “experts” and a judge without any training or knowledge about facial recognition.
- SiempreViernes 7y ago
  What is there to challenge, than you look like someone else?
  - xphilter 7y ago
    Or, as one example, that the algorithm was never tested on people of color and so the false positive rate is higher?
    - gtirloni 7y ago
      You're assuming the "algorithm" has the same faulty memory as witnesses. There are recorded images that can be triple checked by humans before any verdict is given (in the case of a conviction). A judge is not simply going "oh sorry, the system tells me it was you and I can't verify that. Gotta trust it".
      - xphilter 7y ago
        You’re equally assuming plenty. If someone is arrested based on a false positive, they would likely be interrogated by the police. How crazy would it be for that same person to admit to a crime they didn’t do? There are plenty of hypos. My point is that there needs to be strong checks against the use of facial recognition. Citizens should see the algorithm, the training data, and results. The public should test the software rather than a poor criminal defendant.

deleted 7y ago

[deleted]

magwa101 7y ago

There's a first, wrong person apprehended

meruru 7y ago

I'm not concerned with false positives. That's something the legal system can expect and handle properly. My issue with this kind of technology is the potential for abuse by incumbent powers. What happens when the government starts using this to track whistleblowers?

sudoaza 7y ago

In Brazil simply being preto can get you in trouble with military police, can only imagine it has gotten much worse with bolzonaro

srameshc 7y ago

Pardon my ignorance, but isn't this a human rights issue ? Can the UN or other organizations not question the governments who are in such a rush to enforce these system regardless of all the evidence that it isn't reliable yet.

mateus1 7y ago
Yes it is. But under the current authoritarian Brazilian president "human rights" has been demonized. Rio de Janeiro's Governor has advocated for snipers to shoot-to-kill people identified as criminals in the favelas. It's horrible.