7 ms·

> According to Breyer, the existing voluntary system has already proven flawed, with German police reporting that roughly half of all flagged cases turn out to

> According to Breyer, the existing voluntary system has already proven flawed, with German police reporting that roughly half of all flagged cases turn out to be irrelevant. A failure rate of only 50% is absurdly good for a system like this. If we have to: > Imagine your phone scanning every conversation with your partner, your daughter, your therapist, and leaking it just because the word ‘love’ or ‘meet’ appears somewhere. then apparently either there are so many perpetrators that regular conversations with partners etc. are about as common as crime, or such regular conversations don't have such a high risk of being reported after all. I don't think chat surveillance is a good idea. But please use transparent and open communication. Don't manipulate us just like the enemy does.

Humorist2290 11mo ago
It is probably a reference to the report mentioned in this article from September https://reclaimthenet.org/germany-chat-control-false-reports-surveillance-encryption-privacy https://reclaimthenet.org/germany-chat-control-false-reports... According to the Federal Criminal Police Office (BKA), 99,375 of the 205,728 reports forwarded by the US-based National Center for Missing and Exploited Children (NCMEC) were not criminally relevant, an error rate of 48.3%. This is a rise from 2023, when the number of false positives already stood at 90,950. Indeed 50% false positive rate sounds surprisingly good, but this is under the "voluntary scheme" where Meta/Google/MS etc are not obligated to report. Notably missing from the article is the total number of scanned messages to get down to 200k reports. To my knowledge, since it's voluntary, they can also report only the very highest confidence detections. If the Danish regime were to impose reporting quotas the total number of reports would rise. And of course -- these are reports, not actually convictions. Presumably the actual number of criminals caught by this would remain constant, so the FP rate would increase. Unless of course, the definition of criminal expands to keep the FP rate low...
- b112 11mo ago
  I feel this is a good place to add something... I recall a half decade back, there was discussion of the quit rate of employees, maybe Facebook?, due to literal mental trauma from having to look at and validate pedophile flagged images. Understand there is pedophilia, then there's horribly violent, next level abusive pedophilia. I used to work in a department where, adjacently, the RCMP were doing the same. They couldn't handle it, and were constantly resigning. The violence associated with some of the videos and images is what really got them. The worst part is, the more empathetic you are, the more it hurts to work in this area. It seems to me that without this sad and damaging problem fixed, monitoring chats won't help much. How many good people, will we laden with trama, literally waking up screaming at night? It's why the RCMP officers were resigning. I can't imagine being a jury member at such a case.
  - Eddy_Viscosity2 11mo ago
    Because of this issue, many departments put in much stricter protocols for dealing with this kind of material. Only certain people would be exposed to classify/tag it, and these people would only hold that post of a limited period of time. The burden on those people doesn't change, but it can be diluted to mitigate it somewhat. Its a real and sad problem, but not one that I think can be fixed with technology. To much is on the line to allow for a false positive from a hallucinating robot to destroy a person(s) life.
  - qnleigh 11mo ago
    I read about that here: https://erinkissane.com/meta-in-myanmar-part-i-the-setup https://erinkissane.com/meta-in-myanmar-part-i-the-setup This remains one of the best things I've found on HN.
- gus_massa 11mo ago
  OK, 50% "not criminally relevant". How many of the other 50% were guilty and how many innocent after an investigation?
deleted 11mo ago
[deleted]
anonym29 11mo ago
[flagged]
- xethos 11mo ago
  Standing on the street holding a bag open, and catching a fish falling from the sky one time, would also be an absurdly good catch rate no matter how long I had to wait - it's just that shit of a system Only half the conversations being a waste of law enforcement's time is also an impressive rate - we all recognize that it's just a shite system that is bound to have a lot of false positives.
  - anonym29 11mo ago
    Fortunately, law enforcement was able to catch pedophiles and terrorists before they came up with the idea of permanently ending everyone's privacy, unlike your obviously ridiculous analogy - nobody has ever caught fish falling from the sky by standing in the street holding empty bags.
- ChadNauseam 11mo ago
  You're misinterpreting what they mean by "50% false positive rate". Let's say there are 10 child traffickers in the country. A 50% false positive rate might mean this system detects 7 of them, but then also detects 7 people who are not child traffickers. The reason it's an absurdly good rate is because the thing they are looking for is so rare. If one in a million people are child traffickers, and you have a false positive rate of 50%, it means that the average person has no more than a one in a million chance of being a false positive. Let me put it another way. Let's say you are an Egyptian Pharaoh and a man tells you he can predict when solar eclipses will happen. You give him a calendar and he marks down 20 days in the next few decades where he claims there will be an eclipse. On 10 of those marked days, there actually is an eclipse. That would be a 50% false positive rate. By the standards of an Egyptian Pharaoh, that probably looks like an absurdly good false positive rate.
  - rendaw 11mo ago
    I agree the GP math is wrong. But still, is this good? So 14 cases are referred to law enforcement. They catch 7 people doing really terrible things, and they also catch 7 who are innocent. The police are notorious for treating weak signals as strong and forging causes for suspicion, terrorizing suspects, getting them fired from their job, killing their pets, upending their lives, etc. If you ruin as many lives as you save, will you sleep well at night? This is why a 50% false positive rate is terrible and privacy advocates are against this. The consequences of a false positive for an eclipse when one is predicted is trivial, unless there's something about eclipses I don't know. Maybe there's some diffuse monetary loss across society.
beefnugs 11mo ago
Absurdly good? What are you talking about, it means entire company processes of trying to identify this has cost so much time and effort and tracking and lost trust from the public and finally reporting... and then they still screw it up half the time

Humorist2290 11mo ago

It is probably a reference to the report mentioned in this article from September https://reclaimthenet.org/germany-chat-control-false-reports-surveillance-encryption-privacy https://reclaimthenet.org/germany-chat-control-false-reports... According to the Federal Criminal Police Office (BKA), 99,375 of the 205,728 reports forwarded by the US-based National Center for Missing and Exploited Children (NCMEC) were not criminally relevant, an error rate of 48.3%. This is a rise from 2023, when the number of false positives already stood at 90,950. Indeed 50% false positive rate sounds surprisingly good, but this is under the "voluntary scheme" where Meta/Google/MS etc are not obligated to report. Notably missing from the article is the total number of scanned messages to get down to 200k reports. To my knowledge, since it's voluntary, they can also report only the very highest confidence detections. If the Danish regime were to impose reporting quotas the total number of reports would rise. And of course -- these are reports, not actually convictions. Presumably the actual number of criminals caught by this would remain constant, so the FP rate would increase. Unless of course, the definition of criminal expands to keep the FP rate low...

b112 11mo ago
I feel this is a good place to add something... I recall a half decade back, there was discussion of the quit rate of employees, maybe Facebook?, due to literal mental trauma from having to look at and validate pedophile flagged images. Understand there is pedophilia, then there's horribly violent, next level abusive pedophilia. I used to work in a department where, adjacently, the RCMP were doing the same. They couldn't handle it, and were constantly resigning. The violence associated with some of the videos and images is what really got them. The worst part is, the more empathetic you are, the more it hurts to work in this area. It seems to me that without this sad and damaging problem fixed, monitoring chats won't help much. How many good people, will we laden with trama, literally waking up screaming at night? It's why the RCMP officers were resigning. I can't imagine being a jury member at such a case.
- Eddy_Viscosity2 11mo ago
  Because of this issue, many departments put in much stricter protocols for dealing with this kind of material. Only certain people would be exposed to classify/tag it, and these people would only hold that post of a limited period of time. The burden on those people doesn't change, but it can be diluted to mitigate it somewhat. Its a real and sad problem, but not one that I think can be fixed with technology. To much is on the line to allow for a false positive from a hallucinating robot to destroy a person(s) life.
- qnleigh 11mo ago
  I read about that here: https://erinkissane.com/meta-in-myanmar-part-i-the-setup https://erinkissane.com/meta-in-myanmar-part-i-the-setup This remains one of the best things I've found on HN.
gus_massa 11mo ago
OK, 50% "not criminally relevant". How many of the other 50% were guilty and how many innocent after an investigation?

deleted 11mo ago

[deleted]

anonym29 11mo ago

[flagged]

xethos 11mo ago
Standing on the street holding a bag open, and catching a fish falling from the sky one time, would also be an absurdly good catch rate no matter how long I had to wait - it's just that shit of a system Only half the conversations being a waste of law enforcement's time is also an impressive rate - we all recognize that it's just a shite system that is bound to have a lot of false positives.
- anonym29 11mo ago
  Fortunately, law enforcement was able to catch pedophiles and terrorists before they came up with the idea of permanently ending everyone's privacy, unlike your obviously ridiculous analogy - nobody has ever caught fish falling from the sky by standing in the street holding empty bags.
ChadNauseam 11mo ago
You're misinterpreting what they mean by "50% false positive rate". Let's say there are 10 child traffickers in the country. A 50% false positive rate might mean this system detects 7 of them, but then also detects 7 people who are not child traffickers. The reason it's an absurdly good rate is because the thing they are looking for is so rare. If one in a million people are child traffickers, and you have a false positive rate of 50%, it means that the average person has no more than a one in a million chance of being a false positive. Let me put it another way. Let's say you are an Egyptian Pharaoh and a man tells you he can predict when solar eclipses will happen. You give him a calendar and he marks down 20 days in the next few decades where he claims there will be an eclipse. On 10 of those marked days, there actually is an eclipse. That would be a 50% false positive rate. By the standards of an Egyptian Pharaoh, that probably looks like an absurdly good false positive rate.
- rendaw 11mo ago
  I agree the GP math is wrong. But still, is this good? So 14 cases are referred to law enforcement. They catch 7 people doing really terrible things, and they also catch 7 who are innocent. The police are notorious for treating weak signals as strong and forging causes for suspicion, terrorizing suspects, getting them fired from their job, killing their pets, upending their lives, etc. If you ruin as many lives as you save, will you sleep well at night? This is why a 50% false positive rate is terrible and privacy advocates are against this. The consequences of a false positive for an eclipse when one is predicted is trivial, unless there's something about eclipses I don't know. Maybe there's some diffuse monetary loss across society.

beefnugs 11mo ago

Absurdly good? What are you talking about, it means entire company processes of trying to identify this has cost so much time and effort and tracking and lost trust from the public and finally reporting... and then they still screw it up half the time