4 ms·

Yep, and this is a perfect example of a base rate fallacy situation... even if the scanner is 99.99% accurate, because an even higher percentage of photos are i

Yep, and this is a perfect example of a base rate fallacy situation... even if the scanner is 99.99% accurate, because an even higher percentage of photos are innocent, most matches the scanner will find will be false positives.

dtj1123 3mo ago
I thought this was known as Bonferonni'a principle? Or am I getting mixed up?
- vaylian 3mo ago
  Bonferonni correction is relevant when you calculate multiple p-values. Most statistical tests are used with a p-value threshold of 5% to reject the null-hypothesis. But because you are repeatedly testing, the probability for false positives increases and that is why you need to decrease the threshold and make it harder, to obtain a p-value below that threshold to declare a significant result. You typically use the Bonferroni correction when making general statements about a statistical relationship. You wouldn't use it for checking if a particular image shows illegal content. If you kept testing with your image classifier, your significance threshold would need to be continuously lowered and you would asymptotically reach zero. Relevant XKCD: 882
  - dtj1123 3mo ago
    No, I'm not referring to Bonferroni's correction, which obviously has nothing to do with what we're discussing. I'm sure I've heard this called Bonferroni's principle. Edit: Yes, in the section 'Statistical Limits of Data Mining' in the book Mining of mMssive Data Sets it's called Bonferroni's principle.
wesammikhail 3mo ago
Funny you bring this up. Back in the day when I was like 15 and DC++ was still a thing, I used to browse people's shared folders. One day I came across a file called "the paradox of false positive". It was a 1 pager that described how a machine which is 99.9% accurate at identifying terrorists would be completely useless due to this false positive base rate fallacy you're describing. It really stuck with me throughout the years. It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale. Which begs the question: lets assume the intentions are pure (which we know they're not but lets be generous), what other options are there when 99.9% heuristic is not good enough? how do you design systems when they're guaranteed to fail as they scale up? edit: and what do you know, I just saw this as I scrolled down on HN https://news.ycombinator.com/item?id=48816959 https://news.ycombinator.com/item?id=48816959
- m12k 3mo ago
  The intuition I've built is that you can't talk about a false positive rate being high or low on its own - it's always relative to the actual occurrence rate of positives in the tested population. E.g. if there's a 1 in 10000 risk of a false positive, but real positives also are only 1 out of 10000 tested cases, then a positive case will have a 50/50 chance of being a false positive (because for every 10000 tests, you'll have on average one false positive and one real positive). So a false positive rate can only be said to be low if it's significantly lower than the real occurrence rate of positives.
  - ablob 3mo ago
    The mentioned accuracy in the comment you are replying to already encapsulates the relation of true positives to false positives.
    - fc417fc802 3mo ago
      No I don't believe it does. I interpret 99.9% accurate to mean 1 in 1000 false positives. If 0.1% of your population are terrorists that means each alert has a 50% chance of being correct. That's nowhere near good enough to fully automate things but it is quite reasonable assuming this is merely information provided to a human agent. Whereas if only 0.001% of your population are terrorists then 99 out of 100 alerts are false positives at which point the system is well on its way to being useless. There is an important difference between scenarios where we care about the relative versus absolute frequency of errors.
      - ablob 3mo ago
        You're right it doesn't. At least not completely. I was thinking about precision (i.e.: if the test is positive, what are the odds that its prediction is true). It turns out, that accuracy is not defined as "true positive / (true pos. + true neg.)", but "correct predictions / all predictions". The whole point of OP's statement: "It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.", which you actually support with your example. > There is an important difference between scenarios where we care about the relative versus absolute frequency of errors. The context is chat control without probable cause over the whole population of Europe with a low prevalence. My point, and presumably that of OP, is that even a small relative frequency of errors will yield an unsustainably high absolute frequncy of errors. > This is merely information provided to a human agent. It will be in theory. In practice the human agent will just forward the decision. A human agent is not sufficient; you need to test only with probable cause for the kind of scenario we're talking about. The exact opposite of "Chat Control 1.0 and 2.0". P.S.: The comment I originally replied to choose a very convoluted way of saying that the false discovery rate of the test matters for a proper evaluation. Both you and they explain this by throwing numbers without context in combination with slightly inaccurate definitions. I got the definitions mixed up differently, which led to this follow-up.
kleiba2 3mo ago
> even if the scanner is 99.99% accurate, because an even higher percentage of photos are innocent, most matches the scanner will find will be false positives. If the scanner is 99.99% accurate, then most classifications will be correct.
- cenamus 3mo ago
  If you scan 1,000,000 pictures (with let's say 10 CSAM), you'll have 100 false positives and 10 true positives, giving you like only 10% correct results
  - kleiba2 3mo ago
    Ah, sorry, I misread what the OP meant by "matches" - thought they were referring to all classifier outputs, while they specifically meant the positives. I changed my original comment to better reflect what I meant, even though that makes it a bit of a non-sequitur now.
- usrnm 3mo ago
  How many child abusers do you think there are out there?
  - myrmidon 3mo ago
    Even if 10% of population were actively criminal pedos (which is waaaay too high), its pretty safe to assume that the majority of even their online footprint would be ordinary images/messages. So a quota of 0.1% or even less material being detectably criminal sounds realistic (probably not much less, though).
- acksmack 3mo ago
  https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy
tjpnz 3mo ago
Google have already caused significant hardship to a father for such kinds of photos. What's particularly galling is how they've continued to maintain they were in the right, despite the police saying no crime had been committed. https://www.koffellaw.com/blog/google-ai-technology-flags-dad-who-took-photos-o/ https://www.koffellaw.com/blog/google-ai-technology-flags-da...
- joe_mamba 3mo ago
  Of course google and every other big-tech platform is gonna insta-wipe every account containing detected nudes of children, regardless if you're the parent. The corporate liability of such content being found on their cloud is so insanely nuclear, that they're not gonna wait and ask you "hey are those nudes your own kids or are you a pedo?" before they wipe the account with all pics off their servers.
  - zmmmmm 3mo ago
    And yet the will badger you endlessly to the point their photos app is near unusable to turn on auto sync which slurps up every photo and makes it very awkward to then delete them after. To me, this makes Google a liable party even if real CSAM is stored.

dtj1123 3mo ago

I thought this was known as Bonferonni'a principle? Or am I getting mixed up?

vaylian 3mo ago
Bonferonni correction is relevant when you calculate multiple p-values. Most statistical tests are used with a p-value threshold of 5% to reject the null-hypothesis. But because you are repeatedly testing, the probability for false positives increases and that is why you need to decrease the threshold and make it harder, to obtain a p-value below that threshold to declare a significant result. You typically use the Bonferroni correction when making general statements about a statistical relationship. You wouldn't use it for checking if a particular image shows illegal content. If you kept testing with your image classifier, your significance threshold would need to be continuously lowered and you would asymptotically reach zero. Relevant XKCD: 882
- dtj1123 3mo ago
  No, I'm not referring to Bonferroni's correction, which obviously has nothing to do with what we're discussing. I'm sure I've heard this called Bonferroni's principle. Edit: Yes, in the section 'Statistical Limits of Data Mining' in the book Mining of mMssive Data Sets it's called Bonferroni's principle.

wesammikhail 3mo ago

Funny you bring this up. Back in the day when I was like 15 and DC++ was still a thing, I used to browse people's shared folders. One day I came across a file called "the paradox of false positive". It was a 1 pager that described how a machine which is 99.9% accurate at identifying terrorists would be completely useless due to this false positive base rate fallacy you're describing. It really stuck with me throughout the years. It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale. Which begs the question: lets assume the intentions are pure (which we know they're not but lets be generous), what other options are there when 99.9% heuristic is not good enough? how do you design systems when they're guaranteed to fail as they scale up? edit: and what do you know, I just saw this as I scrolled down on HN https://news.ycombinator.com/item?id=48816959 https://news.ycombinator.com/item?id=48816959

m12k 3mo ago
The intuition I've built is that you can't talk about a false positive rate being high or low on its own - it's always relative to the actual occurrence rate of positives in the tested population. E.g. if there's a 1 in 10000 risk of a false positive, but real positives also are only 1 out of 10000 tested cases, then a positive case will have a 50/50 chance of being a false positive (because for every 10000 tests, you'll have on average one false positive and one real positive). So a false positive rate can only be said to be low if it's significantly lower than the real occurrence rate of positives.
- ablob 3mo ago
  The mentioned accuracy in the comment you are replying to already encapsulates the relation of true positives to false positives.
  - fc417fc802 3mo ago
    No I don't believe it does. I interpret 99.9% accurate to mean 1 in 1000 false positives. If 0.1% of your population are terrorists that means each alert has a 50% chance of being correct. That's nowhere near good enough to fully automate things but it is quite reasonable assuming this is merely information provided to a human agent. Whereas if only 0.001% of your population are terrorists then 99 out of 100 alerts are false positives at which point the system is well on its way to being useless. There is an important difference between scenarios where we care about the relative versus absolute frequency of errors.
    - ablob 3mo ago
      You're right it doesn't. At least not completely. I was thinking about precision (i.e.: if the test is positive, what are the odds that its prediction is true). It turns out, that accuracy is not defined as "true positive / (true pos. + true neg.)", but "correct predictions / all predictions". The whole point of OP's statement: "It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.", which you actually support with your example. > There is an important difference between scenarios where we care about the relative versus absolute frequency of errors. The context is chat control without probable cause over the whole population of Europe with a low prevalence. My point, and presumably that of OP, is that even a small relative frequency of errors will yield an unsustainably high absolute frequncy of errors. > This is merely information provided to a human agent. It will be in theory. In practice the human agent will just forward the decision. A human agent is not sufficient; you need to test only with probable cause for the kind of scenario we're talking about. The exact opposite of "Chat Control 1.0 and 2.0". P.S.: The comment I originally replied to choose a very convoluted way of saying that the false discovery rate of the test matters for a proper evaluation. Both you and they explain this by throwing numbers without context in combination with slightly inaccurate definitions. I got the definitions mixed up differently, which led to this follow-up.

kleiba2 3mo ago

> even if the scanner is 99.99% accurate, because an even higher percentage of photos are innocent, most matches the scanner will find will be false positives. If the scanner is 99.99% accurate, then most classifications will be correct.

cenamus 3mo ago
If you scan 1,000,000 pictures (with let's say 10 CSAM), you'll have 100 false positives and 10 true positives, giving you like only 10% correct results
- kleiba2 3mo ago
  Ah, sorry, I misread what the OP meant by "matches" - thought they were referring to all classifier outputs, while they specifically meant the positives. I changed my original comment to better reflect what I meant, even though that makes it a bit of a non-sequitur now.
usrnm 3mo ago
How many child abusers do you think there are out there?
- myrmidon 3mo ago
  Even if 10% of population were actively criminal pedos (which is waaaay too high), its pretty safe to assume that the majority of even their online footprint would be ordinary images/messages. So a quota of 0.1% or even less material being detectably criminal sounds realistic (probably not much less, though).
acksmack 3mo ago
https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy

tjpnz 3mo ago

Google have already caused significant hardship to a father for such kinds of photos. What's particularly galling is how they've continued to maintain they were in the right, despite the police saying no crime had been committed. https://www.koffellaw.com/blog/google-ai-technology-flags-dad-who-took-photos-o/ https://www.koffellaw.com/blog/google-ai-technology-flags-da...

joe_mamba 3mo ago
Of course google and every other big-tech platform is gonna insta-wipe every account containing detected nudes of children, regardless if you're the parent. The corporate liability of such content being found on their cloud is so insanely nuclear, that they're not gonna wait and ask you "hey are those nudes your own kids or are you a pedo?" before they wipe the account with all pics off their servers.
- zmmmmm 3mo ago
  And yet the will badger you endlessly to the point their photos app is near unusable to turn on auto sync which slurps up every photo and makes it very awkward to then delete them after. To me, this makes Google a liable party even if real CSAM is stored.