4 ms·

What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely. Your argument is essentially that a collision

by 0xy 5y ago

What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely. Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size. It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 300,000,000 entries. Because if they make one single mistake, then it supports my argument -- a false positive becomes substantially more likely statistically than a hash collision. You're also providing a pretty large red herring with your suggestion that he could've simply asked NCMEC if it was a false positive or a hash collision. NCMEC would never provide that information, because the database is highly secret. Given those statistics, I think that source is more than valid. >in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up I am happy to let you believe that I'm lying. One industry insider whose employer works with NCMEC cited a false positive rate of 1 in 1,000. [1] The product he works in is used in conjunction with NCMEC's database. Elsewhere, in press releases, the company cites a failure rate of 1% (presumably both false positives and false negatives) [2] [1] https://news.ycombinator.com/item?id=21446562 https://news.ycombinator.com/item?id=21446562 [2] https://www.prnewswire.com/news-releases/thorns-automated-tool-to-remove-child-abuse-content-at-scale-expands-to-more-platforms-through-aws-marketplace-301297111.html https://www.prnewswire.com/news-releases/thorns-automated-to...

simondotau 5y ago
Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data. I am merely pushing back at your evidence-free claim that "it's full of false positives." Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying. And now you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are clearly very confused.
- 0xy 5y ago
  >Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data If NCMEC's databases have a false positive rate of 1 in 1,000, do you realise that means a false positive rate is substantially more likely than a hash collision? >I am merely pushing back at your evidence-free claim that "it's full of false positives." 1 in 1,000 is "full of false positives" by my own standards, to which I've posted evidence from an employee whose company works directly with NCMEC. >you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". NCMEC and law enforcement are one and the same. FBI employees work directly at NCMEC. [1] Law enforcement have direct access to the database, including for categorisation purposes. To suggest law enforcement's dataset is tainted yet NCMEC's is not doesn't make any sense to me. >You are clearly very confused Address the 1 in 1,000 claim. Thorn is an NCMEC partner and a Thorn employee has claimed a false positive rate of 1 in 1,000. In press releases, Thorn is even less sure at 1% failure rates. Thorn uses perceptual hashing. I can't see how you can simultaneously claim the database isn't "full of false positives" while acknowledging a failure rate as abysmal as 1 in 1,000. I also didn't conflate anything, both a hash collision and a perceptual hash collision are less likely than a false positive, by an extraordinary margin. Apple claims their algorithm has a collision 1 in 1,000,000,000,000 times. Compare that to 1 in 1,000. The database is full of false positives, and now presumably you'll deny both industry claims and NCMEC partner claims. [1] https://www.fbi.gov/investigate/violent-crime/cac https://www.fbi.gov/investigate/violent-crime/cac
  - simondotau 5y ago
    Thorn isn't claiming that "the database is full of false positives." Once again you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are so catastrophically confused that ongoing dialogue serves no purpose. Goodbye.

by 0xy 5y ago

simondotau 5y ago

Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data. I am merely pushing back at your evidence-free claim that "it's full of false positives." Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying. And now you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are clearly very confused.

0xy 5y ago
>Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data If NCMEC's databases have a false positive rate of 1 in 1,000, do you realise that means a false positive rate is substantially more likely than a hash collision? >I am merely pushing back at your evidence-free claim that "it's full of false positives." 1 in 1,000 is "full of false positives" by my own standards, to which I've posted evidence from an employee whose company works directly with NCMEC. >you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". NCMEC and law enforcement are one and the same. FBI employees work directly at NCMEC. [1] Law enforcement have direct access to the database, including for categorisation purposes. To suggest law enforcement's dataset is tainted yet NCMEC's is not doesn't make any sense to me. >You are clearly very confused Address the 1 in 1,000 claim. Thorn is an NCMEC partner and a Thorn employee has claimed a false positive rate of 1 in 1,000. In press releases, Thorn is even less sure at 1% failure rates. Thorn uses perceptual hashing. I can't see how you can simultaneously claim the database isn't "full of false positives" while acknowledging a failure rate as abysmal as 1 in 1,000. I also didn't conflate anything, both a hash collision and a perceptual hash collision are less likely than a false positive, by an extraordinary margin. Apple claims their algorithm has a collision 1 in 1,000,000,000,000 times. Compare that to 1 in 1,000. The database is full of false positives, and now presumably you'll deny both industry claims and NCMEC partner claims. [1] https://www.fbi.gov/investigate/violent-crime/cac https://www.fbi.gov/investigate/violent-crime/cac
- simondotau 5y ago
  Thorn isn't claiming that "the database is full of false positives." Once again you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are so catastrophically confused that ongoing dialogue serves no purpose. Goodbye.