7 ms·

How do new hashes get added to this database? How do we know that all the hashes are of CSAM? Who is validating it and is there an audit trail? Or can bad actor

How do new hashes get added to this database? How do we know that all the hashes are of CSAM? Who is validating it and is there an audit trail? Or can bad actors inject their own hashes into the database and make innocent people get reported as pedophiles?

fortenforge 5y ago
Apple has stated point-blank that the only source of CSAM content to generate the hash list will be NCMEC and other child safety organizations. While I fully admit that NCMEC could do a better job with transparency and auditing, they are currently being used by several other platforms right now (Facebook, Google, Microsoft) without issue. Could bad actors inject hashes of non-CSAM content into the database somehow? Well even if they could do this, Apple employes human reviewers who must visually confirm that the flagged photo contains actual CSAM before they report the image. If the image does not contain CSAM, Apple is under no legal obligation to report it. More information here: https://twitter.com/AlexMartin/status/1424703642913935374/photo/1 https://twitter.com/AlexMartin/status/1424703642913935374/ph...
- cdolan 5y ago
  Wait a second. You’re telling me that you believe a low-wage worker reviewing the worst of the worst human depravity, is going to stand up on a soap box and defend another nameless and faceless denizen of the Earth, when the crux of the argument is basically this: “Yeah I know the person tripped the safety threshold for CSAM, but these images aren’t that bad!” It’s not reasonable to trust the human reviewer. They put themselves at risk by going against the automated system. I’d bet 95% of people in this circumstance would pass the buck to the FBI to make their determination, at which point “your” life is already ruined.
  - simondotau 5y ago
    This is a misunderstanding of the system. It's not a classifier, it's a hash. The only images that are going to be seen by this low-wage worker are going to be: A) Images which are a correct hash match to an image already known to NCMEC or other agencies which have already been assigned CSAM category A1 (A=prepubescent, B=sex act); B) Images which are a hash collision. According to Apple, the likelihood of a collision is 1 in 1 trillion per user account. This system isn't a child detector strapped to a porn detector, being backed up by a low-wage worker making legal or editorial judgement calls. It's searching for images already known to child safety organisations—and even then only the most unambiguously horrific classification within the set of known images, far far far beyond the point where any ambiguity could possibly reside.
    - deepspace 5y ago
      > This is a misunderstanding of the system. It's not a classifier, it's a hash You know that a perceptual hash has more in common with a classifier than with a cryptographic hash, right? If they were using a crypto-hash, then, yes, your argument could be valid, but with perceptual hashing your picture of your baby in the bath could VERY well generate the same hash as a CSAM image. And nobody believes Apples "1 in a trillion" number.
      - simondotau 5y ago
        I'd be surprised if a baby in the bath—or indeed any legitmately innocent photograph taken by a parent—could ever be a perceptual match to images tagged as "A1" by NCMEC and other agencies. This is the most extreme of the extreme: prepubescent minors actively engaged in sex acts.
        farmerstan 5y ago
        I’d be surprised that a YouTube video of white noise would be flagged for a copyright violation, and yet here we are. Things may work right now in 2021. Things will always be changing. New hashes will be introduced. New code will be introduce. New laws in different countries will be introduced. Now that Apple has introduced this technology that no other phone manufacturer has, it can and will be changed to decrease privacy and increase government control. If China passes a law that states that IPhone needs to scan everyone’s phones for anti-government material, do you really think Tim Cook will say no? They already acquiesced about storing iCloud backups unencrypted on Chinese servers. And before you say that China would never do that, China and Hong Kong are hunting down the people who were videoed booing Chinese anthem in a shopping mall.
        deleted 5y ago
        [deleted]
        simondotau 5y ago
        > I’d be surprised that a YouTube video of white noise would be flagged for a copyright violation, and yet here we are. Speak for yourself, because that didn't surprise me at all. When your corpus of "copyrighted" material is so utterly massive and almost entirely devoid of defined rules or boundaries, this kind of error is inevitable. If anything I'm more surprised that we haven't seen even more of these kinds of matches, like the sound of generic telephone ringtones, recordings of mechanised church bells, or machinery which performs highly uniform tasks, etc. In the case of A1-categorised CSAM, the quantity of items is many orders of magnitude lower, the degree of technical curation will be orders of magnitude higher, and the thresholds for matches will be narrower. Is there a chance that the A1 corpus will have a few images that should have been classified as A2, B1 or B2? Yes. Is there a chance that it includes pictures of The Statue of Liberty or Westminster Tower? Almost certainly not.
- farmerstan 5y ago
  You are incredibly naive. Apple started out saying that they can't decrypt data from anyone's phone. They fought a lawsuit from the FBI over the San Bernadino terrorist phones. This is one of the reasons why I went all-in on Apple, because they were willing to fight the government over our privacy. Now, years later, they don't encrypt iCloud backups because the FBI told them not to. Google used to human review all copyright violations on Youtube. Fast forward a few years later, and all copyright violations are demonstrably shown to be approved and the content generator needs to prove that they didn't violate copyright. Look at the violations over white noice. Google doesn't even care anymore, they just let the copyright violations go through and affect the content creators with no review. To believe that Apple employees will review CSAM-flagged photos 5 years from now is so incredibly naive, it's actually funny. You can bet they are working on AI that will handle this for them right now in Cupertino. And then, it will be random chance whether or not we are flagged and labelled as pedophiles, or if a government wants to tag us as "problematic" and wants access to our phones because we are journalists and they want to see our anonymous sources. It's naive to think that it won't go in this direction. If you're a pedophile, after this announcement you will delete all the photos off your iPhone and never use it again. After the first few rounds of arrests and cleansing, it will be well knowing within the pedophile community not to use iPhones. And then the only ones who will be getting their photos scanned will be innocent people. So the entire feature doesn't make sense at all. It's a ruse.
stetrain 5y ago
There is a manual review step by Apple. You have to first get a “significant” number of photos flagged, then they have to pass Apple’s manual review (ie looking at photo thumbnails), and only then does Apple report the account.
- deleted 5y ago
  [deleted]
- nomel 5y ago
  I find how severely this is all being misunderstood, as in very few actually RTFM, on a site like HN, incredibly eye opening.
  - farmerstan 5y ago
    It’s eye opening how naive some people on HN are despite decades of evidence that things are always twisted in favor of more government surveillance especially outside of the US.
    - stetrain 5y ago
      If the worry is that a government can order Apple to do something, they could do that before or after this system was added just the same. Again I’m not in favor of this system but I think a lot of the criticism misses the mark.
      - farmerstan 5y ago
        Apple’s whole marketing scheme was that they protected privacy. That was their market differentiation. So no, it wasn’t like this before and this is why so many people like myself feel betrayed by Apple. This is what most of the outrage is about. The largest most profitable company in the world was selling their products saying they believed in privacy and now created a Trojan horse wrapped around CSAM that can be used by governments to spy on whatever is in our phones and I guess our laptops too. Maybe not today. But in 10 years? Yes definitely. They have already bent to government pressure since 2016 what would stop them going forward? Nothing. Tim Cook doesn’t want to go to Guantanamo for defying a government order.

fortenforge 5y ago

Apple has stated point-blank that the only source of CSAM content to generate the hash list will be NCMEC and other child safety organizations. While I fully admit that NCMEC could do a better job with transparency and auditing, they are currently being used by several other platforms right now (Facebook, Google, Microsoft) without issue. Could bad actors inject hashes of non-CSAM content into the database somehow? Well even if they could do this, Apple employes human reviewers who must visually confirm that the flagged photo contains actual CSAM before they report the image. If the image does not contain CSAM, Apple is under no legal obligation to report it. More information here: https://twitter.com/AlexMartin/status/1424703642913935374/photo/1 https://twitter.com/AlexMartin/status/1424703642913935374/ph...

cdolan 5y ago
Wait a second. You’re telling me that you believe a low-wage worker reviewing the worst of the worst human depravity, is going to stand up on a soap box and defend another nameless and faceless denizen of the Earth, when the crux of the argument is basically this: “Yeah I know the person tripped the safety threshold for CSAM, but these images aren’t that bad!” It’s not reasonable to trust the human reviewer. They put themselves at risk by going against the automated system. I’d bet 95% of people in this circumstance would pass the buck to the FBI to make their determination, at which point “your” life is already ruined.
- simondotau 5y ago
  This is a misunderstanding of the system. It's not a classifier, it's a hash. The only images that are going to be seen by this low-wage worker are going to be: A) Images which are a correct hash match to an image already known to NCMEC or other agencies which have already been assigned CSAM category A1 (A=prepubescent, B=sex act); B) Images which are a hash collision. According to Apple, the likelihood of a collision is 1 in 1 trillion per user account. This system isn't a child detector strapped to a porn detector, being backed up by a low-wage worker making legal or editorial judgement calls. It's searching for images already known to child safety organisations—and even then only the most unambiguously horrific classification within the set of known images, far far far beyond the point where any ambiguity could possibly reside.
  - deepspace 5y ago
    > This is a misunderstanding of the system. It's not a classifier, it's a hash You know that a perceptual hash has more in common with a classifier than with a cryptographic hash, right? If they were using a crypto-hash, then, yes, your argument could be valid, but with perceptual hashing your picture of your baby in the bath could VERY well generate the same hash as a CSAM image. And nobody believes Apples "1 in a trillion" number.
    - simondotau 5y ago
      I'd be surprised if a baby in the bath—or indeed any legitmately innocent photograph taken by a parent—could ever be a perceptual match to images tagged as "A1" by NCMEC and other agencies. This is the most extreme of the extreme: prepubescent minors actively engaged in sex acts.
      - farmerstan 5y ago
        I’d be surprised that a YouTube video of white noise would be flagged for a copyright violation, and yet here we are. Things may work right now in 2021. Things will always be changing. New hashes will be introduced. New code will be introduce. New laws in different countries will be introduced. Now that Apple has introduced this technology that no other phone manufacturer has, it can and will be changed to decrease privacy and increase government control. If China passes a law that states that IPhone needs to scan everyone’s phones for anti-government material, do you really think Tim Cook will say no? They already acquiesced about storing iCloud backups unencrypted on Chinese servers. And before you say that China would never do that, China and Hong Kong are hunting down the people who were videoed booing Chinese anthem in a shopping mall.
        deleted 5y ago
        [deleted]
        simondotau 5y ago
        > I’d be surprised that a YouTube video of white noise would be flagged for a copyright violation, and yet here we are. Speak for yourself, because that didn't surprise me at all. When your corpus of "copyrighted" material is so utterly massive and almost entirely devoid of defined rules or boundaries, this kind of error is inevitable. If anything I'm more surprised that we haven't seen even more of these kinds of matches, like the sound of generic telephone ringtones, recordings of mechanised church bells, or machinery which performs highly uniform tasks, etc. In the case of A1-categorised CSAM, the quantity of items is many orders of magnitude lower, the degree of technical curation will be orders of magnitude higher, and the thresholds for matches will be narrower. Is there a chance that the A1 corpus will have a few images that should have been classified as A2, B1 or B2? Yes. Is there a chance that it includes pictures of The Statue of Liberty or Westminster Tower? Almost certainly not.
farmerstan 5y ago
You are incredibly naive. Apple started out saying that they can't decrypt data from anyone's phone. They fought a lawsuit from the FBI over the San Bernadino terrorist phones. This is one of the reasons why I went all-in on Apple, because they were willing to fight the government over our privacy. Now, years later, they don't encrypt iCloud backups because the FBI told them not to. Google used to human review all copyright violations on Youtube. Fast forward a few years later, and all copyright violations are demonstrably shown to be approved and the content generator needs to prove that they didn't violate copyright. Look at the violations over white noice. Google doesn't even care anymore, they just let the copyright violations go through and affect the content creators with no review. To believe that Apple employees will review CSAM-flagged photos 5 years from now is so incredibly naive, it's actually funny. You can bet they are working on AI that will handle this for them right now in Cupertino. And then, it will be random chance whether or not we are flagged and labelled as pedophiles, or if a government wants to tag us as "problematic" and wants access to our phones because we are journalists and they want to see our anonymous sources. It's naive to think that it won't go in this direction. If you're a pedophile, after this announcement you will delete all the photos off your iPhone and never use it again. After the first few rounds of arrests and cleansing, it will be well knowing within the pedophile community not to use iPhones. And then the only ones who will be getting their photos scanned will be innocent people. So the entire feature doesn't make sense at all. It's a ruse.

stetrain 5y ago

There is a manual review step by Apple. You have to first get a “significant” number of photos flagged, then they have to pass Apple’s manual review (ie looking at photo thumbnails), and only then does Apple report the account.

deleted 5y ago
[deleted]
nomel 5y ago
I find how severely this is all being misunderstood, as in very few actually RTFM, on a site like HN, incredibly eye opening.
- farmerstan 5y ago
  It’s eye opening how naive some people on HN are despite decades of evidence that things are always twisted in favor of more government surveillance especially outside of the US.
  - stetrain 5y ago
    If the worry is that a government can order Apple to do something, they could do that before or after this system was added just the same. Again I’m not in favor of this system but I think a lot of the criticism misses the mark.
    - farmerstan 5y ago
      Apple’s whole marketing scheme was that they protected privacy. That was their market differentiation. So no, it wasn’t like this before and this is why so many people like myself feel betrayed by Apple. This is what most of the outrage is about. The largest most profitable company in the world was selling their products saying they believed in privacy and now created a Trojan horse wrapped around CSAM that can be used by governments to spy on whatever is in our phones and I guess our laptops too. Maybe not today. But in 10 years? Yes definitely. They have already bent to government pressure since 2016 what would stop them going forward? Nothing. Tim Cook doesn’t want to go to Guantanamo for defying a government order.