6 ms·

Backup is a good point: - Apple: “Backup your phone to iCloud, it will be safe there.” - 5 minutes later: “We’ve wiped your account because of a photos of (po

Backup is a good point: - Apple: “Backup your phone to iCloud, it will be safe there.” - 5 minutes later: “We’ve wiped your account because of a photos of (porn actor here) which is not CP but technically minor at the time she filmed.” - “Also we’ve wiped your iPhone because we couldn’t knowingly let you keep that. Good luck contacting your parents, we’ve deleted your contacts. Good luck! PS: We’ve reported you to the police.” - Also you can’t connect to your iMac now.

annamargot 5y ago
Or photos of your own children. We have a Tumblr set up for family to view pics of the kids. Several photos and videos of our kids when they were under 2 were taken down either temporarily or permanently by their CP algo. These were a pic or video of kids in the bath or without a shirt. In none of them could you see bum or bits. Just a semi naked baby. Algorithms like this get things wrong all the time
- tpush 5y ago
  This is not the kind of algorithm that Apple is be using. That one only scans for already known CSAM in NCMEC's database.
  - shakna 5y ago
    Which may contain the hashes of their photos, because they've been taken down in the past, which means they probably have been added to certain blacklists that may have been integrated into the blackbox of NCMEC's database.
    - simondotau 5y ago
      Photographs of your naked child in the bath are not illegal, are not CSAM, and are not going to be in the NCMEC's database.
      - shakna 5y ago
        NCMEC's CSAM database already includes images that are not necessarily illegal. If _your particular_ photos have been flagged in the past, they may well be part of the database.
        culturestate 5y ago
        > NCMEC's CSAM database already includes images that are not necessarily illegal. How could this be the case? If it's been determined to be CSAM then it is, by definition, illegal. If it were true that the database is likely to contain legal material, how would we possibly know about it, given that the contents of the database are secret?
        shakna 5y ago
        > How could this be the case? If it's been determined to be CSAM then it is, by definition, illegal. Certain images are CSAM by _context_. They do not necessarily require those within the image to be abused, but rather that the image at one time or another was traded alongside other CSAM. > If it were true that the database is likely to contain legal material, how would we possibly know about it, given that the contents of the database are secret? Tools like Spotlight [0] make use of the database, so certain well-known images are known to flag. Such as Nirvana's controversial cover for Nevermind. [0] https://www.wired.com/story/how-facial-recognition-fighting-child-sex-trafficking/ https://www.wired.com/story/how-facial-recognition-fighting-...
        culturestate 5y ago
        > Certain images are CSAM by _context_. They do not necessarily require those within the image to be abused, but rather that the image at one time or another was traded alongside other CSAM. At the risk of sounding like a broken record, how can we know this is actually true? Every description of the NCMEC database's contents that I've seen is incredibly vague, and as of 2019 it seems like there were fewer than[1] 4 million total hashes available. I would think that if it genuinely did include innocent photos of people's kids, the number would be much higher. > ...certain well-known images are known to flag. Such as Nirvana's controversial cover for Nevermind. I've heard this multiples times now, but I've never been able to find any evidence of it actually happening. The only instance I could find was one where Facebook removed[2] that Nirvana cover once for containing nudity. 1. https://inews.co.uk/news/technology/uk-us-collaborate-crack-down-child-abuse-images-sharing-373919 https://inews.co.uk/news/technology/uk-us-collaborate-crack-... 2. https://www.theguardian.com/music/2011/jul/28/facebook-nirvana-album-cover-nevermind https://www.theguardian.com/music/2011/jul/28/facebook-nirva...
  - ElFitz 5y ago
    Quite funnily and disturbingly, one the databases of "known CSAM" hashes also apparently includes a picture of a clothed man holding a monkey[1] [1]: https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
    - severak_cz 5y ago
      That was just a MD5 collision - an image that has same MD5 hash as some other image (in this case some CP). This is uncommon yet possible thing - see this example[0]. [0] https://natmchugh.blogspot.com/2014/11/three-way-md5-collision.html https://natmchugh.blogspot.com/2014/11/three-way-md5-collisi...
      - tyingq 5y ago
        I think a flawed process where the monkey image ended up in the database is more likely than a random unintentional hash collision.
        zarzavat 5y ago
        Not really. MD5 is thoroughly and completely broken, and has been for years. You can modify an image to be an MD5 collision for another image.
        tyingq 5y ago
        That would be an intentional collision. An unintentional collision remains unlikely for a cryptographic hash.
        nullc 5y ago
        Not just unlikely but astronomically unlikely.
        nullc 5y ago
        No you cannot. A collision requires the attacker to create both images. What you are describing is a second preimage attack-- creating a second input with the same hash as a target. There is no currently known tractable way to create second preimages for MD5.
- simondotau 5y ago
  This isn't an ML algorithm. It's a hash. It only matches already known material.
  - samrolken 5y ago
    It is a hash created with ML. So it’s both. But yes, it only matches already known material.
    - simondotau 5y ago
      I haven't seen anyone claim that any of this algorithm was "created with ML". I'm interested in learning more so do you have a citation for that? Regardless, it's not both. Setting aside how the algorithm was created, it's incorrect to say that an algorithm "created with ML" is itself an ML algorithm. NeuralHash was so named because it was optimised to run on the Apple Neural Engine for the sake of speed and power efficiency.
      - samrolken 5y ago
        It’s both because it’s a multi-step process. The image is not fed directly into the hashing function, like taking an MD5 hash of a file or something. Rather, the image is first evaluated by a neural net that looks at specific visual details, and has been trained to match even if the image has been cropped or anything like that. The results of the neural net evaluation are what is then input for the hashing function. This is explained in detail in Apple’s documentation they released with the announcement.
bingidingi 5y ago
uhh how is it not cp if someone is a minor

annamargot 5y ago

Or photos of your own children. We have a Tumblr set up for family to view pics of the kids. Several photos and videos of our kids when they were under 2 were taken down either temporarily or permanently by their CP algo. These were a pic or video of kids in the bath or without a shirt. In none of them could you see bum or bits. Just a semi naked baby. Algorithms like this get things wrong all the time

tpush 5y ago
This is not the kind of algorithm that Apple is be using. That one only scans for already known CSAM in NCMEC's database.
- shakna 5y ago
  Which may contain the hashes of their photos, because they've been taken down in the past, which means they probably have been added to certain blacklists that may have been integrated into the blackbox of NCMEC's database.
  - simondotau 5y ago
    Photographs of your naked child in the bath are not illegal, are not CSAM, and are not going to be in the NCMEC's database.
    - shakna 5y ago
      NCMEC's CSAM database already includes images that are not necessarily illegal. If _your particular_ photos have been flagged in the past, they may well be part of the database.
      - culturestate 5y ago
        > NCMEC's CSAM database already includes images that are not necessarily illegal. How could this be the case? If it's been determined to be CSAM then it is, by definition, illegal. If it were true that the database is likely to contain legal material, how would we possibly know about it, given that the contents of the database are secret?
        shakna 5y ago
        > How could this be the case? If it's been determined to be CSAM then it is, by definition, illegal. Certain images are CSAM by _context_. They do not necessarily require those within the image to be abused, but rather that the image at one time or another was traded alongside other CSAM. > If it were true that the database is likely to contain legal material, how would we possibly know about it, given that the contents of the database are secret? Tools like Spotlight [0] make use of the database, so certain well-known images are known to flag. Such as Nirvana's controversial cover for Nevermind. [0] https://www.wired.com/story/how-facial-recognition-fighting-child-sex-trafficking/ https://www.wired.com/story/how-facial-recognition-fighting-...
        culturestate 5y ago
        > Certain images are CSAM by _context_. They do not necessarily require those within the image to be abused, but rather that the image at one time or another was traded alongside other CSAM. At the risk of sounding like a broken record, how can we know this is actually true? Every description of the NCMEC database's contents that I've seen is incredibly vague, and as of 2019 it seems like there were fewer than[1] 4 million total hashes available. I would think that if it genuinely did include innocent photos of people's kids, the number would be much higher. > ...certain well-known images are known to flag. Such as Nirvana's controversial cover for Nevermind. I've heard this multiples times now, but I've never been able to find any evidence of it actually happening. The only instance I could find was one where Facebook removed[2] that Nirvana cover once for containing nudity. 1. https://inews.co.uk/news/technology/uk-us-collaborate-crack-down-child-abuse-images-sharing-373919 https://inews.co.uk/news/technology/uk-us-collaborate-crack-... 2. https://www.theguardian.com/music/2011/jul/28/facebook-nirvana-album-cover-nevermind https://www.theguardian.com/music/2011/jul/28/facebook-nirva...
- ElFitz 5y ago
  Quite funnily and disturbingly, one the databases of "known CSAM" hashes also apparently includes a picture of a clothed man holding a monkey[1] [1]: https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
  - severak_cz 5y ago
    That was just a MD5 collision - an image that has same MD5 hash as some other image (in this case some CP). This is uncommon yet possible thing - see this example[0]. [0] https://natmchugh.blogspot.com/2014/11/three-way-md5-collision.html https://natmchugh.blogspot.com/2014/11/three-way-md5-collisi...
    - tyingq 5y ago
      I think a flawed process where the monkey image ended up in the database is more likely than a random unintentional hash collision.
      - zarzavat 5y ago
        Not really. MD5 is thoroughly and completely broken, and has been for years. You can modify an image to be an MD5 collision for another image.
        tyingq 5y ago
        That would be an intentional collision. An unintentional collision remains unlikely for a cryptographic hash.
        nullc 5y ago
        Not just unlikely but astronomically unlikely.
        nullc 5y ago
        No you cannot. A collision requires the attacker to create both images. What you are describing is a second preimage attack-- creating a second input with the same hash as a target. There is no currently known tractable way to create second preimages for MD5.
simondotau 5y ago
This isn't an ML algorithm. It's a hash. It only matches already known material.
- samrolken 5y ago
  It is a hash created with ML. So it’s both. But yes, it only matches already known material.
  - simondotau 5y ago
    I haven't seen anyone claim that any of this algorithm was "created with ML". I'm interested in learning more so do you have a citation for that? Regardless, it's not both. Setting aside how the algorithm was created, it's incorrect to say that an algorithm "created with ML" is itself an ML algorithm. NeuralHash was so named because it was optimised to run on the Apple Neural Engine for the sake of speed and power efficiency.
    - samrolken 5y ago
      It’s both because it’s a multi-step process. The image is not fed directly into the hashing function, like taking an MD5 hash of a file or something. Rather, the image is first evaluated by a neural net that looks at specific visual details, and has been trained to match even if the image has been cropped or anything like that. The results of the neural net evaluation are what is then input for the hashing function. This is explained in detail in Apple’s documentation they released with the announcement.

bingidingi 5y ago

uhh how is it not cp if someone is a minor