4 ms·

1) Some dev team at Apple is writing the on device scanning code. 2) Some dev team at Apple is writing a "sendMessagePolice()" API. 3) Some QA team at Apple is

1) Some dev team at Apple is writing the on device scanning code. 2) Some dev team at Apple is writing a "sendMessagePolice()" API. 3) Some QA team at Apple is testing both systems. I would personally quit if asked to work on any of these teams. If you were the QA team lead for this, how would you test it? I can't imagine anyone would do day-to-day testing with actual illegal images. It would not surprise me if the dev and QA teams at Apple already have hash colliding images to run their tests? This is all awful. Steve Jobs must be rolling over in his grave.

sumedh 5y ago
> I can't imagine anyone would do day-to-day testing with actual illegal images. You select any random image as your test image, marking that image as CP in the backend and then try to upload that image from your test phone.
- testfoobar 5y ago
  Understood that that is how you test in the early stages. But at some point you have to run tests against the actual deployed perceptual hash database.
  - deleted 5y ago
    [deleted]
  - Ambroos 5y ago
    No reason why you can't just generate random images and add them to the real database, indistinguishable from actual hashes. As long as the test images aren't public it's low risk, and even if they leak they could be removed from the database and new test images could be generated.
    - testfoobar 5y ago
      That makes sense. I would still quit the team.
    - johnny53169 5y ago
      Wouldn't that mean that people created a system that can be used with normal pictures? So your code was literally to flag an innocent photo?
nebula8804 5y ago
>This is all awful. Steve Jobs must be rolling over in his grave. They didn't even last 10 years since his death. :/
unityByFreedom 5y ago
> Steve Jobs must be rolling over in his grave. June, 2010 - Jobs: "Privacy means people know what they're signing up for." https://youtu.be/39iKLwlUqBo?t=154 https://youtu.be/39iKLwlUqBo?t=154 December, 2015 - Cook: "If you put a back door in, that back door is for everybody, for good guys and bad guys." https://youtu.be/rQebmygKq7A?t=66 https://youtu.be/rQebmygKq7A?t=66 August, 2021 Interviewer: "Isn't this in a way a back door?" Federighi: "I really don't understand that characterization." https://youtu.be/OQUO1DSwYN0?t=426 https://youtu.be/OQUO1DSwYN0?t=426
- arvinsim 5y ago
  Apple really is talking down on people on that last one.
sampling 5y ago
Apple's "CSAM detection" feature is a two-part on-device/server as I understand it: 1: Run on-device code to perform perceptual hash comparison of each photo against an on-device encrypted database of known CSAM hashes. 2: On iCloud Photos servers, send out the relevant notifications when a user’s iCloud Photos account exceeds a threshold of positive matches. So as a high level testing strategy, I would want to: - Verify on-device lookup of CSAM hashes. This could be tested by provisioning a test device with an on-device database containing CSAM hashes of images that aren't illegal. As a bystander, I think I'd be fairly confident with this approach because I'm guessing the on-device database that Apple ships could conceivably be changed over time to expand the definition of the images it will flag as CSAM. - Do some exploratory testing to discover the threshold of how much image manipulation can be done on a flagged image before the perceptual hash comparison fails to return a match. - Verify that the notification system notifies the correct parties once a user account exceeds the defined threshold of positive CSAM matches. - Ensure the flagged account can still be investigated if user deletes the offending material from iCloud, or their account by the time a real person gets around to investigating. - Ensure that the logging is informative and adequate (contains device name, timestamp, etc.). - Test behaviour on same iCloud account logged in to multiple devices. - Figure out any additional business logic - are positive matches a permanent count on the account or are they reset after a certain amount of time? source: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
ByteWelder 5y ago
You wouldn't test this with child abuse images. You would likely test it with a set of legal images, for which you generated your own hashes.

sumedh 5y ago

> I can't imagine anyone would do day-to-day testing with actual illegal images. You select any random image as your test image, marking that image as CP in the backend and then try to upload that image from your test phone.

testfoobar 5y ago
Understood that that is how you test in the early stages. But at some point you have to run tests against the actual deployed perceptual hash database.
- deleted 5y ago
  [deleted]
- Ambroos 5y ago
  No reason why you can't just generate random images and add them to the real database, indistinguishable from actual hashes. As long as the test images aren't public it's low risk, and even if they leak they could be removed from the database and new test images could be generated.
  - testfoobar 5y ago
    That makes sense. I would still quit the team.
  - johnny53169 5y ago
    Wouldn't that mean that people created a system that can be used with normal pictures? So your code was literally to flag an innocent photo?

nebula8804 5y ago

>This is all awful. Steve Jobs must be rolling over in his grave. They didn't even last 10 years since his death. :/

unityByFreedom 5y ago

> Steve Jobs must be rolling over in his grave. June, 2010 - Jobs: "Privacy means people know what they're signing up for." https://youtu.be/39iKLwlUqBo?t=154 https://youtu.be/39iKLwlUqBo?t=154 December, 2015 - Cook: "If you put a back door in, that back door is for everybody, for good guys and bad guys." https://youtu.be/rQebmygKq7A?t=66 https://youtu.be/rQebmygKq7A?t=66 August, 2021 Interviewer: "Isn't this in a way a back door?" Federighi: "I really don't understand that characterization." https://youtu.be/OQUO1DSwYN0?t=426 https://youtu.be/OQUO1DSwYN0?t=426

arvinsim 5y ago
Apple really is talking down on people on that last one.

sampling 5y ago

Apple's "CSAM detection" feature is a two-part on-device/server as I understand it: 1: Run on-device code to perform perceptual hash comparison of each photo against an on-device encrypted database of known CSAM hashes. 2: On iCloud Photos servers, send out the relevant notifications when a user’s iCloud Photos account exceeds a threshold of positive matches. So as a high level testing strategy, I would want to: - Verify on-device lookup of CSAM hashes. This could be tested by provisioning a test device with an on-device database containing CSAM hashes of images that aren't illegal. As a bystander, I think I'd be fairly confident with this approach because I'm guessing the on-device database that Apple ships could conceivably be changed over time to expand the definition of the images it will flag as CSAM. - Do some exploratory testing to discover the threshold of how much image manipulation can be done on a flagged image before the perceptual hash comparison fails to return a match. - Verify that the notification system notifies the correct parties once a user account exceeds the defined threshold of positive CSAM matches. - Ensure the flagged account can still be investigated if user deletes the offending material from iCloud, or their account by the time a real person gets around to investigating. - Ensure that the logging is informative and adequate (contains device name, timestamp, etc.). - Test behaviour on same iCloud account logged in to multiple devices. - Figure out any additional business logic - are positive matches a permanent count on the account or are they reset after a certain amount of time? source: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...

ByteWelder 5y ago

You wouldn't test this with child abuse images. You would likely test it with a set of legal images, for which you generated your own hashes.