3 ms·

Seems like it would be pretty easy to ruin someone's life by texting them images of your cat, except they get flagged by this algorithm.

by na85 5y ago

marcellus23 5y ago
No, I think you don't know how the proposed CSAM scanning works: 1. It's only done on cloud photos. That person would have to manually save each photo to their photos library (with iCloud photos turned on) for the scanning to occur. 2. After reaching a threshold # of photos automatically flagged, a human reviewer has to confirm that the photos are illegal before any action is taken. So just texting someone even real CP won't do anything.
- na85 5y ago
  I was under the impression that iMessage does its own scanning as well. I could be wrong; I don't use an iphone.
  - simondotau 5y ago
    The proposal to perform scanning of photos in iMessage was an entirely different thing. It would look for photos that were likely to be sexually explicit and show a warning message prior to displaying them, but only where: 1. The account is for a child under the age of 13 2. The account was configured with family sharing 3. The feature was enabled by a parent Results of these scans would remain entirely local to the device and—with the consent of the child—also sent to the parent. But I think this last feature has been dropped.
    - genewitch 5y ago
      Please cite something relating to any of this; in August the uproar was that the phone would scan, too. Not just icloud.
      - marcellus23 5y ago
        Feels a bit silly to have to cite this but Apple’s original proposal is quite readable: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
      - JimDabell 5y ago
        You’re confusing two entirely different features. One feature does blind CSAM detection on iCloud photos. Any photos that are being added to iCloud in the USA have a safety voucher calculated on-device and attached to the upload. The device doesn’t know if there’s a match or not. When the uploads reach Apple’s servers, if there are enough matches, Apple has enough information to decrypt the safety vouchers. The photos are then additionally compared with a second, secret hashing algorithm. If those also match, a human reviewer checks a low-res thumbnail to confirm. If they also confirm, it gets reported. So the initial calculation of the safety voucher is done on-device, but the matching can only be completed on the server, and nothing gets reported unless multiple other checks also match. The other feature uses machine learning to determine whether or not a photo attached to a message contains nudity. If such an image is received by a child’s iPhone, it’s blurred out and they are told that if they choose to view it, their parents will be told about it. The entirety of this scanning happens on-device.
- dane-pgp 5y ago
  > a human reviewer has to confirm that the photos are illegal before any action is taken The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess. Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0]. Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data. [0] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/ https://embracethered.com/blog/posts/2020/husky-ai-image-res...
MauranKilom 5y ago
No, this is not how it works. It's plausible for state actors to accomplish this, but even then the list of requirements is very long: https://news.ycombinator.com/item?id=29183676 https://news.ycombinator.com/item?id=29183676

Seems like it would be pretty easy to ruin someone's life by texting them images of your cat, except they get flagged by this algorithm.

by na85 5y ago

Seems like it would be pretty easy to ruin someone's life by texting them images of your cat, except they get flagged by this algorithm.

marcellus23 5y ago

No, I think you don't know how the proposed CSAM scanning works: 1. It's only done on cloud photos. That person would have to manually save each photo to their photos library (with iCloud photos turned on) for the scanning to occur. 2. After reaching a threshold # of photos automatically flagged, a human reviewer has to confirm that the photos are illegal before any action is taken. So just texting someone even real CP won't do anything.

na85 5y ago
I was under the impression that iMessage does its own scanning as well. I could be wrong; I don't use an iphone.
- simondotau 5y ago
  The proposal to perform scanning of photos in iMessage was an entirely different thing. It would look for photos that were likely to be sexually explicit and show a warning message prior to displaying them, but only where: 1. The account is for a child under the age of 13 2. The account was configured with family sharing 3. The feature was enabled by a parent Results of these scans would remain entirely local to the device and—with the consent of the child—also sent to the parent. But I think this last feature has been dropped.
  - genewitch 5y ago
    Please cite something relating to any of this; in August the uproar was that the phone would scan, too. Not just icloud.
    - marcellus23 5y ago
      Feels a bit silly to have to cite this but Apple’s original proposal is quite readable: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
    - JimDabell 5y ago
      You’re confusing two entirely different features. One feature does blind CSAM detection on iCloud photos. Any photos that are being added to iCloud in the USA have a safety voucher calculated on-device and attached to the upload. The device doesn’t know if there’s a match or not. When the uploads reach Apple’s servers, if there are enough matches, Apple has enough information to decrypt the safety vouchers. The photos are then additionally compared with a second, secret hashing algorithm. If those also match, a human reviewer checks a low-res thumbnail to confirm. If they also confirm, it gets reported. So the initial calculation of the safety voucher is done on-device, but the matching can only be completed on the server, and nothing gets reported unless multiple other checks also match. The other feature uses machine learning to determine whether or not a photo attached to a message contains nudity. If such an image is received by a child’s iPhone, it’s blurred out and they are told that if they choose to view it, their parents will be told about it. The entirety of this scanning happens on-device.
dane-pgp 5y ago
> a human reviewer has to confirm that the photos are illegal before any action is taken The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess. Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0]. Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data. [0] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/ https://embracethered.com/blog/posts/2020/husky-ai-image-res...

MauranKilom 5y ago

No, this is not how it works. It's plausible for state actors to accomplish this, but even then the list of requirements is very long: https://news.ycombinator.com/item?id=29183676 https://news.ycombinator.com/item?id=29183676