6 ms·

For c) you still need to imagine a scenario where Apple is deliberately complicit, because of the human review step. If non-CSAM hashes are inserted into the da

by kemayo 5y ago

For c) you still need to imagine a scenario where Apple is deliberately complicit, because of the human review step. If non-CSAM hashes are inserted into the database, the human reviewers need to know what non-CSAM they're looking for -- otherwise they'll see the e.g. picture of a leaked document, observe that it's not child porn, and flag it as a false positive.

intricatedetail 5y ago
How would they know it is a false positive and not a CSAM image concealed in a document image that NeuralHash was able to "see"? It will be impossible to tell from a lowres picture with certainty.
- kemayo 5y ago
  My understanding is that NeuralHash is only supposed to be looking at visible pixels. Granted, I'm taking their word on that one, but I've not seen any evidence indicating that it does look for stenographically concealed images.
  - intricatedetail 5y ago
    Not necessarily steganography. For example if the image has drastically lowered contrast, the lowres image will look like a false positive, but it will not be.
    - simondotau 5y ago
      Okay, but now you're positing a scenario which is entirely pointless. Seriously, what would be the point? If you know you need to conceal the images to put them into your photo library, you surely know it's a stupid idea to put them there in the first place. It's all but impossible to find a supply of CSAM material without being made acutely aware of their illegal status. Here's what I don't get: who are these people importing CSAM into their camera roll? I for one have never felt the urge to import regular, legal porn into my camera roll. So why would anyone do that with stuff they know could land them in prison? Who the hell co-mingles their deepest darkest dirtiest secret amongst pictures of their family and last night’s dinner? If someone wants to conceal their CSAM library, I'm sure there's probably dozens of apps in the App Store that can store photos securely behind an additional layer of encryption.
mattnewton 5y ago
Constantly flagging my account to Apple is an attack of its own. I don't need Apple to be complicit, just make a mistake by erring on the side of caution when reporting.
- simondotau 5y ago
  I don't see how there's a "side of caution" when it comes to images classified as "A1" of prepubescent minors ("A") engaged in sex acts ("1"). What does it take to have 30 or more images in your photo library that have been all been manipulated to be a perceptual hash match to 30 or more A1-classified images, while also being able to pass human review ("erring on the side of caution") with respect to them being maybe fitting the A1 classification. There's no ambiguity in prepubescent minors engaged in sex acts.
  - meowster 5y ago
    The content reviewer isn't going to see an image in enough detail to determine that it is obviously A1 material, so yes, the content reviewer will err on the side of caution.
    - simondotau 5y ago
      You seem to be fairly certain about exactly how much detail is in the visual derivative contained within the safety token. Care to share your source?
JohnFen 5y ago
> you still need to imagine a scenario where Apple is deliberately complicit Which is very easy to imagine. All I have to do is imagine a national security letter.
- kemayo 5y ago
  Isn't that no greater risk than the status quo, though? Apple already has all these photos uploaded to their servers and knows the keys as iCloud Photos isn't end-to-end encrypted, so if a government can coerce them into cooperating with running scans like that, this new system doesn't seem to make any difference. (If the argument is that they can be coerced into changing the new system to scan non-uploaded photos and reporting them outside of the iCloud Photos upload process... that's a threat that applies just as much to every phone. It's an argument for not trusting any OS that you didn't compile from source yourself.) That is to say, it takes this away from the threat the article is presenting as "governments could sneakily use Apple's system to find dissidents despite Apple's best intentions" and into "governments could use the law to make Apple do things".
  - JohnFen 5y ago
    > Isn't that no greater risk than the status quo, though? I don't have an issue with Apple scanning the stuff they're keeping on their servers, in accordance with their ToS. Have at. The problem here is that they're doing the scanning on your device.
- simondotau 5y ago
  A national security letter can do a lot, but it cannot supersede the United States Constitution. Any attempt to co-opt the on-device search mechanism would be an unequivocal violation of the 4th Amendment of the United States Constitution—it's a search of private property being compelled by the Government. (As distinct from Apple voluntarily searching for CSAM, which will be part of the terms of service. And distinct from being compelled to search cloud servers, which is exempt from 4A under the "third party doctrine".)
  - 5y ago

by kemayo 5y ago

intricatedetail 5y ago

How would they know it is a false positive and not a CSAM image concealed in a document image that NeuralHash was able to "see"? It will be impossible to tell from a lowres picture with certainty.

kemayo 5y ago
My understanding is that NeuralHash is only supposed to be looking at visible pixels. Granted, I'm taking their word on that one, but I've not seen any evidence indicating that it does look for stenographically concealed images.
- intricatedetail 5y ago
  Not necessarily steganography. For example if the image has drastically lowered contrast, the lowres image will look like a false positive, but it will not be.
  - simondotau 5y ago
    Okay, but now you're positing a scenario which is entirely pointless. Seriously, what would be the point? If you know you need to conceal the images to put them into your photo library, you surely know it's a stupid idea to put them there in the first place. It's all but impossible to find a supply of CSAM material without being made acutely aware of their illegal status. Here's what I don't get: who are these people importing CSAM into their camera roll? I for one have never felt the urge to import regular, legal porn into my camera roll. So why would anyone do that with stuff they know could land them in prison? Who the hell co-mingles their deepest darkest dirtiest secret amongst pictures of their family and last night’s dinner? If someone wants to conceal their CSAM library, I'm sure there's probably dozens of apps in the App Store that can store photos securely behind an additional layer of encryption.

mattnewton 5y ago

Constantly flagging my account to Apple is an attack of its own. I don't need Apple to be complicit, just make a mistake by erring on the side of caution when reporting.

simondotau 5y ago
I don't see how there's a "side of caution" when it comes to images classified as "A1" of prepubescent minors ("A") engaged in sex acts ("1"). What does it take to have 30 or more images in your photo library that have been all been manipulated to be a perceptual hash match to 30 or more A1-classified images, while also being able to pass human review ("erring on the side of caution") with respect to them being maybe fitting the A1 classification. There's no ambiguity in prepubescent minors engaged in sex acts.
- meowster 5y ago
  The content reviewer isn't going to see an image in enough detail to determine that it is obviously A1 material, so yes, the content reviewer will err on the side of caution.
  - simondotau 5y ago
    You seem to be fairly certain about exactly how much detail is in the visual derivative contained within the safety token. Care to share your source?

JohnFen 5y ago

> you still need to imagine a scenario where Apple is deliberately complicit Which is very easy to imagine. All I have to do is imagine a national security letter.

kemayo 5y ago
Isn't that no greater risk than the status quo, though? Apple already has all these photos uploaded to their servers and knows the keys as iCloud Photos isn't end-to-end encrypted, so if a government can coerce them into cooperating with running scans like that, this new system doesn't seem to make any difference. (If the argument is that they can be coerced into changing the new system to scan non-uploaded photos and reporting them outside of the iCloud Photos upload process... that's a threat that applies just as much to every phone. It's an argument for not trusting any OS that you didn't compile from source yourself.) That is to say, it takes this away from the threat the article is presenting as "governments could sneakily use Apple's system to find dissidents despite Apple's best intentions" and into "governments could use the law to make Apple do things".
- JohnFen 5y ago
  > Isn't that no greater risk than the status quo, though? I don't have an issue with Apple scanning the stuff they're keeping on their servers, in accordance with their ToS. Have at. The problem here is that they're doing the scanning on your device.
simondotau 5y ago
A national security letter can do a lot, but it cannot supersede the United States Constitution. Any attempt to co-opt the on-device search mechanism would be an unequivocal violation of the 4th Amendment of the United States Constitution—it's a search of private property being compelled by the Government. (As distinct from Apple voluntarily searching for CSAM, which will be part of the terms of service. And distinct from being compelled to search cloud servers, which is exempt from 4A under the "third party doctrine".)
- 5y ago