4 ms·

I'm fairly sure they do, actually. It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server

by 63 5y ago

I'm fairly sure they do, actually. It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side after it gets flagged by the client side neural hash. Then only after 30 such images from a single user are identified as CSAM by both algorithms will they be sent to a human reviewer who will confirm their contents. Then, finally, law enforcement will be alerted. There has been a lot of hyperbole going around and the original premise that this is a breach of privacy is still true, but in my opinion the actual repercussions of attacks and collisions are being grossly exaggerated. One would have to create a collision with known CSAM for both algorithms (one of which is secret) which also overlaps with a legal porn image that could be misconstrued as CSAM by a human reviewer, or at the very least create and distribute hundreds of double collisions to DOS the reviewers.

silvestrov 5y ago
> It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side But then they still need to upload the original image to the server, and what was the reason for doing the scanning client-side then when they still upload it?
- Thorrez 5y ago
  >and what was the reason for doing the scanning client-side then when they still upload it? Probably so that China cannot force Apple to hand over arbitrary images in iCloud (or all images in iCloud). With Apple's design the only images China can get from you are malicious images that people send you. If Apple scanned every image serverside without any clientside scanning, then theoretically China could get all newly-uploaded iCloud images.
blintz 5y ago
But relying on an algorithm staying secret is security-by-obscurity 101. You can rely on a cryptographic key staying secret; you can't rely on the design of an algorithm staying secret (I do agree there's a little blurring of these lines with large, trained models, but the gist remains - you can't just hope that nobody sees the structure/weights of your second hash function).
- londons_explore 5y ago
  Security by obscurity has protected a lot of things successfully...
  - mikeyla85 5y ago
    In this context, security by obscurity is literally what protects private keys.
- MrMoenty 5y ago
  Given that training the same network (with the same structure) will result in different weights and different hashes with high probability, I would argue that the weights actually have all the important properties of a secret key. You would just need to treat them as one operationally, i.e., make sure as few people as possible have access, and use truly random instead of pseudorandom numbers during training.
- user-the-name 5y ago
  Why can you rely on a cryptographic key staying secret, but not on a trained model staying secret? Both are pieces of binary data that you keep on your own machines and do not give to others. They are exactly the same case.
  - saithound 5y ago
    There's a massive difference. In fact, the data describing the trained model is not at all analogous to a cryptographic key. A cryptographic key is a piece of information which, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system. This means that your system should remain secure even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). If the key fails to have this property, your cryptosystem is broken. The trained model (or the weights of a NN) does not have this property at all. Keeping the model secret does not ensure the confidentiality or integrity of the system. E.g. just knowing some inputs and outputs of the secret model allows you to train your own classifier which behaves similarly enough to let you find perceptual hash collisions. If you treat your model as a cryptosystem, this would be a known-plaintext attack: any system vulnerable to these is considered completely and utterly broken. You'd have to keep all of the following secret: the model, all its inputs, all its outputs. If you manage to do that, this might be secure. Might. But probably not. See also Part 2 of my FAQ, which happens to cover this question. [1] [1] https://news.ycombinator.com/item?id=28232625 https://news.ycombinator.com/item?id=28232625

by 63 5y ago

silvestrov 5y ago

> It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side But then they still need to upload the original image to the server, and what was the reason for doing the scanning client-side then when they still upload it?

Thorrez 5y ago
>and what was the reason for doing the scanning client-side then when they still upload it? Probably so that China cannot force Apple to hand over arbitrary images in iCloud (or all images in iCloud). With Apple's design the only images China can get from you are malicious images that people send you. If Apple scanned every image serverside without any clientside scanning, then theoretically China could get all newly-uploaded iCloud images.

blintz 5y ago

But relying on an algorithm staying secret is security-by-obscurity 101. You can rely on a cryptographic key staying secret; you can't rely on the design of an algorithm staying secret (I do agree there's a little blurring of these lines with large, trained models, but the gist remains - you can't just hope that nobody sees the structure/weights of your second hash function).

londons_explore 5y ago
Security by obscurity has protected a lot of things successfully...
- mikeyla85 5y ago
  In this context, security by obscurity is literally what protects private keys.
MrMoenty 5y ago
Given that training the same network (with the same structure) will result in different weights and different hashes with high probability, I would argue that the weights actually have all the important properties of a secret key. You would just need to treat them as one operationally, i.e., make sure as few people as possible have access, and use truly random instead of pseudorandom numbers during training.
user-the-name 5y ago
Why can you rely on a cryptographic key staying secret, but not on a trained model staying secret? Both are pieces of binary data that you keep on your own machines and do not give to others. They are exactly the same case.
- saithound 5y ago
  There's a massive difference. In fact, the data describing the trained model is not at all analogous to a cryptographic key. A cryptographic key is a piece of information which, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system. This means that your system should remain secure even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). If the key fails to have this property, your cryptosystem is broken. The trained model (or the weights of a NN) does not have this property at all. Keeping the model secret does not ensure the confidentiality or integrity of the system. E.g. just knowing some inputs and outputs of the secret model allows you to train your own classifier which behaves similarly enough to let you find perceptual hash collisions. If you treat your model as a cryptosystem, this would be a known-plaintext attack: any system vulnerable to these is considered completely and utterly broken. You'd have to keep all of the following secret: the model, all its inputs, all its outputs. If you manage to do that, this might be secure. Might. But probably not. See also Part 2 of my FAQ, which happens to cover this question. [1] [1] https://news.ycombinator.com/item?id=28232625 https://news.ycombinator.com/item?id=28232625