10 ms·

This simple site is a far better demo and explanation of the extreme danger of Apple's proposal than any of the long articles written about it. Thank you for c

by smt88 5y ago

This simple site is a far better demo and explanation of the extreme danger of Apple's proposal than any of the long articles written about it. Thank you for caring enough to put this together and publish it.

aomobile 5y ago
Is Apple proposing anything? I think they are just going ahead with it
- nicce 5y ago
  Sigh… All these articles and demos are based on their old release, which is also emulated on onnxruntime. Who knows how they actually use it. We should cool down a little and wait for the real system, before expecting anything from them. Let’s see then how bad it is, now we are just victims of speculation.
  - echelon 5y ago
    You mean let Apple slide here?
  - JohnJamesRambo 5y ago
    You are basically saying “Let’s just wait and let them scan our phones.”
    - nicce 5y ago
      At this point, it is hard to argue anything which sides with Apple, as the most have set their minds. However, this is something which have been planned for years. Bad PR won't undo this, and stepping back is not happening overnight. There is so much misinformation in this subject, because it is really hard to see the bigger picture. We are in this situation, because of the leaked information. Was it intentional for political reasons, who knows. Apple is definitely going to announce some E2EE features very soon, such as backups in the next event. It is very hard to invent better system, than they have released now, to enable E2EE and apply CSAM scanning. When you step into the Apple system (which is quite closed, and nobody really can make extensive review for it), you give all your trust for Apple. Whether this scanning happens on your phone, or in the cloud, it does not really matter, as long as the end result is the same (same files are scanned). How about the abuse of the system? Prior on-device scanning, government could ask Apple to add feature for telling, if users have Winnie the Pooh images on their phone. (Could you spy for us?) After adding the on-device scanning, now they ask Apple to add this hash dataset to find Winnie the Pooh images. (Could you spy for us?) The question is still the same. Does some technical change affect for the managers and the chair? How about forging CSAM images to make FBI to show up in the neighbors door? Not going to happen easily. The only problem here is to brute force their human review process. In that case, it is further double checked by NCMEC. But still, at first we need some hash which exist in their database, and secondly that image should be uploaded to iCloud. And not only once, 30 times with different hash.
      - fragmede 5y ago
        > which is quite closed, and nobody really can make extensive review for it That's what Apple wants you to think, and they're very successful at marketing, but it's not really true. The jailbreak scene is not what it once was, but there are innards to Apple devices and services. Apple engineers put their pants on one leg at a time, the same as everyone else.
        nicce 5y ago
        This is true, but there has been always a great delay on reverse process. One year can mean a lot in the current world.
  - vineyardmike 5y ago
    It seems that this is based of a github project that takes the hash algo directly from binaries in MacOS - so it is what apple shipped.
    - deleted 5y ago
      [deleted]
  - smt88 5y ago
    Test in prod? Wait until one of these false positives gets to court?
  - matheusmoreira 5y ago
    They're the ones who need to cool it with the global surveillance.
- ostenning 5y ago
  Probably gaslight their users for the next 4 years and then backflip with a Key Note saying how they are revolutionising privacy by removing it
- hypothesis 5y ago
  They are going right ahead with this and there is no stopping it. And, if history is any indication, we’re going to hear more of this when needed… [1] https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings https://www.theguardian.com/technology/2019/jul/26/apple-con...
  - DavideNL 5y ago
    Not in the EU, it will be blocked by privacy regulation..
Natsu 5y ago
Does it? This just gets things flagged for another test and then manual review, it doesn't flag you as a pedophile or something. It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.
- setr 5y ago
  Any notion of privacy is eliminated, if you can reasonably expect arbitrary photos to be selected and sent to apple for manual review (where selection could make up really any % of the photos you upload, and presumably you won’t be told which ones were flagged for review). It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
  - selsta 5y ago
    This has always been the case when uploading something to the cloud unencrypted. You never had any privacy in the first place.
    - vineyardmike 5y ago
      But this isn't uploaded to the cloud. Its your device.
      - axoltl 5y ago
        It’s only done to pictures that you’re about to upload to the cloud.
        torstenvl 5y ago
        A claim that can't be independently verified by the owner of the device. Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.
zepto 5y ago
This site demonstrates no danger at all. Hash collisions like this are expected. Anyone who claims this is evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.
- firebaze 5y ago
  It's hard to reply to your post in the spirit of HN in assuming best intentions. But please suppose most of the readers here do understand how the system works, and most of them don't approve how it works.
  - wmf 5y ago
    I understand how the system works so I know that hash collisions are not dangerous because they have to be manually reviewed by multiple humans. Since you understand how the system works, please explain the danger of hash collisions instead of meta-commenting.
    - smoldesu 5y ago
      I'm not them, but they have a point. This ends up getting re-explained in every single one of these threads, so I'll keep it brief: - There are two main components here: automated flagging and supervised, human confirmation - It has been proven that the automated flagging system can be fooled - The last line of defense is now a human: they have to look at an obfuscated version of your colliding image to determine if it's objectionable The current discussion is mostly around the second portion, moreover what kind of social-engineering attacks would be most effective to overcome a relatively flimsy single-human point of failure.
      - zepto 5y ago
        > - It has been proven that the automated flagging system can be fooled This statement is absolutely false. No such proof has been presented anywhere, including in the linked article. Adversarial Neuralhash collisions are expected and the system is designed avoid false positives if they are encountered. Here is the relevant paragraph from Apple’s documentation: “as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.” https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
mfer 5y ago
To add some context... When photos are uploaded to places like Google images they are scanned for this material. Apple is currently only processing images that are being uploaded to iCloud. So, these are photos people intended to upload to them. This means it's not unwanted upload of photos to Apple servers. That is happening anyway. Note, I think scanning should happen on Apple servers and not the phones. This is an invasion onto the devices in my book. I wonder if this is a way to scale the system and reduce their servers and power use.
mono-bob 5y ago
Thank you for your kind words. This situation is something close to my heart, I think privacy should be a fundamental right for everyone, and it is essential to inform people.

by smt88 5y ago

aomobile 5y ago

Is Apple proposing anything? I think they are just going ahead with it

nicce 5y ago
Sigh… All these articles and demos are based on their old release, which is also emulated on onnxruntime. Who knows how they actually use it. We should cool down a little and wait for the real system, before expecting anything from them. Let’s see then how bad it is, now we are just victims of speculation.
- echelon 5y ago
  You mean let Apple slide here?
- JohnJamesRambo 5y ago
  You are basically saying “Let’s just wait and let them scan our phones.”
  - nicce 5y ago
    At this point, it is hard to argue anything which sides with Apple, as the most have set their minds. However, this is something which have been planned for years. Bad PR won't undo this, and stepping back is not happening overnight. There is so much misinformation in this subject, because it is really hard to see the bigger picture. We are in this situation, because of the leaked information. Was it intentional for political reasons, who knows. Apple is definitely going to announce some E2EE features very soon, such as backups in the next event. It is very hard to invent better system, than they have released now, to enable E2EE and apply CSAM scanning. When you step into the Apple system (which is quite closed, and nobody really can make extensive review for it), you give all your trust for Apple. Whether this scanning happens on your phone, or in the cloud, it does not really matter, as long as the end result is the same (same files are scanned). How about the abuse of the system? Prior on-device scanning, government could ask Apple to add feature for telling, if users have Winnie the Pooh images on their phone. (Could you spy for us?) After adding the on-device scanning, now they ask Apple to add this hash dataset to find Winnie the Pooh images. (Could you spy for us?) The question is still the same. Does some technical change affect for the managers and the chair? How about forging CSAM images to make FBI to show up in the neighbors door? Not going to happen easily. The only problem here is to brute force their human review process. In that case, it is further double checked by NCMEC. But still, at first we need some hash which exist in their database, and secondly that image should be uploaded to iCloud. And not only once, 30 times with different hash.
    - fragmede 5y ago
      > which is quite closed, and nobody really can make extensive review for it That's what Apple wants you to think, and they're very successful at marketing, but it's not really true. The jailbreak scene is not what it once was, but there are innards to Apple devices and services. Apple engineers put their pants on one leg at a time, the same as everyone else.
      - nicce 5y ago
        This is true, but there has been always a great delay on reverse process. One year can mean a lot in the current world.
- vineyardmike 5y ago
  It seems that this is based of a github project that takes the hash algo directly from binaries in MacOS - so it is what apple shipped.
  - deleted 5y ago
    [deleted]
- smt88 5y ago
  Test in prod? Wait until one of these false positives gets to court?
- matheusmoreira 5y ago
  They're the ones who need to cool it with the global surveillance.
ostenning 5y ago
Probably gaslight their users for the next 4 years and then backflip with a Key Note saying how they are revolutionising privacy by removing it
hypothesis 5y ago
They are going right ahead with this and there is no stopping it. And, if history is any indication, we’re going to hear more of this when needed… [1] https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings https://www.theguardian.com/technology/2019/jul/26/apple-con...
- DavideNL 5y ago
  Not in the EU, it will be blocked by privacy regulation..

Natsu 5y ago

Does it? This just gets things flagged for another test and then manual review, it doesn't flag you as a pedophile or something. It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.

setr 5y ago
Any notion of privacy is eliminated, if you can reasonably expect arbitrary photos to be selected and sent to apple for manual review (where selection could make up really any % of the photos you upload, and presumably you won’t be told which ones were flagged for review). It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
- selsta 5y ago
  This has always been the case when uploading something to the cloud unencrypted. You never had any privacy in the first place.
  - vineyardmike 5y ago
    But this isn't uploaded to the cloud. Its your device.
    - axoltl 5y ago
      It’s only done to pictures that you’re about to upload to the cloud.
      - torstenvl 5y ago
        A claim that can't be independently verified by the owner of the device. Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.

zepto 5y ago

This site demonstrates no danger at all. Hash collisions like this are expected. Anyone who claims this is evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.

firebaze 5y ago
It's hard to reply to your post in the spirit of HN in assuming best intentions. But please suppose most of the readers here do understand how the system works, and most of them don't approve how it works.
- wmf 5y ago
  I understand how the system works so I know that hash collisions are not dangerous because they have to be manually reviewed by multiple humans. Since you understand how the system works, please explain the danger of hash collisions instead of meta-commenting.
  - smoldesu 5y ago
    I'm not them, but they have a point. This ends up getting re-explained in every single one of these threads, so I'll keep it brief: - There are two main components here: automated flagging and supervised, human confirmation - It has been proven that the automated flagging system can be fooled - The last line of defense is now a human: they have to look at an obfuscated version of your colliding image to determine if it's objectionable The current discussion is mostly around the second portion, moreover what kind of social-engineering attacks would be most effective to overcome a relatively flimsy single-human point of failure.
    - zepto 5y ago
      > - It has been proven that the automated flagging system can be fooled This statement is absolutely false. No such proof has been presented anywhere, including in the linked article. Adversarial Neuralhash collisions are expected and the system is designed avoid false positives if they are encountered. Here is the relevant paragraph from Apple’s documentation: “as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.” https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...

mfer 5y ago

To add some context... When photos are uploaded to places like Google images they are scanned for this material. Apple is currently only processing images that are being uploaded to iCloud. So, these are photos people intended to upload to them. This means it's not unwanted upload of photos to Apple servers. That is happening anyway. Note, I think scanning should happen on Apple servers and not the phones. This is an invasion onto the devices in my book. I wonder if this is a way to scale the system and reduce their servers and power use.

mono-bob 5y ago

Thank you for your kind words. This situation is something close to my heart, I think privacy should be a fundamental right for everyone, and it is essential to inform people.