5 ms·

Sigh, for the last time, it doesn't actually matter if the NeuralHash is identical. You need multiple images matching, and then the images are compared by anoth

• collaborative 5y ago
  We still need to rely on the automated "secret backend system" that nobody supposedly knows anything about
  • toxik 5y ago
    You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.
    • Hackbraten 5y ago
      1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side. 2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.
      • idunnoman 5y ago
        "Hey Siri, accuse me of something that will ruin my life and reputation even if I'm not guilty".
      • fetzu 5y ago
        What I somehow fail to grasp in the first argument is that this whole system is designed specifically so that it runs client-side. AFAIK all the alternatives (as in « cloud photo services ») have been doing the exact same thing on the server side for decades. If you upload your photos to the cloud, a lot of service actually already have the power to destroy your life.
        • collaborative 5y ago
          This is all about Apple users waking up to the fact that they've been had
    • strangetortoise 5y ago
      afaik none of Apples other "proprietary software" is designed to pass my personal images to a human for visual inspection if it mistakenly outputs 2 high-enough numbers after a handful of convolution operations and matrix multiplications.
      • FabHK 5y ago
        They pass a "visual derivative" to "a human", but only after some matrix multiplications etc. that result in extremely low probability false positives. It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1]. [1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5-of-smartphones-lost-every-year/ https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
    • collaborative 5y ago
      Tell this to the victims of Pegasus. If anyone were able to get their hands on the "secret backend system" we wouldn't be talking about spy games, we'd be talking about people's lives being ruined
      • roody15 5y ago
        Apple still has not patched the security exploit in iMessage used by Pegasus. Apple has released two ios security updates since the Pegasus revelations but still has not patched it most widely used exploit… hmmmm. Now apple is getting a local client side scanning tool ready. Interesting timing.
        • collaborative 5y ago
          I don't think what you are saying is far-fetched
    • read_if_gay_ 5y ago
      No. I trusted Apple’s proprietary software before this, because they maintained that they care about privacy, and there have been examples of that. Now, I don’t trust them anymore.
    • simion314 5y ago
      There is a big difference, if say Apple tracking of what you run is problematic you get a bad user experience like apps starting after 1 minute of waiting, or if Apple App Store contains malware you will probably get some annoying issue while Apple will try to silently cleanup their mess BUT with this system there is a difference, this is designed not to serve you or protect you, it is designed to do some checks then if some specific rules match send some guys on you to destroy your life, today is FBI tomorrow other authoritarians. Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law. For now I see only disadvantages but please let me know of any real advantage and not speculation Disadvantages: - closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments) - bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff. - this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
• tsimionescu 5y ago
  Then let's get rid of the NeuralHash entirely, if it doesn't matter, right? If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
  • toxik 5y ago
    The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
    • Retr0id 5y ago
      It doesn't really matter whether all images are uploaded, or just 1 in x (for large value of x), due to the Panopticon effect.
      • ithkuil 5y ago
        Let's not forget what the alternative is: this is about images that are uploaded on icloud anyway. The alternative is to upload the image in clear (or with ane encryption key that apple controls), and let apple run the CSAM filter on their servers. Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about) This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
• draw_down 5y ago
  Come on.
• saithound 5y ago
  Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is. 0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever. 1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image. 2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No. First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens. Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security. Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly". 3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not. [1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-images-flagged-in-multiple-nations-568625 https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag... [2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle [3] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [4] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html https://security.googleblog.com/2017/02/announcing-first-sha... [5] https://www.zdnet.com/article/sha-1-collision-attacks-are-now-actually-practical-and-a-looming-danger/ https://www.zdnet.com/article/sha-1-collision-attacks-are-no...
• nullc 5y ago
  (A threshold of) matchings result in the private keys for the images being leaked to Apple, where they're vulnerable to: (1) Review by apple staff (2) Access and leaking by other apple staff (3) Access by hackers who have compromised their system (4) Access by parties coercing apple/staff, including via national security letters. All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place. Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).