3 ms·

What's novel about this? iirc, Apple withdrew its plan to hash photos client-side a couple years ago after an outcry. Dropbox has been hashing every file foreve

What's novel about this? iirc, Apple withdrew its plan to hash photos client-side a couple years ago after an outcry. Dropbox has been hashing every file forever to save storage space. Store your shit with a cloud provider, expect it to get scanned, right? Also, there are a million cute methods to make two different photos produce the same hash; that was actually what the outcry about Apple's version was about. The more the hash algorithm tried to produce the same hash for different variants of a photo, the more likely it was that someone could get their hands on a flagged hash and theoretically send you an innocuous looking photo that registered as CSAM. Pretty sure that's why Apple pulled it.

kalleboo 2y ago
> theoretically send you an innocuous looking photo that registered as CSAM. Pretty sure that's why Apple pulled it. Apple also had human reviewers in the mix. The only reason they pulled it was PR/optics.
- zaptrem 2y ago
  Wouldn't that require exfiltrating the original photo? I remember them swearing that wasn't part of the deal.
  - jchw 2y ago
    Going from memory here but IIRC the deal was that on device they'd produce a hash using a known pHash, and if that was positive, they'd send the photo to check it against a second pHash that wasn't publicly-disclosed (to try to mitigate the problem of intentional collisions) and then if both of them were positive matches, they would have human reviewers in the loop.
    - nullc 2y ago
      Not "if then send" -- if that were the case you could detect that your images were matching and then use the system as an oracle to e.g. detect that the database might contain disfavored political content in addition to whatever they claim it contains. The system was proposed as part of the upload process to your private encrypted cloud storage. So they already have the files. The system was designed so that if there were a sufficient number of hits against an encrypted database then a party possessing a particular private key would learn the private key for the encrypted files, as a passive effect of there being matches. Like if I mutter random letters of my password while entering it, you'd eventually passively learn my password just by hanging around while I logged in each morning. This way any matching is completely undetectable on the users end, at least until the private key possessing parties or the people they share the information with choose to take some action against the user.
  - JimDabell 2y ago
    It was a lot more advanced and abuse-resistant than people assumed. I really wish people had read how it worked instead of guessing it was something a lot simpler. There were two different perceptual hashes. If both matched, and the number of positive matches was high enough, a thumbnail would be able to be decrypted by Apple. Neither the device nor the server were able to independently check for a match, so the device wasn’t able to just scan all your files and flag the matches. It was tied into the iCloud upload process.
    - zaptrem 2y ago
      While this is understandable, the unfortunate issue was that Apple could be coerced into adding images certain authoritarian governments didn’t like to the list. Though imo it’s all moot if iCloud Photos aren’t end to end encrypted anyway.
      - miki123211 2y ago
        The fact that it is CSAM makes it an even harder problem to solve. With e.g. copyright infringement, you could keep some kind of records why a particular file is in the system, potentially even letting trusted and vetted organizations audit the files, but doing that for CSAM would be highly illegal and defeat the purpose of the system.
        noduerme 2y ago
        This is a great point. How do you audit whether they flagged "Tank Man" photos as criminal if you can't actually control for CSAM images? Talk about the thin end of the authoritarian wedge...
      - ogurechny 2y ago
        “Coerced”? Check some recent news to see that for Apple rainbow-washing stops at the very moment they are held responsible for providing basic censorship circumvention tools. I am amazed how people still cling to the hope that one day a corporation will do something nice for them without any hidden motive.
- jchw 2y ago
  I would assert the only reason they pursued it in the first place is PR/optics, since the "optics" of not being able to proactively police what users do using E2EE services you provide is somewhat a problem. That said, I think the concept of having your own computer covertly report you to the authorities is a level too dystopian to accept even from Apple.
- thayne 2y ago
  I agree the reason they pulled it was probably PR/optics. But given the problems with human reviews of apps on the app store, I wouldn't be confident that an underpaid employee somewhere wouldn't blindly agree with the algorithm.
- hggigg 2y ago
  Personally I think they had a lot of pressure from naive external and internal sources and had to take the PR hit on this one. If Apple were to do something like that with any level of real intent, the first thing you’d find out about it was when it went live.
- noduerme 2y ago
  Soooo... I should be OK with Apple's human reviewers visually checking any photo on my phone? Assuming it triggered some flag in their entirely opaque review process? Do you realize that the potential for a human review process of innocent people's photographs makes this 100x worse? That in itself would be a reason to avoid their platform. Thanks but no thanks.
JimDabell 2y ago
> that was actually what the outcry about Apple's version was about. The more the hash algorithm tried to produce the same hash for different variants of a photo, the more likely it was that someone could get their hands on a flagged hash and theoretically send you an innocuous looking photo that registered as CSAM. That was totally infeasible. There were two separate hashes, a public one and a private one, and there needed to be multiple false positives for the system to trigger. So not only would you need to generate collisions for two separate hashes simultaneously, including one for which there are no public details, you would need to do it for several images. People made a lot of assumptions about how it would work without actually reading the papers Apple published on how it would work. So there’s this caricature of the system in people’s minds that is a lot simpler and easier to fool than the reality. That’s what Apple was forced to react to.
- Ylpertnodi 2y ago
  >That’s what Apple was forced to react to. Perhaps apple could have made a better explanation available? Unless, there wasn't one?
- nullc 2y ago
  Plenty of commentary did address those points. The users privacy is compromised at the point that the public hash had too many hits. An attacker that can implant one hit (e.g. by giving you an altered image that matched genuine illicit material or by using stable diffusion to generate fake illicit material and altering it to match the unaltered hash of an image you possess then submitting it to NCMEC) can obviously also implant multiple. At that juncture the cryptographic keys are leaked to apple, and all further security depends on apple telling the truth about their process, not being compelled by administrative subpoena, and not ever being unwittingly compromised by hackers or intelligence operatives. The extra steps of a second perceptual hash and human review are thus not all that relevant, and nor were they clearly enough defined for any analysis of their security properties. Particularly the second perceptual hash's security is apparently at least partially dependent on its obscurity, but you have no reason to believe that it won't be obtained by hackers, rogue employees, intelligence operatives, etc. (And if its obscurity isn't relevant, then why not publish it?). Even if the hashes were flawless however, the system would be relatively straight forward to attack through less sophisticated means and would retain the overarching philosphical flaw: Your computing device is your trusted agent-- you share with it material more confidential than your doctor, your lawyer, or your priest. You paid to purchase it. You pay to power it. Increasingly you cannot communicate with family, business partners, or carry out essential and mandatory interactions with you government without using it. Your computing device mediates almost every aspect of your life. As a trusted agent it has absolutely no business scanning your files against unaccountable secret databases, encrypted against your inspection, and undetectably phoning home matches like a KGB spy that you're forced to confide in and house. To do so is a gross betrayal, one that shouldn't just be a bad idea-- it ought to be unlawful. Service providers scanning content is morally fraught itself, but in our unfortunate current legal standard you have little to no expectation of privacy in information to provide to a third party. And that against-your-own-interests scanning is done on computers owned and operated by the scanners, rather than you. And it's done using access to your information that they already have, so it's a realization of the consequences of existent poor privacy rather than an a new invasion. The transition to your own devices scanning against you is a bridge to far, no matter how much technical obfuscation is layered onto it. As someone who has developed privacy technology I found the entire presentation additionally offensive because apple misrepresented the PSI components as protecting the users privacy, when in reality the only purpose for their existence was concealing the list of hashes from the users and thus protecting it from review and criticism. It's one thing for a security scheme to provide insufficient protections, it's quite another to fraudulently present technology which is weaponized against the user as somehow being for them.
the_mitsuhiko 2y ago
> What's novel about this? Literally in the first paragraph you can see that it’s not novel. > PhotoDNA was developed by Microsoft Research and Hany Farid, professor at Dartmouth College, beginning in 2009
- formerly_proven 2y ago
  Also this isn't novel.ycombinator.com
ogurechny 2y ago
Technical details are actually irrelevant, people just like to show sophistication by discussing patterns on emperor's new clothes. It was, most likely, a second ad-hoc solution after first ad-hoc solution (most likely, having something to do with md5()) stopped working well enough. What is important in this article is what isn't written there, and has to be deduced. How exactly “terrorist content” was included. What else has been discussed behind the closed doors. Who actually decides how the thing works.

kalleboo 2y ago

> theoretically send you an innocuous looking photo that registered as CSAM. Pretty sure that's why Apple pulled it. Apple also had human reviewers in the mix. The only reason they pulled it was PR/optics.

zaptrem 2y ago
Wouldn't that require exfiltrating the original photo? I remember them swearing that wasn't part of the deal.
- jchw 2y ago
  Going from memory here but IIRC the deal was that on device they'd produce a hash using a known pHash, and if that was positive, they'd send the photo to check it against a second pHash that wasn't publicly-disclosed (to try to mitigate the problem of intentional collisions) and then if both of them were positive matches, they would have human reviewers in the loop.
  - nullc 2y ago
    Not "if then send" -- if that were the case you could detect that your images were matching and then use the system as an oracle to e.g. detect that the database might contain disfavored political content in addition to whatever they claim it contains. The system was proposed as part of the upload process to your private encrypted cloud storage. So they already have the files. The system was designed so that if there were a sufficient number of hits against an encrypted database then a party possessing a particular private key would learn the private key for the encrypted files, as a passive effect of there being matches. Like if I mutter random letters of my password while entering it, you'd eventually passively learn my password just by hanging around while I logged in each morning. This way any matching is completely undetectable on the users end, at least until the private key possessing parties or the people they share the information with choose to take some action against the user.
- JimDabell 2y ago
  It was a lot more advanced and abuse-resistant than people assumed. I really wish people had read how it worked instead of guessing it was something a lot simpler. There were two different perceptual hashes. If both matched, and the number of positive matches was high enough, a thumbnail would be able to be decrypted by Apple. Neither the device nor the server were able to independently check for a match, so the device wasn’t able to just scan all your files and flag the matches. It was tied into the iCloud upload process.
  - zaptrem 2y ago
    While this is understandable, the unfortunate issue was that Apple could be coerced into adding images certain authoritarian governments didn’t like to the list. Though imo it’s all moot if iCloud Photos aren’t end to end encrypted anyway.
    - miki123211 2y ago
      The fact that it is CSAM makes it an even harder problem to solve. With e.g. copyright infringement, you could keep some kind of records why a particular file is in the system, potentially even letting trusted and vetted organizations audit the files, but doing that for CSAM would be highly illegal and defeat the purpose of the system.
      - noduerme 2y ago
        This is a great point. How do you audit whether they flagged "Tank Man" photos as criminal if you can't actually control for CSAM images? Talk about the thin end of the authoritarian wedge...
    - ogurechny 2y ago
      “Coerced”? Check some recent news to see that for Apple rainbow-washing stops at the very moment they are held responsible for providing basic censorship circumvention tools. I am amazed how people still cling to the hope that one day a corporation will do something nice for them without any hidden motive.
jchw 2y ago
I would assert the only reason they pursued it in the first place is PR/optics, since the "optics" of not being able to proactively police what users do using E2EE services you provide is somewhat a problem. That said, I think the concept of having your own computer covertly report you to the authorities is a level too dystopian to accept even from Apple.
thayne 2y ago
I agree the reason they pulled it was probably PR/optics. But given the problems with human reviews of apps on the app store, I wouldn't be confident that an underpaid employee somewhere wouldn't blindly agree with the algorithm.
hggigg 2y ago
Personally I think they had a lot of pressure from naive external and internal sources and had to take the PR hit on this one. If Apple were to do something like that with any level of real intent, the first thing you’d find out about it was when it went live.
noduerme 2y ago
Soooo... I should be OK with Apple's human reviewers visually checking any photo on my phone? Assuming it triggered some flag in their entirely opaque review process? Do you realize that the potential for a human review process of innocent people's photographs makes this 100x worse? That in itself would be a reason to avoid their platform. Thanks but no thanks.

JimDabell 2y ago

> that was actually what the outcry about Apple's version was about. The more the hash algorithm tried to produce the same hash for different variants of a photo, the more likely it was that someone could get their hands on a flagged hash and theoretically send you an innocuous looking photo that registered as CSAM. That was totally infeasible. There were two separate hashes, a public one and a private one, and there needed to be multiple false positives for the system to trigger. So not only would you need to generate collisions for two separate hashes simultaneously, including one for which there are no public details, you would need to do it for several images. People made a lot of assumptions about how it would work without actually reading the papers Apple published on how it would work. So there’s this caricature of the system in people’s minds that is a lot simpler and easier to fool than the reality. That’s what Apple was forced to react to.

Ylpertnodi 2y ago
>That’s what Apple was forced to react to. Perhaps apple could have made a better explanation available? Unless, there wasn't one?
nullc 2y ago
Plenty of commentary did address those points. The users privacy is compromised at the point that the public hash had too many hits. An attacker that can implant one hit (e.g. by giving you an altered image that matched genuine illicit material or by using stable diffusion to generate fake illicit material and altering it to match the unaltered hash of an image you possess then submitting it to NCMEC) can obviously also implant multiple. At that juncture the cryptographic keys are leaked to apple, and all further security depends on apple telling the truth about their process, not being compelled by administrative subpoena, and not ever being unwittingly compromised by hackers or intelligence operatives. The extra steps of a second perceptual hash and human review are thus not all that relevant, and nor were they clearly enough defined for any analysis of their security properties. Particularly the second perceptual hash's security is apparently at least partially dependent on its obscurity, but you have no reason to believe that it won't be obtained by hackers, rogue employees, intelligence operatives, etc. (And if its obscurity isn't relevant, then why not publish it?). Even if the hashes were flawless however, the system would be relatively straight forward to attack through less sophisticated means and would retain the overarching philosphical flaw: Your computing device is your trusted agent-- you share with it material more confidential than your doctor, your lawyer, or your priest. You paid to purchase it. You pay to power it. Increasingly you cannot communicate with family, business partners, or carry out essential and mandatory interactions with you government without using it. Your computing device mediates almost every aspect of your life. As a trusted agent it has absolutely no business scanning your files against unaccountable secret databases, encrypted against your inspection, and undetectably phoning home matches like a KGB spy that you're forced to confide in and house. To do so is a gross betrayal, one that shouldn't just be a bad idea-- it ought to be unlawful. Service providers scanning content is morally fraught itself, but in our unfortunate current legal standard you have little to no expectation of privacy in information to provide to a third party. And that against-your-own-interests scanning is done on computers owned and operated by the scanners, rather than you. And it's done using access to your information that they already have, so it's a realization of the consequences of existent poor privacy rather than an a new invasion. The transition to your own devices scanning against you is a bridge to far, no matter how much technical obfuscation is layered onto it. As someone who has developed privacy technology I found the entire presentation additionally offensive because apple misrepresented the PSI components as protecting the users privacy, when in reality the only purpose for their existence was concealing the list of hashes from the users and thus protecting it from review and criticism. It's one thing for a security scheme to provide insufficient protections, it's quite another to fraudulently present technology which is weaponized against the user as somehow being for them.

the_mitsuhiko 2y ago

> What's novel about this? Literally in the first paragraph you can see that it’s not novel. > PhotoDNA was developed by Microsoft Research and Hany Farid, professor at Dartmouth College, beginning in 2009

formerly_proven 2y ago
Also this isn't novel.ycombinator.com

ogurechny 2y ago

Technical details are actually irrelevant, people just like to show sophistication by discussing patterns on emperor's new clothes. It was, most likely, a second ad-hoc solution after first ad-hoc solution (most likely, having something to do with md5()) stopped working well enough. What is important in this article is what isn't written there, and has to be deduced. How exactly “terrorist content” was included. What else has been discussed behind the closed doors. Who actually decides how the thing works.