9 ms·

Apple's New CSAM Protections May Make iCloud Photos Bruteforceable

233 points by NTroy 5y ago

kook_throwaway 5y ago
Barely related, but is CSAM a new acronym? I hadn't heard it until this fiasco.
- floatingatoll 5y ago
  No.
  - Teever 5y ago
    What is the difference between CP and CSAM and why is everyone suddenly using the term CSAM instead of CP?
    - function_seven 5y ago
      Good link to explain why "CSAM" is being used in lieu of "CP" https://www.adfsolutions.com/news/what-is-csam https://www.adfsolutions.com/news/what-is-csam > However, the phrase “child pornography” is almost too sterile and generic to properly exemplify the horrors of what is being created. That is why many advocates, including the National Center for Missing and Exploited Children (NCMEC), believe this phrase to be outdated. > NCMEC refers to these kinds of material as Child Sexual Abuse Material (CSAM), in order to “most accurately reflect what is depicted- the sexual abuse and exploitation of children”. > As a result, many organizations and advocates now refer to this material by the new term rather than "child pornography" because it explicitly ties the material to the source of the problem; the abuse that is being perpetuated to create it. Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop.
      - Teever 5y ago
        What terrible reasoning. 'Child pornography' is already too long to say repeatedly in a sentence so it is often shortened to "child porn." CSAM has far too many syllables so it too is shortened to two syllables in the form of the spoken acronym "CSAM" and so we're back to square one. With that said this is a fascinating display someone pressing the reverse button on the euphemism treadmill. I don't think I've ever seen that before.
        headmelted 5y ago
        The logic seems backwards here. I think saying “child porn” is clearer. That term is horrific as it is. If anyone sees that in a sentence they’ll be rightly revulsed and know what’s being talked about right away. I had to have someone explain the CSAM acronym. How many people are going to skip over that because they assume its something benign and unrelated?
        function_seven 5y ago
        I don't think the number of syllables is the important thing here. Rather that the term "porn" doesn't—by itself!—impute any moral judgement. It's just a thing. Some people think it's all bad, others think some is okay while some isn't, and some think anything with consenting performers is fair game. But children are incapable of being consenting performers. That's what separates abuse material from porn. So to make damn sure there isn't any overlap in the Venn diagram of media that depicts sexual acts, they'd rather not associate "child porn" with anything else that exists in the universe of "porn". It's a completely separate category, not some "bad" end of a spectrum. That's the reasoning I've heard before, anyway. And I agree with you on the reverse treadmill thing. It's interesting. On a related tangent: I've always hated how journalists use the term "sexual assault" to refer to a wide range of offenses, from forcible rape to a passing grope. Although those are both bad things, it's clear that one is tremendously more harmful than the other. We should use language to clarify that.
        Teever 5y ago
        The number of syllables matter if the goal is to prevent a term from feeling too sterile. If a term is too long and unwieldy then people will naturally shorten it to an acronym. And once they're using an acronym then we're back to square one because acronyms are devoid of the meaning that words have. I've wondered if there is an intent to distinguish between child pornography that teenagers are producing on their own to share with each other and the kind child pornography that pedophiles and child rapists create forcibly against the will of the children in the content. Maybe that's the actual difference between CP and CSAM. Maybe both are a subset of CP.
      - norov 5y ago
        I realized child beauty pagents and other sexual exploitation of children would be considered Child Sexual Abuse Material in some cases. People, even those with good intentions, are not going to like where this is going.
      - torified 5y ago
        >Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop. I've seen this argument many times, and I agree that initial act is horrendous of course, but I believe this is overstating the ongoing damage.
    - shakna 5y ago
      CSAM is not necessarily pornographic material, but merely material that becomes objectionable within context. For example, parents' photos of kids in the bath isn't CP. However _someone else_ having a _quantity_ of bath photos is CSAM, if they have no reasonable reason for possessing them.
      - deleted 5y ago
        [deleted]
      - jobigoud 5y ago
        In this case the change of acronym to stress the abuse vs pornography part is completely backward.
      - deleted 5y ago
        [deleted]
      - kergonath 5y ago
        It’s very muddy, though. The number of pictures on one’s hard drive is irrelevant to the fact that a child has been abused or not. In your example, none of the children would have been abused. Also, who gets to define how much “a lot” is? We can’t say that it’s ok if it’s your children (or grand-children’s, or nephews, etc), because most of child abuse cases involve close family members or close friends. We definitely should punish exploitation of children, including sexual, and we definitely should punish distributing images of this. But conflating exploiting, distributing, and viewing, and then putting a big taboo on this, is really not ideal. These things are different. Otherwise what we end up with is righteous frenzy when someone gets punished for sexting.
        shakna 5y ago
        That is correct. The children within the images that are classified as CSAM don't necessarily have (though frequently are) to be abused. For example, the NCMEC database contains hashes for Nirvana's Nevermind cover. Completely innocent to possess within it's original and intended context. I have not said whether I agree with this, because I do see problems when automating the process. However, the precedent for it being used as a flag for law enforcement has already happened. Having a collection of similar imagery is considered CSAM - and that is likely correct. A collection is probably not innocent. But having one or two images may happen incidentally without your awareness of it. As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
    - starwatch 5y ago
      Last year Sam Harris interviewed Gabriel J.X. Dance, the deputy investigations editor at The New York Times [0]. They speak to this. It's a tough episode to listen to - they cover many uncomfortable topics along this theme. From what I recall of the episode, porn (rightly or wrongly) is seen as opt-in for the participants. But if children are involved it cannot be opt-in and is more appropriately described as rape or sexual abuse. Thus CSAM is the preferred term. [0]: https://samharris.org/podcasts/213-worst-epidemic/ https://samharris.org/podcasts/213-worst-epidemic/
      - dannyw 5y ago
        Depends on age and context doesn't it? Is a 17 year old snapchatting their private parts to her 17 year old boyfriend CSAM? Should both parties be imprisoned and go on sex offenders registries for life?
        starwatch 5y ago
        I'm not well versed enough in the topic (or the law) to have answers. I do think the podcast episode I pointed to does give a lot of food for thought. For me it was an eye opener, as a software engineer trying to build a beautiful future with strong beliefs around e2e encryption, etc.
        withinboredom 5y ago
        Depends on the age of consent in both person’s jurisdiction and where they go with those pictures. My 16 year-old self got quite a lecture when my girlfriend and I took some pics of each other and my parents found them. (Age of consent was 13 where we lived, but it was 18 just up the road in another state).
        myspy 5y ago
        According to Grubers analysis it compares the csam pictures against the library. Your 17 year old is not part of that database. Hence no hash matches. CSAM provides a library with hashes that are compared against the photos on the phone. The real problem comes when governments make laws that open the mechanism to different libraries of photos.
    - bitwize 5y ago
      CSAM indicates you think the material, and the practice of making it, is abhorrent. "Child porn" is something some (severely deranged) people might actually want to see. The upshot of this is, if you put "ios child porn detection features" into Google, Google will see "child porn" and may think that's what you're searching for, put up warning banners reminding you that child porn is super illegal and that you should not be searching for it, and probably notify the FBI who will add you to a watchlist. If you put "ios csam detection feature" into Google, it helps Google know that you are not searching for child porn itself.
      - colejohnson66 5y ago
        What stops google from monitoring for “CSAM” as well? Security through obscurity (“if Google doesn’t know about the term, they won’t report us!”)
        bitwize 5y ago
        Social dynamics. The kind of person who calls CSAM CSAM is unlikely to favor it, let alone distribute it, so people interested in it are unlikely to search for it by that name.
shuckles 5y ago
The question presumes the database leak also comes with the server side secret for blinding the CSAM database, which is unlikely (that’s not how HSMs work) and would be a general catastrophe (it would leak the Neural Hashes of photos in the NCMEC database, which are supposed to remain secret).
- NTroy 5y ago
  The question doesn't presume that, as the the secret for blinding the CSAM database would only be helpful if a third party were also looking to see which accounts contained CSAM. In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
  - shuckles 5y ago
    > What's to stop an attacker from generating a NeuralHash of popular memes, deriving a key, then bruteforcing the leaked data until it successfully decrypts an entry, thus verifying the contents within a specific user's cloud photo library, and degrading their level of privacy? Decrypting vouchers requires the server blinding key and the NeuralHash derived metadata of the input image (technical summary page 10, Bellare Fig. 1 line 18). This attacker only has the latter.
    - NTroy 5y ago
      > For CSAM matches, the cryptographic header in the voucher combines with the server-side blinding secret (that was used to blind the known CSAM database at setup time) to successfully decrypt the outer layer of encryption. In the text you referenced, it specifically says that the blinding key would be needed to decrypt vouchers which are CSAM matches. This is because Apple set up their CSAM database in a blinded manner. Therefore to access a hash from the database from which to derive a decryption key, Apple would need the blinding key to first decrypt that hash value. However, and attacker would be generating their own (presumably unblinded) database, and therefore wouldn't need to access Apple's blinding key.
jonathanmayer 5y ago
(Context: I teach computer security at Princeton and have a paper at this week's Usenix Security Symposium describing and analyzing a protocol that is similar to Apple's: https://www.usenix.org/conference/usenixsecurity21/presentation/kulshrestha https://www.usenix.org/conference/usenixsecurity21/presentat....) The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.
- jobigoud 5y ago
  > only able to decrypt if there's a hash match This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
  - NTroy 5y ago
    Yes, this ^^^^^^ > The proposed attack on Apple's protocol doesn't work. With all due respect, I think you may have misunderstood the proposed attack @jonathanmayer, as what @jobigoud said is correct.
- amelius 5y ago
  There may be another attack. Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
  - cube00 5y ago
    It'd be interesting to see how the way common images are reused (for example in memes by only adding text) would be enough to change that hash. If it wasn't enough it could spread very quickly. Of course I'd dare not research or tinker with it lest I'll be added to a list somewhere such is the chilling effect. I guess in that case they'd delete that single hash from the database because they'd still have an endless (sadly) supply of other bad image hashes to use instead.
whatever1 5y ago
Why does Apple even bother with encryption? They should just skip all of the warrant requirements etc and use their iCloud keys to unlock our content and store it unencrypted at rest. Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.
- gorgonzolachz 5y ago
  Facetious as this is, I can't imagine this is anything other than Apple's endgame here. The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.
- laurent92 5y ago
  Funny. The way I imagine NSA’s and FBI’s secret cooperation with Google is exactly this: Provide a search API that gives access to anything.
- x2r 5y ago
  They already have that. https://en.wikipedia.org/wiki/PRISM_(surveillance_program) https://en.wikipedia.org/wiki/PRISM_(surveillance_program) But the 'security' services want access to what's on people's phones too.
joe_the_user 5y ago
Regardless of whether this attack works or not, you'd assume this scheme produces a wider attack surface against pictures in iCloud and against iCloud users. One attack I could imagine is a hacker uploading child porn to a hacked device to trigger immediate enforcement against a user (and sure, maybe there are more controls involved but would you carry around a very well-protected, well-designed hand grenade in your wallet just so you're bad, it'll explode).
- mnd999 5y ago
  Or even a hash collision with a banned image. Actually, if that could be generated this thing could fall apart pretty quickly if such collisions could be widely distributed.
- selsta 5y ago
  How is this iCloud specific? You could do the same with Google Photos or OneDrive.
  - nicce 5y ago
    Literally for almost every other big cloud provider. (Facebook, Instagram, Discord, Reddit, Twitter and so on.) Granting that you have access by phone.
  - joe_the_user 5y ago
    "How is this iCloud specific?" In case you didn't the topic, what is specific (for now, for now...)to iCloud/apple is the "we're scanning your photos on your device and maybe reporting them if they're bad" approach. So you get the local hashes on the supposedly encrypted files and you get the situation of local files trigger global effects like the police swooping down and arresting you. So that's why despicable and hair-brained scheme in specific produces a greater "attack surface" in multiple ways. And again, sure, Apple doing this quite possibly will set a precedent for Google et al to answer the other ambiguous meanings your ambiguous comment has.
jl6 5y ago
For some reason, after reading the initial reporting on this system, I thought it was running against any photos on your iPhone, but now I read the actual paper, it seems like it only applies to photos destined to be uploaded to iCloud? So users can opt out by not using iCloud?
- RegnisGnaw 5y ago
  Also don’t upload to MS, Google, Dropbox as they also scan for CSAM.
- avianlyric 5y ago
  Yeah pretty much. Another way of thinking about it, is that to upload an image to iCloud, your phone must provide a cryptographic safety voucher to prove the image isn’t CSAM.
- foerbert 5y ago
  Much of the discussion is about how trivial it would be for Apple to start scanning any photos on the phone at a later date. Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
  - nicce 5y ago
    They can do it already. System is full black box, and all we have is their word. So, saying that adding something might enable something else, is not strong argument.
    - foerbert 5y ago
      By that logic we may as well never question anything any of these companies - or even governments really - do because they might just find a way to do it secretly and maybe nobody would ever figure it out.
- tandav 5y ago
kfprt 5y ago
It won't be long until these type of systems are mandated. Combined with a hardware root of trust it's not inconceivable that modifying your hardware not to report home will also be made a crime. It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use. The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.
- stjohnswarts 5y ago
  Total surveillance is definitely the end goal of policing forces. It's in their very nature of getting their job done (what better way to catch criminals than a computer constantly scanning every move of everyone) and why people need to always push back against these "think of the children" scapegoats they use to get their foot in the door and get more control.
- zimpenfish 5y ago
  > It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use. But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it? If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
  - randomhodler84 5y ago
    Don’t be so naive. It took less than 3 years for dns blocking to go from csam to copyright infringement. It always was about building censorship infra. I’ve been fighting internet censorship for over a decade and it only gets worse and worse every generation of technology. I want to throw all this government spyware away. Dystopia landed a long time ago.
ashneo76 5y ago
Pretty soon housing your own infra and not using the mandated govt phone could be made a crime. But think of the children and security of the society. Couple that with constant monitoring of your car and you can be monitored anywhere
- kaba0 5y ago
  It already is. You are only allowed to use specific wavelengths, and basically every modem is proprietary.
roody15 5y ago
thanks e

233 points by NTroy 5y ago

kook_throwaway 5y ago

Barely related, but is CSAM a new acronym? I hadn't heard it until this fiasco.

floatingatoll 5y ago
No.
- Teever 5y ago
  What is the difference between CP and CSAM and why is everyone suddenly using the term CSAM instead of CP?
  - function_seven 5y ago
    Good link to explain why "CSAM" is being used in lieu of "CP" https://www.adfsolutions.com/news/what-is-csam https://www.adfsolutions.com/news/what-is-csam > However, the phrase “child pornography” is almost too sterile and generic to properly exemplify the horrors of what is being created. That is why many advocates, including the National Center for Missing and Exploited Children (NCMEC), believe this phrase to be outdated. > NCMEC refers to these kinds of material as Child Sexual Abuse Material (CSAM), in order to “most accurately reflect what is depicted- the sexual abuse and exploitation of children”. > As a result, many organizations and advocates now refer to this material by the new term rather than "child pornography" because it explicitly ties the material to the source of the problem; the abuse that is being perpetuated to create it. Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop.
    - Teever 5y ago
      What terrible reasoning. 'Child pornography' is already too long to say repeatedly in a sentence so it is often shortened to "child porn." CSAM has far too many syllables so it too is shortened to two syllables in the form of the spoken acronym "CSAM" and so we're back to square one. With that said this is a fascinating display someone pressing the reverse button on the euphemism treadmill. I don't think I've ever seen that before.
      - headmelted 5y ago
        The logic seems backwards here. I think saying “child porn” is clearer. That term is horrific as it is. If anyone sees that in a sentence they’ll be rightly revulsed and know what’s being talked about right away. I had to have someone explain the CSAM acronym. How many people are going to skip over that because they assume its something benign and unrelated?
      - function_seven 5y ago
        I don't think the number of syllables is the important thing here. Rather that the term "porn" doesn't—by itself!—impute any moral judgement. It's just a thing. Some people think it's all bad, others think some is okay while some isn't, and some think anything with consenting performers is fair game. But children are incapable of being consenting performers. That's what separates abuse material from porn. So to make damn sure there isn't any overlap in the Venn diagram of media that depicts sexual acts, they'd rather not associate "child porn" with anything else that exists in the universe of "porn". It's a completely separate category, not some "bad" end of a spectrum. That's the reasoning I've heard before, anyway. And I agree with you on the reverse treadmill thing. It's interesting. On a related tangent: I've always hated how journalists use the term "sexual assault" to refer to a wide range of offenses, from forcible rape to a passing grope. Although those are both bad things, it's clear that one is tremendously more harmful than the other. We should use language to clarify that.
        Teever 5y ago
        The number of syllables matter if the goal is to prevent a term from feeling too sterile. If a term is too long and unwieldy then people will naturally shorten it to an acronym. And once they're using an acronym then we're back to square one because acronyms are devoid of the meaning that words have. I've wondered if there is an intent to distinguish between child pornography that teenagers are producing on their own to share with each other and the kind child pornography that pedophiles and child rapists create forcibly against the will of the children in the content. Maybe that's the actual difference between CP and CSAM. Maybe both are a subset of CP.
    - norov 5y ago
      I realized child beauty pagents and other sexual exploitation of children would be considered Child Sexual Abuse Material in some cases. People, even those with good intentions, are not going to like where this is going.
    - torified 5y ago
      >Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop. I've seen this argument many times, and I agree that initial act is horrendous of course, but I believe this is overstating the ongoing damage.
  - shakna 5y ago
    CSAM is not necessarily pornographic material, but merely material that becomes objectionable within context. For example, parents' photos of kids in the bath isn't CP. However _someone else_ having a _quantity_ of bath photos is CSAM, if they have no reasonable reason for possessing them.
    - deleted 5y ago
      [deleted]
    - jobigoud 5y ago
      In this case the change of acronym to stress the abuse vs pornography part is completely backward.
    - deleted 5y ago
      [deleted]
    - kergonath 5y ago
      It’s very muddy, though. The number of pictures on one’s hard drive is irrelevant to the fact that a child has been abused or not. In your example, none of the children would have been abused. Also, who gets to define how much “a lot” is? We can’t say that it’s ok if it’s your children (or grand-children’s, or nephews, etc), because most of child abuse cases involve close family members or close friends. We definitely should punish exploitation of children, including sexual, and we definitely should punish distributing images of this. But conflating exploiting, distributing, and viewing, and then putting a big taboo on this, is really not ideal. These things are different. Otherwise what we end up with is righteous frenzy when someone gets punished for sexting.
      - shakna 5y ago
        That is correct. The children within the images that are classified as CSAM don't necessarily have (though frequently are) to be abused. For example, the NCMEC database contains hashes for Nirvana's Nevermind cover. Completely innocent to possess within it's original and intended context. I have not said whether I agree with this, because I do see problems when automating the process. However, the precedent for it being used as a flag for law enforcement has already happened. Having a collection of similar imagery is considered CSAM - and that is likely correct. A collection is probably not innocent. But having one or two images may happen incidentally without your awareness of it. As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
  - starwatch 5y ago
    Last year Sam Harris interviewed Gabriel J.X. Dance, the deputy investigations editor at The New York Times [0]. They speak to this. It's a tough episode to listen to - they cover many uncomfortable topics along this theme. From what I recall of the episode, porn (rightly or wrongly) is seen as opt-in for the participants. But if children are involved it cannot be opt-in and is more appropriately described as rape or sexual abuse. Thus CSAM is the preferred term. [0]: https://samharris.org/podcasts/213-worst-epidemic/ https://samharris.org/podcasts/213-worst-epidemic/
    - dannyw 5y ago
      Depends on age and context doesn't it? Is a 17 year old snapchatting their private parts to her 17 year old boyfriend CSAM? Should both parties be imprisoned and go on sex offenders registries for life?
      - starwatch 5y ago
        I'm not well versed enough in the topic (or the law) to have answers. I do think the podcast episode I pointed to does give a lot of food for thought. For me it was an eye opener, as a software engineer trying to build a beautiful future with strong beliefs around e2e encryption, etc.
      - withinboredom 5y ago
        Depends on the age of consent in both person’s jurisdiction and where they go with those pictures. My 16 year-old self got quite a lecture when my girlfriend and I took some pics of each other and my parents found them. (Age of consent was 13 where we lived, but it was 18 just up the road in another state).
      - myspy 5y ago
        According to Grubers analysis it compares the csam pictures against the library. Your 17 year old is not part of that database. Hence no hash matches. CSAM provides a library with hashes that are compared against the photos on the phone. The real problem comes when governments make laws that open the mechanism to different libraries of photos.
  - bitwize 5y ago
    CSAM indicates you think the material, and the practice of making it, is abhorrent. "Child porn" is something some (severely deranged) people might actually want to see. The upshot of this is, if you put "ios child porn detection features" into Google, Google will see "child porn" and may think that's what you're searching for, put up warning banners reminding you that child porn is super illegal and that you should not be searching for it, and probably notify the FBI who will add you to a watchlist. If you put "ios csam detection feature" into Google, it helps Google know that you are not searching for child porn itself.
    - colejohnson66 5y ago
      What stops google from monitoring for “CSAM” as well? Security through obscurity (“if Google doesn’t know about the term, they won’t report us!”)
      - bitwize 5y ago
        Social dynamics. The kind of person who calls CSAM CSAM is unlikely to favor it, let alone distribute it, so people interested in it are unlikely to search for it by that name.

shuckles 5y ago

The question presumes the database leak also comes with the server side secret for blinding the CSAM database, which is unlikely (that’s not how HSMs work) and would be a general catastrophe (it would leak the Neural Hashes of photos in the NCMEC database, which are supposed to remain secret).

NTroy 5y ago
The question doesn't presume that, as the the secret for blinding the CSAM database would only be helpful if a third party were also looking to see which accounts contained CSAM. In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
- shuckles 5y ago
  > What's to stop an attacker from generating a NeuralHash of popular memes, deriving a key, then bruteforcing the leaked data until it successfully decrypts an entry, thus verifying the contents within a specific user's cloud photo library, and degrading their level of privacy? Decrypting vouchers requires the server blinding key and the NeuralHash derived metadata of the input image (technical summary page 10, Bellare Fig. 1 line 18). This attacker only has the latter.
  - NTroy 5y ago
    > For CSAM matches, the cryptographic header in the voucher combines with the server-side blinding secret (that was used to blind the known CSAM database at setup time) to successfully decrypt the outer layer of encryption. In the text you referenced, it specifically says that the blinding key would be needed to decrypt vouchers which are CSAM matches. This is because Apple set up their CSAM database in a blinded manner. Therefore to access a hash from the database from which to derive a decryption key, Apple would need the blinding key to first decrypt that hash value. However, and attacker would be generating their own (presumably unblinded) database, and therefore wouldn't need to access Apple's blinding key.

jonathanmayer 5y ago

(Context: I teach computer security at Princeton and have a paper at this week's Usenix Security Symposium describing and analyzing a protocol that is similar to Apple's: https://www.usenix.org/conference/usenixsecurity21/presentation/kulshrestha https://www.usenix.org/conference/usenixsecurity21/presentat....) The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.

jobigoud 5y ago
> only able to decrypt if there's a hash match This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
- NTroy 5y ago
  Yes, this ^^^^^^ > The proposed attack on Apple's protocol doesn't work. With all due respect, I think you may have misunderstood the proposed attack @jonathanmayer, as what @jobigoud said is correct.
amelius 5y ago
There may be another attack. Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
- cube00 5y ago
  It'd be interesting to see how the way common images are reused (for example in memes by only adding text) would be enough to change that hash. If it wasn't enough it could spread very quickly. Of course I'd dare not research or tinker with it lest I'll be added to a list somewhere such is the chilling effect. I guess in that case they'd delete that single hash from the database because they'd still have an endless (sadly) supply of other bad image hashes to use instead.

whatever1 5y ago

Why does Apple even bother with encryption? They should just skip all of the warrant requirements etc and use their iCloud keys to unlock our content and store it unencrypted at rest. Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.

gorgonzolachz 5y ago
Facetious as this is, I can't imagine this is anything other than Apple's endgame here. The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.
laurent92 5y ago
Funny. The way I imagine NSA’s and FBI’s secret cooperation with Google is exactly this: Provide a search API that gives access to anything.
x2r 5y ago
They already have that. https://en.wikipedia.org/wiki/PRISM_(surveillance_program) https://en.wikipedia.org/wiki/PRISM_(surveillance_program) But the 'security' services want access to what's on people's phones too.

joe_the_user 5y ago

Regardless of whether this attack works or not, you'd assume this scheme produces a wider attack surface against pictures in iCloud and against iCloud users. One attack I could imagine is a hacker uploading child porn to a hacked device to trigger immediate enforcement against a user (and sure, maybe there are more controls involved but would you carry around a very well-protected, well-designed hand grenade in your wallet just so you're bad, it'll explode).

mnd999 5y ago
Or even a hash collision with a banned image. Actually, if that could be generated this thing could fall apart pretty quickly if such collisions could be widely distributed.
selsta 5y ago
How is this iCloud specific? You could do the same with Google Photos or OneDrive.
- nicce 5y ago
  Literally for almost every other big cloud provider. (Facebook, Instagram, Discord, Reddit, Twitter and so on.) Granting that you have access by phone.
- joe_the_user 5y ago
  "How is this iCloud specific?" In case you didn't the topic, what is specific (for now, for now...)to iCloud/apple is the "we're scanning your photos on your device and maybe reporting them if they're bad" approach. So you get the local hashes on the supposedly encrypted files and you get the situation of local files trigger global effects like the police swooping down and arresting you. So that's why despicable and hair-brained scheme in specific produces a greater "attack surface" in multiple ways. And again, sure, Apple doing this quite possibly will set a precedent for Google et al to answer the other ambiguous meanings your ambiguous comment has.

jl6 5y ago

For some reason, after reading the initial reporting on this system, I thought it was running against any photos on your iPhone, but now I read the actual paper, it seems like it only applies to photos destined to be uploaded to iCloud? So users can opt out by not using iCloud?

RegnisGnaw 5y ago
Also don’t upload to MS, Google, Dropbox as they also scan for CSAM.
avianlyric 5y ago
Yeah pretty much. Another way of thinking about it, is that to upload an image to iCloud, your phone must provide a cryptographic safety voucher to prove the image isn’t CSAM.
foerbert 5y ago
Much of the discussion is about how trivial it would be for Apple to start scanning any photos on the phone at a later date. Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
- nicce 5y ago
  They can do it already. System is full black box, and all we have is their word. So, saying that adding something might enable something else, is not strong argument.
  - foerbert 5y ago
    By that logic we may as well never question anything any of these companies - or even governments really - do because they might just find a way to do it secretly and maybe nobody would ever figure it out.
tandav 5y ago

kfprt 5y ago

It won't be long until these type of systems are mandated. Combined with a hardware root of trust it's not inconceivable that modifying your hardware not to report home will also be made a crime. It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use. The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.

stjohnswarts 5y ago
Total surveillance is definitely the end goal of policing forces. It's in their very nature of getting their job done (what better way to catch criminals than a computer constantly scanning every move of everyone) and why people need to always push back against these "think of the children" scapegoats they use to get their foot in the door and get more control.
zimpenfish 5y ago
> It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use. But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it? If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
- randomhodler84 5y ago
  Don’t be so naive. It took less than 3 years for dns blocking to go from csam to copyright infringement. It always was about building censorship infra. I’ve been fighting internet censorship for over a decade and it only gets worse and worse every generation of technology. I want to throw all this government spyware away. Dystopia landed a long time ago.

ashneo76 5y ago

Pretty soon housing your own infra and not using the mandated govt phone could be made a crime. But think of the children and security of the society. Couple that with constant monitoring of your car and you can be monitored anywhere

kaba0 5y ago
It already is. You are only allowed to use specific wavelengths, and basically every modem is proprietary.

roody15 5y ago

thanks e