6 ms·

I don't really understand the argument that was thrown out by many people: "Attackers will frame people by planting hash collisions on their phones!" So... a)

by mlac 5y ago

I don't really understand the argument that was thrown out by many people: "Attackers will frame people by planting hash collisions on their phones!" So... a) a bad actor is going to target someone, and have the resources to generate enough collisions that(b) look like CP, but aren't CP? but are close enough (c) to pass human review and cause an investigation so (d) they need the hash collisions to look like CP, but not be real CP? or ???? If a bad actor wants to frame someone, it's easy to do this today - hack their system or home network, open it to the internet, place the photos, call the FBI and report an anonymous tip, with the URL where it is hosted and open to the internet. Don't need hash collisions. Hacking someone's iPhone (How do you get the photos on there without forensic logs that they were added?) or iCloud so that you can place hash collisions that look like crap and fail to pass the review doesn't make sense and leaves too much of a trail. Oh? And someone won't notice thousands of photos just added to their phone? A bigger threat would be deepfake CP. When that becomes a reality, it will be a mess, because an attacker could theoretically generate an unlimited amount of it, and it will be extremely difficult to tell if it is authentic. Those hashes wouldn't be in the CSAM database, but if the attacker put them out on a server to be taken down, they would get added eventually and then show up in a scan elsewhere. But I'd be pretty horrified (and definitely call my attorney, Apple, and local FBI field office) if thousands of CSAM images just showed up in my iPhone photo stream. Edit: Downvotes are fine, and I completely understand other arguments against this, but this one has just not made sense to me...

gunapologist99 5y ago
This is a very simple attack to pull off; email someone with the colliding image embedded in the MIME document. Set height and width equal to zero or one pixel. Even if the user never actually sees it or marks it as spam, it's still on their device and in their email. For bonus points, spoof the "From" as from Apple. At the very least, they're getting SWATted.
- sweetheart 5y ago
  If this were a reasonable vector, it'd already be an issue, at least if you use large email clients like Gmail, which already scan for CSAM. Edit: Downvotes because?...
  - speeder 5y ago
    Because it IS already an issue. For example when CIA/NSA tools leaked, one of them had precisely this purpose.
  - cmsj 5y ago
    Downvotes because nerds love to construct a super super unlikely corner case to argue against something ;)
    - sweetheart 5y ago
      I'd call that out as totally unfair... if I haven't found myself doing that many times before hahaha.
    - CamperBob2 5y ago
      We, the nerds, only have to be wrong once in order to allow immense harm to occur. Apple voluntarily put itself in that position. Conversely, the governments of the world get to keep trying, over and over.
  - brigade 5y ago
    Those use a different perceptual hash that's intended to be kept secret. It's significantly harder to develop collisions for an algorithm you cannot inspect or obtain the output of. (well also, emailing actual CSAM is way easier and mostly just gets the sender reported)
    - foldr 5y ago
      Surely the sender is also likely to get reported if they email the image with the perceptual hash collision. Lots of companies run these scans. E.g. if you send the image via gmail it will most likely be scanned by Google.
    - cherioo 5y ago
      Apple also uses a second, different perceptual hash that's intended to be kept secret, on server side.
- misnome 5y ago
  How does this theoretical attack get them to save the attached image to iCloud?
  - skoskie 5y ago
    I think there's an unspoken assumption that photos are only the first part of this scanning exercise. IMO, Apple will likely end up scanning all content you choose to put on their servers. Tangentially, I think keeping their servers clear of illegal material is actually Apple's main motivation. This, in turn, supports claims made by nay-sayers that Apple could scan for other types of images/content in more repressive countries (but not necessarily report the people who did it). However, this assumption also contradicts arguments that Apple will start scanning for pictures of (e.g.) drugs, or weapons. Such images are not inherently illegal and therefore of no interest to Apple.
    - misnome 5y ago
      > Tangentially, I think keeping their servers clear of illegal material is actually Apple's main motivation My guess about this whole debacle is that - with pressure from the government to scan their cloud storage - that this is the alternate scenario to avoid giving up (or being forced to) the "encryption" guarantees of their cloud. I'm not sure what technical process they have in place to "only decrypt with valid law enforcement requests" or allow account rescue, but it seems likely that not just any employee can view whatever they want, before or after this system. Saying that I can maybe see a way the pressures are on this doesn't mean that I'm saying this is a good solution though. Clearly technically implementing this is opening a can of worms that can't really be closed again and makes a lot of other scenarios "closer". Also, evidently, people are a lot more comfortable with the idea of them actively scanning stuff people store in the cloud than transmitting the information in a side channel so they don't even need to handle decrypted data without a hit.
      - skoskie 5y ago
        > My guess about this whole debacle is that - with pressure from the government to scan their cloud storage - that this is the alternate scenario to avoid giving up (or being forced to) the "encryption" guarantees of their cloud. This is exactly the case. https://www.eff.org/deeplinks/2019/12/senate-judiciary-committee-wants-everyone-know-its-concerned-about-encryption https://www.eff.org/deeplinks/2019/12/senate-judiciary-commi... > I'm not sure what technical process they have in place to "only decrypt with valid law enforcement requests" or allow account rescue, but it seems likely that not just any employee can view whatever they want, before or after this system. They have master keys that can be used to decrypt almost everything you upload. They can be compelled to decrypt and turn over information on anyone. Another (unsourced) comment in this thread indicated they do so 30,000 times per year. The new encryption scheme will effectively stop this for photos, and no doubt other files in the future. Apple's side will begin using shared key encryption, which will require ALL ~31 keys to decrypt the offending images. The decryption keys are only generated on-device, and only from a hash that results from a CSAM match. The other photos, simply won't have the decryption keys generated, so they don't even exist. As an interesting side note, this means that a person who surpasses the CSAM threshold will still only reveal the images that actually match the CSAM database. Every other image, including those that have CSAM unknown to authorities remain encrypted. This is hardly a big win for the big scary government. They now have far less ability to search for evidence of any other crimes. You could upload video of your bank robbery to iCloud, and as long as your personal device remains secure, nobody will know.
- mlac 5y ago
  I would expect the human review ("Swatting") to be transparent, unless it was actually CP or looked like CP, at which point the person would be taken for a ride... But this is a real threat with actual CP now. And again, there is the whole fact that you received the email and there is a log that you received it.
  - vlovich123 5y ago
    "Human review" here means basically minimum wage person who's looking through a lot of abuse images on a daily basis & has significant fatigue. I view it more as a CYA thing than any meaningful protection mechanism. Additionally, the images sent to review are significantly downscaled versions of the original & could easily be made to be ambiguous. The most difficult challenge in this SWAT story is that Apple has a secret secondary hash that's checked on a collision. That's the part of the SWATting story that feels difficult on a technical level. However, there are also really smart people out there so it wouldn't surprise me if a successful attack strategy is developed at some point given time.
    - mlac 5y ago
      Sure. I mean, my main concern would be people planting actual illegal images. Not just colliding hashes. If anything, a handful of images that have hash collisions, once they made it to the authorities, would then be reviewed in detail and shown as a false positive, and then ignored in the future / whitelisted. Or investigated because someone is generating images to directly tamper with the program / frame people. No one is going to be prosecuted on "significantly downscaled ... ambiguous" versions of original fake images with a hash collision that flagged a review and was handed to the FBI accidentally because a "minimum wage" fatigued person passed it on. I get the counter-arguments, but the hash collision thing is just, sort of... weird? I even get the argument that an innocent hash collision may have your personal and private images reviewed by some other human - and that's weird. But I can't really see it going further (you'll be arrested and sentenced to life in prison from the HASH COLLISIONS!). It's just using technical terms to scare people who don't understand hashes and collisions and probability, and not really founded on reason.
- FabHK 5y ago
  Apple's proposed system does not scan all the pictures on the device, only images about to be uploaded to iCloud. A picture in an email would not be uploaded (whether visible or not). Even if uploaded to iCloud (such as pictures sent via WhatsApp by default), and above the threshold, they would still be scanned by a second algorithm and subject to human review. So, your "very simple" attack fails on at least three counts.
- a1369209993 5y ago
  > email someone with the colliding image embedded in the MIME document. Set height and width equal to zero or one pixel. (I think) the complaint is: how is that different from just using CSAM images, no collision required?
- chipotle_coyote 5y ago
  Assuming the system works as Apple's described it, then this attack doesn't work. The image needs to be added to the Photos library and the Photos library needs to be syncing with iCloud. Images in email aren't scanned, and an image that's zero or one pixel can't even be clicked on to be added to the library. And, to at least respond to two obvious counter-arguments I've looked into: "But it's just one line of code to change it to scan images (that aren't uploaded to iCloud) (that are anywhere on the device)!" No, it isn't; if you read the technical documentation and the more technically-oriented interviews Apple's given on this, there isn't just one hash that needs to be matches, there are two hashes, one on the device and one on the server. (I think Apple did a very poor job of communicating this to a general audience; it certainly wouldn't have alleviated all the concerns, but if it was understand as "client-server scanning" rather than "client-only scanning" it might have at least changed the tenor of the conversation.) That doesn't mean they can't do a combination client-server scan on every single image or even file on the device, but it makes it both more difficult to do and more difficult to hide. "But what if the system doesn't work as Apple's described it?" Well, if you don't trust Apple to some degree, all bets are off. They already do ML-based image analysis of all photos in your photo library regardless of iCloud status and they've literally demoed this on stage during iPhone keynotes, so if Apple was going to secretly give government access to on-device scanning, a different technology -- one that works (questionably well) on all images, not just already-learned ones -- is literally already there. The only way you "know" what Apple is doing with your data on or off device comes from a combination of what they tell you and what third-party security researchers discover.
  - jackTheMan 5y ago
    That is a huge-huge difference if they do it secretly, or if they do it announced. If they get caught doing secretly this in China that would be a big blow. But if they are doing openly and it is known that the government provides the hash, they can wash their hands.
sweetheart 5y ago
I also don't understand the vector everyone seems worried about. Also considering that perceptual hashing isn't new, as far as I'm aware, and it hasn't seemingly yet led to any sort of wave of innocent folks being screwed over sham illicit images. I think there _is_ an argument to be made about a system like this being used to track the spread of political material, and it's easy to see how such a system would be terrible for anyone trying to hide from an authoritarian power of some type, but that'd already require Apple is absolutely and completely compromised by an authoritarian regime, which isn't high on my list of concerns.
- azalemeth 5y ago
  The main issue is that the system is basically a bit of snitching spyware dolled up with cryptographic magic to make it only work with really bad stuff, honest. As recently posted on HN [1] one should be very wary of backdoors, no matter how much one believes only the right guys could ever use them. Once they're there, they're there: it's arrogant beyond believe to think that opponents of yours won't exploit them. I won't use an apple product with this feature. I've been using apple's ecosystem since about 1994. [1] https://twitter.com/matthew_d_green/status/1433470109742518273 https://twitter.com/matthew_d_green/status/14334701097425182... and https://news.ycombinator.com/item?id=28404219 https://news.ycombinator.com/item?id=28404219
  - mlac 5y ago
    I understand your points on backdoors, but I don't understand people trying to argue that the "Cryptographic Magic" is the bad part that will send people to prison due immediately upon "HASH COLLISIONS", in my view, takes away from the actual concerning points.
    - Zak 5y ago
      What most people are focused on is that this is spyware built in to the operating system. The fact that perceptual hashes have some weak points could also lead to problems, but that's not the main focus of most opposition to this system.

by mlac 5y ago

gunapologist99 5y ago

This is a very simple attack to pull off; email someone with the colliding image embedded in the MIME document. Set height and width equal to zero or one pixel. Even if the user never actually sees it or marks it as spam, it's still on their device and in their email. For bonus points, spoof the "From" as from Apple. At the very least, they're getting SWATted.

sweetheart 5y ago
If this were a reasonable vector, it'd already be an issue, at least if you use large email clients like Gmail, which already scan for CSAM. Edit: Downvotes because?...
- speeder 5y ago
  Because it IS already an issue. For example when CIA/NSA tools leaked, one of them had precisely this purpose.
- cmsj 5y ago
  Downvotes because nerds love to construct a super super unlikely corner case to argue against something ;)
  - sweetheart 5y ago
    I'd call that out as totally unfair... if I haven't found myself doing that many times before hahaha.
  - CamperBob2 5y ago
    We, the nerds, only have to be wrong once in order to allow immense harm to occur. Apple voluntarily put itself in that position. Conversely, the governments of the world get to keep trying, over and over.
- brigade 5y ago
  Those use a different perceptual hash that's intended to be kept secret. It's significantly harder to develop collisions for an algorithm you cannot inspect or obtain the output of. (well also, emailing actual CSAM is way easier and mostly just gets the sender reported)
  - foldr 5y ago
    Surely the sender is also likely to get reported if they email the image with the perceptual hash collision. Lots of companies run these scans. E.g. if you send the image via gmail it will most likely be scanned by Google.
  - cherioo 5y ago
    Apple also uses a second, different perceptual hash that's intended to be kept secret, on server side.
misnome 5y ago
How does this theoretical attack get them to save the attached image to iCloud?
- skoskie 5y ago
  I think there's an unspoken assumption that photos are only the first part of this scanning exercise. IMO, Apple will likely end up scanning all content you choose to put on their servers. Tangentially, I think keeping their servers clear of illegal material is actually Apple's main motivation. This, in turn, supports claims made by nay-sayers that Apple could scan for other types of images/content in more repressive countries (but not necessarily report the people who did it). However, this assumption also contradicts arguments that Apple will start scanning for pictures of (e.g.) drugs, or weapons. Such images are not inherently illegal and therefore of no interest to Apple.
  - misnome 5y ago
    > Tangentially, I think keeping their servers clear of illegal material is actually Apple's main motivation My guess about this whole debacle is that - with pressure from the government to scan their cloud storage - that this is the alternate scenario to avoid giving up (or being forced to) the "encryption" guarantees of their cloud. I'm not sure what technical process they have in place to "only decrypt with valid law enforcement requests" or allow account rescue, but it seems likely that not just any employee can view whatever they want, before or after this system. Saying that I can maybe see a way the pressures are on this doesn't mean that I'm saying this is a good solution though. Clearly technically implementing this is opening a can of worms that can't really be closed again and makes a lot of other scenarios "closer". Also, evidently, people are a lot more comfortable with the idea of them actively scanning stuff people store in the cloud than transmitting the information in a side channel so they don't even need to handle decrypted data without a hit.
    - skoskie 5y ago
      > My guess about this whole debacle is that - with pressure from the government to scan their cloud storage - that this is the alternate scenario to avoid giving up (or being forced to) the "encryption" guarantees of their cloud. This is exactly the case. https://www.eff.org/deeplinks/2019/12/senate-judiciary-committee-wants-everyone-know-its-concerned-about-encryption https://www.eff.org/deeplinks/2019/12/senate-judiciary-commi... > I'm not sure what technical process they have in place to "only decrypt with valid law enforcement requests" or allow account rescue, but it seems likely that not just any employee can view whatever they want, before or after this system. They have master keys that can be used to decrypt almost everything you upload. They can be compelled to decrypt and turn over information on anyone. Another (unsourced) comment in this thread indicated they do so 30,000 times per year. The new encryption scheme will effectively stop this for photos, and no doubt other files in the future. Apple's side will begin using shared key encryption, which will require ALL ~31 keys to decrypt the offending images. The decryption keys are only generated on-device, and only from a hash that results from a CSAM match. The other photos, simply won't have the decryption keys generated, so they don't even exist. As an interesting side note, this means that a person who surpasses the CSAM threshold will still only reveal the images that actually match the CSAM database. Every other image, including those that have CSAM unknown to authorities remain encrypted. This is hardly a big win for the big scary government. They now have far less ability to search for evidence of any other crimes. You could upload video of your bank robbery to iCloud, and as long as your personal device remains secure, nobody will know.
mlac 5y ago
I would expect the human review ("Swatting") to be transparent, unless it was actually CP or looked like CP, at which point the person would be taken for a ride... But this is a real threat with actual CP now. And again, there is the whole fact that you received the email and there is a log that you received it.
- vlovich123 5y ago
  "Human review" here means basically minimum wage person who's looking through a lot of abuse images on a daily basis & has significant fatigue. I view it more as a CYA thing than any meaningful protection mechanism. Additionally, the images sent to review are significantly downscaled versions of the original & could easily be made to be ambiguous. The most difficult challenge in this SWAT story is that Apple has a secret secondary hash that's checked on a collision. That's the part of the SWATting story that feels difficult on a technical level. However, there are also really smart people out there so it wouldn't surprise me if a successful attack strategy is developed at some point given time.
  - mlac 5y ago
    Sure. I mean, my main concern would be people planting actual illegal images. Not just colliding hashes. If anything, a handful of images that have hash collisions, once they made it to the authorities, would then be reviewed in detail and shown as a false positive, and then ignored in the future / whitelisted. Or investigated because someone is generating images to directly tamper with the program / frame people. No one is going to be prosecuted on "significantly downscaled ... ambiguous" versions of original fake images with a hash collision that flagged a review and was handed to the FBI accidentally because a "minimum wage" fatigued person passed it on. I get the counter-arguments, but the hash collision thing is just, sort of... weird? I even get the argument that an innocent hash collision may have your personal and private images reviewed by some other human - and that's weird. But I can't really see it going further (you'll be arrested and sentenced to life in prison from the HASH COLLISIONS!). It's just using technical terms to scare people who don't understand hashes and collisions and probability, and not really founded on reason.
FabHK 5y ago
Apple's proposed system does not scan all the pictures on the device, only images about to be uploaded to iCloud. A picture in an email would not be uploaded (whether visible or not). Even if uploaded to iCloud (such as pictures sent via WhatsApp by default), and above the threshold, they would still be scanned by a second algorithm and subject to human review. So, your "very simple" attack fails on at least three counts.
a1369209993 5y ago
> email someone with the colliding image embedded in the MIME document. Set height and width equal to zero or one pixel. (I think) the complaint is: how is that different from just using CSAM images, no collision required?
chipotle_coyote 5y ago
Assuming the system works as Apple's described it, then this attack doesn't work. The image needs to be added to the Photos library and the Photos library needs to be syncing with iCloud. Images in email aren't scanned, and an image that's zero or one pixel can't even be clicked on to be added to the library. And, to at least respond to two obvious counter-arguments I've looked into: "But it's just one line of code to change it to scan images (that aren't uploaded to iCloud) (that are anywhere on the device)!" No, it isn't; if you read the technical documentation and the more technically-oriented interviews Apple's given on this, there isn't just one hash that needs to be matches, there are two hashes, one on the device and one on the server. (I think Apple did a very poor job of communicating this to a general audience; it certainly wouldn't have alleviated all the concerns, but if it was understand as "client-server scanning" rather than "client-only scanning" it might have at least changed the tenor of the conversation.) That doesn't mean they can't do a combination client-server scan on every single image or even file on the device, but it makes it both more difficult to do and more difficult to hide. "But what if the system doesn't work as Apple's described it?" Well, if you don't trust Apple to some degree, all bets are off. They already do ML-based image analysis of all photos in your photo library regardless of iCloud status and they've literally demoed this on stage during iPhone keynotes, so if Apple was going to secretly give government access to on-device scanning, a different technology -- one that works (questionably well) on all images, not just already-learned ones -- is literally already there. The only way you "know" what Apple is doing with your data on or off device comes from a combination of what they tell you and what third-party security researchers discover.
- jackTheMan 5y ago
  That is a huge-huge difference if they do it secretly, or if they do it announced. If they get caught doing secretly this in China that would be a big blow. But if they are doing openly and it is known that the government provides the hash, they can wash their hands.

sweetheart 5y ago

I also don't understand the vector everyone seems worried about. Also considering that perceptual hashing isn't new, as far as I'm aware, and it hasn't seemingly yet led to any sort of wave of innocent folks being screwed over sham illicit images. I think there _is_ an argument to be made about a system like this being used to track the spread of political material, and it's easy to see how such a system would be terrible for anyone trying to hide from an authoritarian power of some type, but that'd already require Apple is absolutely and completely compromised by an authoritarian regime, which isn't high on my list of concerns.

azalemeth 5y ago
The main issue is that the system is basically a bit of snitching spyware dolled up with cryptographic magic to make it only work with really bad stuff, honest. As recently posted on HN [1] one should be very wary of backdoors, no matter how much one believes only the right guys could ever use them. Once they're there, they're there: it's arrogant beyond believe to think that opponents of yours won't exploit them. I won't use an apple product with this feature. I've been using apple's ecosystem since about 1994. [1] https://twitter.com/matthew_d_green/status/1433470109742518273 https://twitter.com/matthew_d_green/status/14334701097425182... and https://news.ycombinator.com/item?id=28404219 https://news.ycombinator.com/item?id=28404219
- mlac 5y ago
  I understand your points on backdoors, but I don't understand people trying to argue that the "Cryptographic Magic" is the bad part that will send people to prison due immediately upon "HASH COLLISIONS", in my view, takes away from the actual concerning points.
  - Zak 5y ago
    What most people are focused on is that this is spyware built in to the operating system. The fact that perceptual hashes have some weak points could also lead to problems, but that's not the main focus of most opposition to this system.