7 ms·

This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything re

This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on uploaded user content since it is actually voluntary. Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.

luckylion 5y ago
> All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
- slumdev 5y ago
  Like swatting, but worse.
- mataug 5y ago
  A high rate of false positives will create distrust in the system, and ultimately make it less effective since no one will take it seriously.
  - dontblink 5y ago
    Could be used to target specific individuals and discredit them (e.g. critics of government).
  - luckylion 5y ago
    As in an activist might distribute them far and wide to sabotage the system, yeah. If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them". But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
- twojacobtwo 5y ago
  Pardon my ignorance on the subject, but is it feasible to create a false-positive hash match? Isn't a hash directly correlated to the file content? So if the file content is changed at all, the hash should be completely different, should it not?
  - __blockcipher__ 5y ago
    You have two primary misunderstandings AFAICT. (1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes (2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
    - maerF0x0 5y ago
      irrc a cryptosecure hash is computationally infeasible to generate a new sample that has the same hash. that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
      - __blockcipher__ 5y ago
        Agreed. From my comment: > although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
- nullifidian 5y ago
  Even better -- someone could remotely exploit their adversary with malware like Pegasus, implant actual CP on the phone, hide malware traces, and then the Apple's system would alert the authorities. Who would believe the protests of innocence when there is the evidence on the phone, and the infraction is so horrific? A Perfect crime.
shuckles 5y ago
The article states that the scanning is implemented as some kind of mutual operation between device and server when the device tries to upload photos into Apple’s cloud services. Presumably this is better than solely scanning photos on the server, because a client has even less control over what types of scanning Apple deploys there.

luckylion 5y ago

> All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless. And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.

slumdev 5y ago
Like swatting, but worse.
mataug 5y ago
A high rate of false positives will create distrust in the system, and ultimately make it less effective since no one will take it seriously.
- dontblink 5y ago
  Could be used to target specific individuals and discredit them (e.g. critics of government).
- luckylion 5y ago
  As in an activist might distribute them far and wide to sabotage the system, yeah. If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them". But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
twojacobtwo 5y ago
Pardon my ignorance on the subject, but is it feasible to create a false-positive hash match? Isn't a hash directly correlated to the file content? So if the file content is changed at all, the hash should be completely different, should it not?
- __blockcipher__ 5y ago
  You have two primary misunderstandings AFAICT. (1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes (2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
  - maerF0x0 5y ago
    irrc a cryptosecure hash is computationally infeasible to generate a new sample that has the same hash. that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
    - __blockcipher__ 5y ago
      Agreed. From my comment: > although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
nullifidian 5y ago
Even better -- someone could remotely exploit their adversary with malware like Pegasus, implant actual CP on the phone, hide malware traces, and then the Apple's system would alert the authorities. Who would believe the protests of innocence when there is the evidence on the phone, and the infraction is so horrific? A Perfect crime.

shuckles 5y ago

The article states that the scanning is implemented as some kind of mutual operation between device and server when the device tries to upload photos into Apple’s cloud services. Presumably this is better than solely scanning photos on the server, because a client has even less control over what types of scanning Apple deploys there.