6 ms·

Second preimage attacks are trivial because of how the algorithm works. The image goes through a neural network (one to which everyone has access), the output v

Second preimage attacks are trivial because of how the algorithm works. The image goes through a neural network (one to which everyone has access), the output vector is put through a linear transformation, and that vector is binarized, then cryptographically hashed. It's trivial to perturb any image you might wish so as to be close to the original output vector. This will result in it having the same binarization, hence the same hash. I believe the neural network is a pretty conventional convolutional one, so adversarial perturbations will exist that are invisible to the naked eye. This is useful for two purposes I can think of. One, you can randomize all the vectors on all of your images. Two, you can make problems for others by giving them harmless-looking images that have been cooked to give particular hashes. I'm not sure how bad those problems would be – at some point a police officer does have to look at the image in order to get probable cause. Perhaps it could lead to your Apple account being suspended, however.

Kalium 5y ago
A police raid on a person's home, or even a gentler thorough search, can be enough to quite seriously disrupt a person's life. Certainly having the police walk away with all your electronics in evidence bags will complicate trying to work remotely. Of course, this is assuming everything works as intended and they don't find anything else they can use to charge you with something as they search your home. If you smoke cannabis while being in the wrong state, you're now in several more kinds of trouble.
- topynate 5y ago
  This could happen with a perturbed image, but I doubt it. Apple will send the suspicious images to the relevant authorities. Those authorities will then look at the images. The chances are low that they will then seek a search, even though the images are innocent upon visual inspection. But maybe in some places a ping from Apple is good enough for a search and seizure.
  - Kalium 5y ago
    In broad strokes, I agree with you. I think you're absolutely correct that most trained, educated, technologically sophisticated law enforcement bodies will do exactly as you suggest and decide that there's not enough to investigate. That said, I'm not willing to say it won't happen. There are too many law enforcement entities of wildly varying levels of professionalism, staffing, and technical sophistication. Someone innocent, somewhere, is likely to have a multi-year legal drama because their local PD got an email from Apple. And we haven't even gotten to subjects like how some LEOs will happily plant evidence once they decide you're guilty.
  - falcolas 5y ago
    FWIW, they won't send the images. Even in the pursuit of knocking back CSAM, there are strict restrictions on the transmission and viewing of CSAM - in some cases even the defendant's lawyers don't usually see the images themselves in preparation for a trial, just a description of the contents. Apple employees or contractors will likely not look at the images themselves, only visual hashes. They will instead contact the police and say "Person X has Y images that are on list Z," and let the police get a warrant based off that information and execute it to check for actual CSAM.
    - topynate 5y ago
      On reflection, yes, there must be warrants involved. I'm raising my estimate of how likely it is that innocent people get raided due to this. The warrant would properly only be to search iCloud, not some guy's house, but I can easily see overly-broad warrants being issued.
      - gowld 5y ago
        > The warrant would properly only be to search iCloud, iCloud is encrypted, so that warrant is useless. They need to unlock and search the device.
        topynate 5y ago
- cirrus3 5y ago
  The police do not show up until a human has compared the matched image to the actual. Just stop.
  - teachrdan 5y ago
    The more collisions, the more chances of a false positive by a (tired, underpaid) human. I don't envy the innocent person whose home gets raided by a SWAT team convinced they're busting a child sex trafficker.
    - hda2 5y ago
      The database is known to contain non-csam images like porn. I doubt the reviewer will be qualified to discern which is which.
      - Operyl 5y ago
        Not that I doubt your motives, but can I get your source on this? It seems like a huge blunder if so.
      - jgreen401 5y ago
        Known by whom to contain legal images? The "reviewer" might well be a retired Pediatrician who can identify the age of the subject by looking at the photographs and will be extremely accurate.
e_proxus 5y ago
Wouldn’t it also just be possible to turn a jailbroken iDevice into a CSAM cleaner/hider? You could take actual CSAM, check if it matches the hashes and keep modifying the material until it doesn’t (adding borders, watermarking, changing dimensions etc.). Then just save it as usual without any risk.
- cyanite 5y ago
  No it’s not. The client hashes against a blinded set, and doesn’t know whether or not the hash is a hit or miss.
- skygazer 5y ago
  I can’t tell what would prevent a jailbroken device from pairing an illegal CSAM image with the safety voucher from a known-innocuous image, since the whole system depends on the trustworthiness of the safety voucher and that it truly represents the image in question. If the device is compromised I would think all bets would be off. To me, one potential flaw of this system may be that Apple inherently trusts the device. The existence of a jailbreak seems like a massive risk to the system. In fact, Apple themselves generate fake/meaningless safety vouchers a certain percentage of the time (see synthetic safety vouchers.) If a jailbroken phone could trigger that code path for all images in the pipeline, apple’s system would be completely broken. On the other hand, this may be just the excuse apple needs to lock down the phone further, to “protect the integrity of the CSAM detection system.” Perhaps they could persuade congress to make jailbreaking a federal crime. Perhaps they’re more clever than I ever imagined. Or perhaps they can fend off alternate App Store talk for sake of protecting the integrity of the system. Or perhaps staying up too late makes me excessively conspiratorial.

Kalium 5y ago

A police raid on a person's home, or even a gentler thorough search, can be enough to quite seriously disrupt a person's life. Certainly having the police walk away with all your electronics in evidence bags will complicate trying to work remotely. Of course, this is assuming everything works as intended and they don't find anything else they can use to charge you with something as they search your home. If you smoke cannabis while being in the wrong state, you're now in several more kinds of trouble.

topynate 5y ago
This could happen with a perturbed image, but I doubt it. Apple will send the suspicious images to the relevant authorities. Those authorities will then look at the images. The chances are low that they will then seek a search, even though the images are innocent upon visual inspection. But maybe in some places a ping from Apple is good enough for a search and seizure.
- Kalium 5y ago
  In broad strokes, I agree with you. I think you're absolutely correct that most trained, educated, technologically sophisticated law enforcement bodies will do exactly as you suggest and decide that there's not enough to investigate. That said, I'm not willing to say it won't happen. There are too many law enforcement entities of wildly varying levels of professionalism, staffing, and technical sophistication. Someone innocent, somewhere, is likely to have a multi-year legal drama because their local PD got an email from Apple. And we haven't even gotten to subjects like how some LEOs will happily plant evidence once they decide you're guilty.
- falcolas 5y ago
  FWIW, they won't send the images. Even in the pursuit of knocking back CSAM, there are strict restrictions on the transmission and viewing of CSAM - in some cases even the defendant's lawyers don't usually see the images themselves in preparation for a trial, just a description of the contents. Apple employees or contractors will likely not look at the images themselves, only visual hashes. They will instead contact the police and say "Person X has Y images that are on list Z," and let the police get a warrant based off that information and execute it to check for actual CSAM.
  - topynate 5y ago
    On reflection, yes, there must be warrants involved. I'm raising my estimate of how likely it is that innocent people get raided due to this. The warrant would properly only be to search iCloud, not some guy's house, but I can easily see overly-broad warrants being issued.
    - gowld 5y ago
      > The warrant would properly only be to search iCloud, iCloud is encrypted, so that warrant is useless. They need to unlock and search the device.
      - topynate 5y ago
cirrus3 5y ago
The police do not show up until a human has compared the matched image to the actual. Just stop.
- teachrdan 5y ago
  The more collisions, the more chances of a false positive by a (tired, underpaid) human. I don't envy the innocent person whose home gets raided by a SWAT team convinced they're busting a child sex trafficker.
  - hda2 5y ago
    The database is known to contain non-csam images like porn. I doubt the reviewer will be qualified to discern which is which.
    - Operyl 5y ago
      Not that I doubt your motives, but can I get your source on this? It seems like a huge blunder if so.
    - jgreen401 5y ago
      Known by whom to contain legal images? The "reviewer" might well be a retired Pediatrician who can identify the age of the subject by looking at the photographs and will be extremely accurate.

e_proxus 5y ago

Wouldn’t it also just be possible to turn a jailbroken iDevice into a CSAM cleaner/hider? You could take actual CSAM, check if it matches the hashes and keep modifying the material until it doesn’t (adding borders, watermarking, changing dimensions etc.). Then just save it as usual without any risk.

cyanite 5y ago
No it’s not. The client hashes against a blinded set, and doesn’t know whether or not the hash is a hit or miss.
skygazer 5y ago
I can’t tell what would prevent a jailbroken device from pairing an illegal CSAM image with the safety voucher from a known-innocuous image, since the whole system depends on the trustworthiness of the safety voucher and that it truly represents the image in question. If the device is compromised I would think all bets would be off. To me, one potential flaw of this system may be that Apple inherently trusts the device. The existence of a jailbreak seems like a massive risk to the system. In fact, Apple themselves generate fake/meaningless safety vouchers a certain percentage of the time (see synthetic safety vouchers.) If a jailbroken phone could trigger that code path for all images in the pipeline, apple’s system would be completely broken. On the other hand, this may be just the excuse apple needs to lock down the phone further, to “protect the integrity of the CSAM detection system.” Perhaps they could persuade congress to make jailbreaking a federal crime. Perhaps they’re more clever than I ever imagined. Or perhaps they can fend off alternate App Store talk for sake of protecting the integrity of the system. Or perhaps staying up too late makes me excessively conspiratorial.