6 ms·

It can't. No actions are taken on hashes alone. The procedure is, if an account uploads some number of images with matching hashes, those images are verified by

It can't. No actions are taken on hashes alone. The procedure is, if an account uploads some number of images with matching hashes, those images are verified by a human. This can attack that system itself, though, by overloading those humans with too much work looking at random noise, but that requires quite a large organised effort. It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.

atrus 5y ago
I doubt someone who is able/wanting to attack such a system would be unwilling to own some of the material themselves.
reuben_scratton 5y ago
"Verified by a human" - and that human will be an overworked, underpaid, overseas subcontractor who may well have an incentive to mash the "Confirm match" button from time to time to improve his performance.
- short_sells_poo 5y ago
  Beyond that, are we seriously going to ignore the fact that eventually this system will share the private photos of someone's naked kid with some random subcontractor? How is that even remotely OK? Or that it will find and share actual CSAM with said subcontractor?
  - FabHK 5y ago
    It only shares "visual derivatives" of images whose NeuralHash match the NeuralHash of known CSAM (either by being the same image ("perceptually") or a collision).
    - short_sells_poo 5y ago
      That to me just sounds like weasel words to avoid having to say that it shares images. Let's not beat about the bush, the "visual derivative" has to be good enough to identify what's going on in it for the manual confirmation. Are you actually arguing in good faith here at all? Because I can't see how a "visual derivative" that's nevertheless good enough for manual confirmation is any better than the source image?
      - cyanite 5y ago
        > Are you actually arguing in good faith here at all? Are you? Because you just seemed to claim that it could match against innocent pictures of your naked children, but this tells me that you don’t understand that this system looks for known pictures, not for something that looks like naked children. Edit: if you do, apologies, but then I’d say that Apple has suggested that it’s a low resolution version of the picture. This should be contrasted with server side scanning, where the server accesses all pictures fully.
  - stetrain 5y ago
    This system does not use ML to find new CSAM images. It only checks for ones already in a known database. Your pictures of kids in the bathtub are not on the list. What is show to the reviewer is a "visual derivative" which hasn't been clearly defined. A thumbnail image? Something with a censored section? We don't really know.
    - short_sells_poo 5y ago
      Yes I'm aware that it checks against a known database but clearly there can be collisions. So eventually it will share someone's private images.
      - robertoandred 5y ago
        Or rather, it will share some gray blob apparently.
        short_sells_poo 5y ago
        What is the point of sharing a gray blob? How is that going to prove anything?
        stetrain 5y ago
        I think the gray blob refers to engineered hash collisions, like the example in the article link.
      - bengale 5y ago
        It would need to have 30 collisions before anything even took place, which realistically isn't going to happen.
        brokenmachine 5y ago
        Most cameras nowadays can easily take a burst of 30 visually very similar images in a second. Lots of people leave their cameras in burst mode.
- apetrovic 5y ago
  I find it very hard to believe that Apple will put literal future of iPhone (just imagine the amount of bad press if one false negative comes out of review process and is reported to the authorities) to "overworked, underpaid overseas subcontractor". Apple is greedy, ruthless, tone-deaf machine, but I don't think they're stupid.
  - notriddle 5y ago
    That’s already how app reviews work.
    - apetrovic 5y ago
      I think rejecting one of million apps and angering one developer is a bit different than reporting someone about a child pornography/abuse, ruining their life and making a global scandal that can put future iPhone sales in jeopardy.
      - read_if_gay_ 5y ago
        Barely anyone ever cared about privacy. Convenience, no matter how insignificant, always wins. I will be surprised if this is the last straw that finally gets the masses to care about privacy. In reality there might be a few scandals but at some point no one will care anymore and Apple will remain popular.
- nomel 5y ago
  The "verified by a human" is only the the last step of the process for Apple. After that, it's given to NCMEC for review, then it's reported to authorities. The Apple subcontractor isn't the only one verifying things here. What you're saying is not realistic.
hughrr 5y ago
That assumes that the human review process is competent, your own images aren’t poisoned in some way (consider your own kids in the bath with some noise added) etc. In the mean time they lock your account which means your entire digital life stops dead until their review process is done. No way do I accept any of this. Also the hashes are on the device I understand and it’s not going to be that difficult to extract them.
- user-the-name 5y ago
  The human reviewer would be able to check against the exact image that generated the hash in the first place. Taking another completely unrelated image and perturbing it would be immediately obvious.
  - hughrr 5y ago
    So there’s an office somewhere with computers full of illegal child porn that people are staring at and comparing your photos to? There’s some irony in that.
    - henrikeh 5y ago
      Yes. That is called NCMEC in the US and it is a core aspect of how this whole process works. If you don’t understand the details of this, I’ll recommend this podcast episode which sums it up and discusses the implications https://atp.fm/443 https://atp.fm/443
      - hughrr 5y ago
        I am aware of the details. The episode of Brass Eye comes into context here, the relevant clip being as follows, showing exactly the issues of competence. https://youtu.be/_U-7L1tmBAo https://youtu.be/_U-7L1tmBAo
        henrikeh 5y ago
        I’m sorry, what is your point? I legitimately don’t understand what you mean.
csande17 5y ago
> It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography. I've never personally seen or looked for any images like this, but if they weren't already proliferating online and widely available to criminals, why would we need to build an elaborate client-side scanning system to detect and report people who have copies of them?
- user-the-name 5y ago
  Do you have access to darknet child pornography trading networks? I don't.
  - deleted 5y ago
    [deleted]
  - tzs 5y ago
    OK, now I'm curious. How does darknet stuff actually work? Is there some darknet search engine that you access over Tor and type whatever illegal thing you seek such as "child porn" or "stolen credit cards" or "heroin" or "money laundering" into and it points you to providers of those illegal goods and services reachable over some reasonably secure channel? Or is it one of those things where someone already using a particular child porn or stolen card or money laundering or heroin selling site has to put you in contact with them?
    - quenix 5y ago
      Hmm. Tor onion addresses are generally super long and have a large random element to them, such as facebookwkhpilnemxj7asaniu7vnjj.onion where the "Facebook" part is there for convenience and may likely not be present for actual illegal addresses. As such, you probably won't be able to find a CP website unless you actively brute force the entire onion address space. I don't know how the sharing actually happens, but I don't think a real search engine for the dark net exists. There are websites that try (Ahlia, I think?) but they are super limited in scope. I suspect addresses for drugs and CP are shared person-to-person.
    - 5y ago
nabakin 5y ago
Agreed. This attack still has consequences for people but they will not run into any legal trouble as far as I'm aware.
shmatt 5y ago
the blacklisted hashes are openly given to thousands of companies by a foundation called the National Center for Missing & Exploited Children, in both PhotoDNA and MD5 versions of the hashes. Yes, I can't give you an open link, but I imagine at least hundreds of thousands of people have access to download them. Some companies with better security than others It only takes 1 guy to sell it to an NSO-type company, who would then use it to target attacks for $$$
- user-the-name 5y ago
  Where would that $$$ come from? Like I said, the only attack possible is to overload the human reviewers, and even that is difficult to pull off except as a coordinated effort by lots of people.

atrus 5y ago

I doubt someone who is able/wanting to attack such a system would be unwilling to own some of the material themselves.

reuben_scratton 5y ago

"Verified by a human" - and that human will be an overworked, underpaid, overseas subcontractor who may well have an incentive to mash the "Confirm match" button from time to time to improve his performance.

short_sells_poo 5y ago
Beyond that, are we seriously going to ignore the fact that eventually this system will share the private photos of someone's naked kid with some random subcontractor? How is that even remotely OK? Or that it will find and share actual CSAM with said subcontractor?
- FabHK 5y ago
  It only shares "visual derivatives" of images whose NeuralHash match the NeuralHash of known CSAM (either by being the same image ("perceptually") or a collision).
  - short_sells_poo 5y ago
    That to me just sounds like weasel words to avoid having to say that it shares images. Let's not beat about the bush, the "visual derivative" has to be good enough to identify what's going on in it for the manual confirmation. Are you actually arguing in good faith here at all? Because I can't see how a "visual derivative" that's nevertheless good enough for manual confirmation is any better than the source image?
    - cyanite 5y ago
      > Are you actually arguing in good faith here at all? Are you? Because you just seemed to claim that it could match against innocent pictures of your naked children, but this tells me that you don’t understand that this system looks for known pictures, not for something that looks like naked children. Edit: if you do, apologies, but then I’d say that Apple has suggested that it’s a low resolution version of the picture. This should be contrasted with server side scanning, where the server accesses all pictures fully.
- stetrain 5y ago
  This system does not use ML to find new CSAM images. It only checks for ones already in a known database. Your pictures of kids in the bathtub are not on the list. What is show to the reviewer is a "visual derivative" which hasn't been clearly defined. A thumbnail image? Something with a censored section? We don't really know.
  - short_sells_poo 5y ago
    Yes I'm aware that it checks against a known database but clearly there can be collisions. So eventually it will share someone's private images.
    - robertoandred 5y ago
      Or rather, it will share some gray blob apparently.
      - short_sells_poo 5y ago
        What is the point of sharing a gray blob? How is that going to prove anything?
        stetrain 5y ago
        I think the gray blob refers to engineered hash collisions, like the example in the article link.
    - bengale 5y ago
      It would need to have 30 collisions before anything even took place, which realistically isn't going to happen.
      - brokenmachine 5y ago
        Most cameras nowadays can easily take a burst of 30 visually very similar images in a second. Lots of people leave their cameras in burst mode.
apetrovic 5y ago
I find it very hard to believe that Apple will put literal future of iPhone (just imagine the amount of bad press if one false negative comes out of review process and is reported to the authorities) to "overworked, underpaid overseas subcontractor". Apple is greedy, ruthless, tone-deaf machine, but I don't think they're stupid.
- notriddle 5y ago
  That’s already how app reviews work.
  - apetrovic 5y ago
    I think rejecting one of million apps and angering one developer is a bit different than reporting someone about a child pornography/abuse, ruining their life and making a global scandal that can put future iPhone sales in jeopardy.
    - read_if_gay_ 5y ago
      Barely anyone ever cared about privacy. Convenience, no matter how insignificant, always wins. I will be surprised if this is the last straw that finally gets the masses to care about privacy. In reality there might be a few scandals but at some point no one will care anymore and Apple will remain popular.
nomel 5y ago
The "verified by a human" is only the the last step of the process for Apple. After that, it's given to NCMEC for review, then it's reported to authorities. The Apple subcontractor isn't the only one verifying things here. What you're saying is not realistic.

hughrr 5y ago

That assumes that the human review process is competent, your own images aren’t poisoned in some way (consider your own kids in the bath with some noise added) etc. In the mean time they lock your account which means your entire digital life stops dead until their review process is done. No way do I accept any of this. Also the hashes are on the device I understand and it’s not going to be that difficult to extract them.

user-the-name 5y ago
The human reviewer would be able to check against the exact image that generated the hash in the first place. Taking another completely unrelated image and perturbing it would be immediately obvious.
- hughrr 5y ago
  So there’s an office somewhere with computers full of illegal child porn that people are staring at and comparing your photos to? There’s some irony in that.
  - henrikeh 5y ago
    Yes. That is called NCMEC in the US and it is a core aspect of how this whole process works. If you don’t understand the details of this, I’ll recommend this podcast episode which sums it up and discusses the implications https://atp.fm/443 https://atp.fm/443
    - hughrr 5y ago
      I am aware of the details. The episode of Brass Eye comes into context here, the relevant clip being as follows, showing exactly the issues of competence. https://youtu.be/_U-7L1tmBAo https://youtu.be/_U-7L1tmBAo
      - henrikeh 5y ago
        I’m sorry, what is your point? I legitimately don’t understand what you mean.

csande17 5y ago

> It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography. I've never personally seen or looked for any images like this, but if they weren't already proliferating online and widely available to criminals, why would we need to build an elaborate client-side scanning system to detect and report people who have copies of them?

user-the-name 5y ago
Do you have access to darknet child pornography trading networks? I don't.
- deleted 5y ago
  [deleted]
- tzs 5y ago
  OK, now I'm curious. How does darknet stuff actually work? Is there some darknet search engine that you access over Tor and type whatever illegal thing you seek such as "child porn" or "stolen credit cards" or "heroin" or "money laundering" into and it points you to providers of those illegal goods and services reachable over some reasonably secure channel? Or is it one of those things where someone already using a particular child porn or stolen card or money laundering or heroin selling site has to put you in contact with them?
  - quenix 5y ago
    Hmm. Tor onion addresses are generally super long and have a large random element to them, such as facebookwkhpilnemxj7asaniu7vnjj.onion where the "Facebook" part is there for convenience and may likely not be present for actual illegal addresses. As such, you probably won't be able to find a CP website unless you actively brute force the entire onion address space. I don't know how the sharing actually happens, but I don't think a real search engine for the dark net exists. There are websites that try (Ahlia, I think?) but they are super limited in scope. I suspect addresses for drugs and CP are shared person-to-person.
  - 5y ago

nabakin 5y ago

Agreed. This attack still has consequences for people but they will not run into any legal trouble as far as I'm aware.

shmatt 5y ago

the blacklisted hashes are openly given to thousands of companies by a foundation called the National Center for Missing & Exploited Children, in both PhotoDNA and MD5 versions of the hashes. Yes, I can't give you an open link, but I imagine at least hundreds of thousands of people have access to download them. Some companies with better security than others It only takes 1 guy to sell it to an NSO-type company, who would then use it to target attacks for $$$

user-the-name 5y ago
Where would that $$$ come from? Like I said, the only attack possible is to overload the human reviewers, and even that is difficult to pull off except as a coordinated effort by lots of people.