2 ms·

I think you're missing the broader point. This isn't about any one step. I know you can make counter-arguments for each individual step in isolation, but in ord

I think you're missing the broader point. This isn't about any one step. I know you can make counter-arguments for each individual step in isolation, but in order for this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit. > True for millions of people But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here? > Because this is protected by security through obscurity we can't assess this step, but it's a standard practice in security assessment to assume obscure mechanisms add zero security. With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed. > So. Ordinary, perfectly lawful hardcore pornography-- Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets. > But also: At this point when someone is reviewing any of their images the users privacy is already violated. Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.) > I've demonstrated that for neuralhash you can make images match hashes without looking modified I haven't seen any examples that didn't look modified. That image of Lenna you posted earlier is quite clearly modified. > Apple is converting their customer privacy into a private profit. You couldn't possibly accuse them of doing so surreptitiously. If there are any consequences of such schemes being implemented, the consequences will have already happened because of similar implementations by Facebook, Google, etc. The only difference with Apple's approach is where certain CPU cycles are occurring. Apple could do all of the hash scanning on cloud servers (like Google) and the functional outcome with respect to your personal liberties would be exactly the same.

nullc 5y ago
> this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit. >> True for millions of people > But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here? Please try to imagine for a moment applying your same style of argument to a typical buffer overflow attack. Attackers aren't random processes. No CGI is going to get exploited and escalated to root access and website defacement through a random process, sure. Yet the attacks happen. The fact that the attack only works on 20-40% of people to begin with isn't particularly relevant in modeling the risk. > With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed. The same could have been said about the microsoft PhotoDNA ... they kept it secret to keep "bad guys" from access it. But I have a copy of it now. (also people successfully reproduced it from vague descriptions in academic papers) > Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets. I've certainly known people that had their phones full of porn. > Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.) Yes they do. They see ("a derivative of") every database match once the user has reached the threshold number of matches. Edit: On reread I understand what you were arguing there. I misunderstood you at first. I still disagree with you: Images users got from elsewhere are still their private files and are no less so because they didn't make them themselves. > That image of Lenna you posted earlier is quite clearly modified. I have a hard time believing that you aren't only saying that because you've seen the original. That image wouldn't strike anyone as unusual. ( nor would https://user-images.githubusercontent.com/858454/130296784-bdf329d0-06f6-454e-ba87-d83b010614a7.png https://user-images.githubusercontent.com/858454/130296784-b... ) > You couldn't possibly accuse them of doing so surreptitiously. I can-- but even if I couldn't, the fact that it's not very surreptitious doesn't make it good. I think Apple has mislead people about the level of privacy protection provided via cryptowashing, e.g. going on about the private set intersection when it doesn't provide any privacy for your users but just hides the database from the public to prevent accountability. Apple also made misleading claims e.g. about the false positive rate of neuralhash-- claiming that it had "trillion to one" FP rates, when its real FP rate in image collections is higher (e.g. there are colliding imagenet images) and constructed attack images will always FP. The fact that advocates have expended time and effort to dismantle Apple's claims doesn't change that they were misleading to begin with. :) > The only difference with Apple's approach is where certain CPU cycles are occurring. To the extent that other parties scanning also applies on person to person messaging that has a reasonable expectation of privacy, I would also hold that the scanning is unethical. But since the service providers own equipment is involved I would also freely admit that the ethically of it is debatable, at least if its disclosed, and that reasonable people could disagree. In the US this kind of scanning only ends up being lawful at all due to legal inconsistencies. If the communication were not stored it would be an unlawful wiretap (under both federal and many states laws) to scan users private communications. Saving it to a mailspool magically changes the law that applies, even if the behavior visible to the user is identical. But the distinction that it is the user's on the user's own device eliminates the "it's on the provider's server" excuse. Your own device being programmed against your will to surveil you and report you is just unethical, in my view. Like many ethical questions there is a boundary. Scanning public communications? Fine. Scanning third party private communications on your own devices? Uhh. Maybe, there be dragons. Your own device scanning your private files and reporting you against your will? Not okay. Even if the operating behavior is close to the same the principle matters.
- simondotau 5y ago
  You are still trying to skirt round the point that all stars must align for this attack to work. And there are enough stars to make it an absurd proposition. Multiply these likelihoods together and it quickly surpasses the likelihood that the user getting struck by lightning at the exact moment that they win the lottery. As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I know it's an inappropriate thing to say on HN but claiming that the "image wouldn't strike anyone as unusual" is so far beyond reasonable that I feel myself seriously questioning your sincerity.
  - nullc 5y ago
    > all stars must align for this attack to work. I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced. > As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I think it looks like a scan of a crappy smudged 30 year old photograph. Which is also exactly what it is. FWIW, I thought your comment saying the lena was obviously manipulated was transparently in bad faith, but I held my tongue-- so don't feel too guilty. I would take a sizable wager on that people wouldn't identify it as anything special, and I'm quite surprised to hear that you think differently. (In one of the threads I posted in it one of the people through it was the barbara image that was changed rather than the lena one...)
    - simondotau 5y ago
      > I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced. I'm not convinced that you're being serious. I rejected that response because it's self-evidently absurd. There were twelve mandatory prerequisites I identified for successful exploitation and the vast majority of them are not under the influence of any attacker, no matter how non-random the hypothetical attacker is. Do you understand what happens when you're forced to multiply probabilities? How does this attacker acquire a functioning copy of this secondary hash algorithm, which for all you know is held at least as securely than Apple holds private keys for stuff like iOS cryptographic signing. How does that attacker change the mind of a person who would never import hard core porn into their phone's photo gallery? Do you know if Apple's human review requires fooling one reviewer or multiple? I personally have no idea but as someone who has implemented a human review system for critical data, placing such consequential decisions in the hands of a single human is a non-starter. For all you know, the 30 images will be looked at by 30 different human reviewers. How does the attacker ensure that ALL of them will overlook the absence of a PRE-PUBESCENT CHILD in the photograph? No sir, I do not think you are not being serious. And that's intended as a compliment, because you're clearly a smart person.

nullc 5y ago

> this to work, every step needs to work. Every. Single. Step. This means it is beyond implausible to exploit. >> True for millions of people > But still only 20–40% of people. This requirement alone slashes the viability in half. See where I'm going here? Please try to imagine for a moment applying your same style of argument to a typical buffer overflow attack. Attackers aren't random processes. No CGI is going to get exploited and escalated to root access and website defacement through a random process, sure. Yet the attacks happen. The fact that the attack only works on 20-40% of people to begin with isn't particularly relevant in modeling the risk. > With respect, this is not security through obscurity. Nothing is merely obscured. The entire process is occurring in a secure, private space which you don't have access to. For your dastardly scheme to work, you need access to this second hash algorithm. Until you have this, you're stuck. You can't proceed. The same could have been said about the microsoft PhotoDNA ... they kept it secret to keep "bad guys" from access it. But I have a copy of it now. (also people successfully reproduced it from vague descriptions in academic papers) > Good luck getting your target to import 30 images of hardcore pornography into their iCloud Photo Gallery. Not impossible, but again, this will be a likely point of failure for at least 99% of targets. I've certainly known people that had their phones full of porn. > Apple's human reviewer doesn't ever see anyone's private photos. (Unless the hardcore porn you mangled happened to be their own, I suppose.) Yes they do. They see ("a derivative of") every database match once the user has reached the threshold number of matches. Edit: On reread I understand what you were arguing there. I misunderstood you at first. I still disagree with you: Images users got from elsewhere are still their private files and are no less so because they didn't make them themselves. > That image of Lenna you posted earlier is quite clearly modified. I have a hard time believing that you aren't only saying that because you've seen the original. That image wouldn't strike anyone as unusual. ( nor would https://user-images.githubusercontent.com/858454/130296784-bdf329d0-06f6-454e-ba87-d83b010614a7.png https://user-images.githubusercontent.com/858454/130296784-b... ) > You couldn't possibly accuse them of doing so surreptitiously. I can-- but even if I couldn't, the fact that it's not very surreptitious doesn't make it good. I think Apple has mislead people about the level of privacy protection provided via cryptowashing, e.g. going on about the private set intersection when it doesn't provide any privacy for your users but just hides the database from the public to prevent accountability. Apple also made misleading claims e.g. about the false positive rate of neuralhash-- claiming that it had "trillion to one" FP rates, when its real FP rate in image collections is higher (e.g. there are colliding imagenet images) and constructed attack images will always FP. The fact that advocates have expended time and effort to dismantle Apple's claims doesn't change that they were misleading to begin with. :) > The only difference with Apple's approach is where certain CPU cycles are occurring. To the extent that other parties scanning also applies on person to person messaging that has a reasonable expectation of privacy, I would also hold that the scanning is unethical. But since the service providers own equipment is involved I would also freely admit that the ethically of it is debatable, at least if its disclosed, and that reasonable people could disagree. In the US this kind of scanning only ends up being lawful at all due to legal inconsistencies. If the communication were not stored it would be an unlawful wiretap (under both federal and many states laws) to scan users private communications. Saving it to a mailspool magically changes the law that applies, even if the behavior visible to the user is identical. But the distinction that it is the user's on the user's own device eliminates the "it's on the provider's server" excuse. Your own device being programmed against your will to surveil you and report you is just unethical, in my view. Like many ethical questions there is a boundary. Scanning public communications? Fine. Scanning third party private communications on your own devices? Uhh. Maybe, there be dragons. Your own device scanning your private files and reporting you against your will? Not okay. Even if the operating behavior is close to the same the principle matters.

simondotau 5y ago
You are still trying to skirt round the point that all stars must align for this attack to work. And there are enough stars to make it an absurd proposition. Multiply these likelihoods together and it quickly surpasses the likelihood that the user getting struck by lightning at the exact moment that they win the lottery. As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I know it's an inappropriate thing to say on HN but claiming that the "image wouldn't strike anyone as unusual" is so far beyond reasonable that I feel myself seriously questioning your sincerity.
- nullc 5y ago
  > all stars must align for this attack to work. I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced. > As for your images, please. Your new example is substantially worse than your Lenna. Absolutely anyone could see that this image is unusual, even if they couldn't be sure if it was physical or digital damage. I think it looks like a scan of a crappy smudged 30 year old photograph. Which is also exactly what it is. FWIW, I thought your comment saying the lena was obviously manipulated was transparently in bad faith, but I held my tongue-- so don't feel too guilty. I would take a sizable wager on that people wouldn't identify it as anything special, and I'm quite surprised to hear that you think differently. (In one of the threads I posted in it one of the people through it was the barbara image that was changed rather than the lena one...)
  - simondotau 5y ago
    > I thought I addressed that pretty directly. "Attackers aren't random processes." -- but OK, feel free to be not convinced. I'm not convinced that you're being serious. I rejected that response because it's self-evidently absurd. There were twelve mandatory prerequisites I identified for successful exploitation and the vast majority of them are not under the influence of any attacker, no matter how non-random the hypothetical attacker is. Do you understand what happens when you're forced to multiply probabilities? How does this attacker acquire a functioning copy of this secondary hash algorithm, which for all you know is held at least as securely than Apple holds private keys for stuff like iOS cryptographic signing. How does that attacker change the mind of a person who would never import hard core porn into their phone's photo gallery? Do you know if Apple's human review requires fooling one reviewer or multiple? I personally have no idea but as someone who has implemented a human review system for critical data, placing such consequential decisions in the hands of a single human is a non-starter. For all you know, the 30 images will be looked at by 30 different human reviewers. How does the attacker ensure that ALL of them will overlook the absence of a PRE-PUBESCENT CHILD in the photograph? No sir, I do not think you are not being serious. And that's intended as a compliment, because you're clearly a smart person.