5 ms·

> being hardened against intentional false positives isn't necessary for it to serve its purpose. Except that every time they get a positive, the device will u

> being hardened against intentional false positives isn't necessary for it to serve its purpose. Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it. Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM. Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives. I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion). That won't go over well once the first people share such an experience.

simondotau 5y ago
This is false. 1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers. 2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review. 3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced. 4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
- tsimionescu 5y ago
  Your argument works just as well if for any x, NeuralHash(x)=1. The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
  - MauranKilom 5y ago
    > Your argument works just as well if for any x, NeuralHash(x)=1. That's good, because they weren't arguing that NeuralHash is infallible, but that its fallibility does not lead to the consequences outlined by the GP. > The fact is that a 'first line of defense' that is trivial to fool is useless. Useless to you, but not to apple. It reduces the set of pictures to process from "everything" to "those identified by the algorithm as designed, plus artificial false positives". It's impossible for the second set to be larger than the first, so it's by definition not useless. This is why it's strange how people are focusing so hard on the weaknesses of NeuralHash when what we should care about is how well the system as a whole works.
- DeathArrow 5y ago
  >The image would need also to match an unspecified second hash algorithm which runs on Apple servers. Security by obscurity in 2021?
  - simondotau 5y ago
    It’s not security by obscurity, it’s a privately held and operated secret.
- jdlshore 5y ago
  And 5. The “bad” hashes are secret, so it’s not possible to create a false positive that would trigger Apple’s human review.
  - charcircuit 5y ago
    They aren't all secret. Finding a site with pictures in that database is a single google search away.
    - maratc 5y ago
      The pictures aren't secret. The hashes are. We don't know what hash algorithm is employed on Apple's server side, so we can't feed that algorithm with these non-secret images and get the hashes.
      - charcircuit 5y ago
        The hash algorithm shouldn't matter if you have something you know it will detect
        simondotau 5y ago
        Sure, but then you have CSAM and Apple’s system is working as intended.
        skinkestek 5y ago
        no, that is the entire point: it is trivially simple to make an innocent picture match in neuralhash.
        nullc 5y ago
        You're talking across each other. Apple claims without evidence that there is a second secret hash which is run serverside. Perhaps it's just Microsoft PhotoDNA (which isn't secret anymore), perhaps it's just another copy of neuralhash (which might share the same collisions)-- it isn't currently known to the public. It is, however, known to anyone inside the project at apple and anyone involved in the abuse image databases at a minimum. I'm of the opinion that any security properties of this supposed second hash should be disregarded in public analysis of the system-- for all we know it's just a "return true". Other people make a reasonable argument that this additional function makes some of the attacks people describe harder. In practice it probably does, but security analysis should concern itself with the worst cases admitted by the known system.
dillondoyle 5y ago
If they implement should we all start intentionally spamming 'hot hash hits' to overwhelm the system? I guess someone would either need the CP hash already or break the law and morals to get one.
- DeathArrow 5y ago
  > I guess someone would either need the CP hash already or break the law and morals to get one. I presume Apple didn't break the law and morals to construct the hashes.
- charcircuit 5y ago
  Not everyone's morals are the same. There are a lot of people who don't think it's immoral to simply download an image. Especially if you didn't pay for it with money / ads.
  - Joker_vD 5y ago
    And some people don't think it's immoral to stare at CSAM all day long if it's what you're paid to do in order to confirm that it is indeed CSAM especially since this knowledge will be somehow used to prevent further creation of CSAM... somehow? Probably? I am being facetious, I know, but my point is that humans' morals and ethics are incredibly widely diverse ― especially if we're talking about the "effective" instead of "proclaimed" ethics (i.e., the values people actually follow instead of values they believe/claim to follow).
  - cto_of_antifa 5y ago
    Probably. Theyre wrong, though.
skinkestek 5y ago
> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This might actually be brilliant and we might end up thanking them for it. It might pretty much destroy this filter. > This is just a slight twist on the common prank of making people Google terms related to CSAM. And here is the problem: there is one failure mode of this and that is if a bunch of ordinary people get jailed for innocent memes before the public gets whiff of whats happening. So if they do they should probably be targeted as a waterhole "attack" at Apple execs and pro censorship politicians. Apple employees reading this: think very closely before downloading wallpapers the next few years.
maratc 5y ago
> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already That's a very optimistic view. In reality, they need to automate something while they never get any feedback whatsoever if their efforts are successful, let alone worthwhile. Crooks that attack Paypal at least know when they succeed - they see the money. In this case, not only they won't see the money, they won't know if what they're doing is "overwhelming" the system or not.
- quitit 5y ago
  What also seems to be missed is that it’s trivial to foil such attacks in an automated way. The work required on Apple’s end would be smaller than that of the attackers. Complaints about the proposal seem to overly focus on the reliability of singular aspects of the automated detection system when this has already been planned for - additionally those comments seem entirely unaware of the threshold technology in play.

simondotau 5y ago

This is false. 1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers. 2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review. 3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced. 4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.

tsimionescu 5y ago
Your argument works just as well if for any x, NeuralHash(x)=1. The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
- MauranKilom 5y ago
  > Your argument works just as well if for any x, NeuralHash(x)=1. That's good, because they weren't arguing that NeuralHash is infallible, but that its fallibility does not lead to the consequences outlined by the GP. > The fact is that a 'first line of defense' that is trivial to fool is useless. Useless to you, but not to apple. It reduces the set of pictures to process from "everything" to "those identified by the algorithm as designed, plus artificial false positives". It's impossible for the second set to be larger than the first, so it's by definition not useless. This is why it's strange how people are focusing so hard on the weaknesses of NeuralHash when what we should care about is how well the system as a whole works.
DeathArrow 5y ago
>The image would need also to match an unspecified second hash algorithm which runs on Apple servers. Security by obscurity in 2021?
- simondotau 5y ago
  It’s not security by obscurity, it’s a privately held and operated secret.
jdlshore 5y ago
And 5. The “bad” hashes are secret, so it’s not possible to create a false positive that would trigger Apple’s human review.
- charcircuit 5y ago
  They aren't all secret. Finding a site with pictures in that database is a single google search away.
  - maratc 5y ago
    The pictures aren't secret. The hashes are. We don't know what hash algorithm is employed on Apple's server side, so we can't feed that algorithm with these non-secret images and get the hashes.
    - charcircuit 5y ago
      The hash algorithm shouldn't matter if you have something you know it will detect
      - simondotau 5y ago
        Sure, but then you have CSAM and Apple’s system is working as intended.
        skinkestek 5y ago
        no, that is the entire point: it is trivially simple to make an innocent picture match in neuralhash.
        nullc 5y ago
        You're talking across each other. Apple claims without evidence that there is a second secret hash which is run serverside. Perhaps it's just Microsoft PhotoDNA (which isn't secret anymore), perhaps it's just another copy of neuralhash (which might share the same collisions)-- it isn't currently known to the public. It is, however, known to anyone inside the project at apple and anyone involved in the abuse image databases at a minimum. I'm of the opinion that any security properties of this supposed second hash should be disregarded in public analysis of the system-- for all we know it's just a "return true". Other people make a reasonable argument that this additional function makes some of the attacks people describe harder. In practice it probably does, but security analysis should concern itself with the worst cases admitted by the known system.

dillondoyle 5y ago

If they implement should we all start intentionally spamming 'hot hash hits' to overwhelm the system? I guess someone would either need the CP hash already or break the law and morals to get one.

DeathArrow 5y ago
> I guess someone would either need the CP hash already or break the law and morals to get one. I presume Apple didn't break the law and morals to construct the hashes.
charcircuit 5y ago
Not everyone's morals are the same. There are a lot of people who don't think it's immoral to simply download an image. Especially if you didn't pay for it with money / ads.
- Joker_vD 5y ago
  And some people don't think it's immoral to stare at CSAM all day long if it's what you're paid to do in order to confirm that it is indeed CSAM especially since this knowledge will be somehow used to prevent further creation of CSAM... somehow? Probably? I am being facetious, I know, but my point is that humans' morals and ethics are incredibly widely diverse ― especially if we're talking about the "effective" instead of "proclaimed" ethics (i.e., the values people actually follow instead of values they believe/claim to follow).
- cto_of_antifa 5y ago
  Probably. Theyre wrong, though.

skinkestek 5y ago

> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This might actually be brilliant and we might end up thanking them for it. It might pretty much destroy this filter. > This is just a slight twist on the common prank of making people Google terms related to CSAM. And here is the problem: there is one failure mode of this and that is if a bunch of ordinary people get jailed for innocent memes before the public gets whiff of whats happening. So if they do they should probably be targeted as a waterhole "attack" at Apple execs and pro censorship politicians. Apple employees reading this: think very closely before downloading wallpapers the next few years.

maratc 5y ago

> Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already That's a very optimistic view. In reality, they need to automate something while they never get any feedback whatsoever if their efforts are successful, let alone worthwhile. Crooks that attack Paypal at least know when they succeed - they see the money. In this case, not only they won't see the money, they won't know if what they're doing is "overwhelming" the system or not.

quitit 5y ago
What also seems to be missed is that it’s trivial to foil such attacks in an automated way. The work required on Apple’s end would be smaller than that of the attackers. Complaints about the proposal seem to overly focus on the reliability of singular aspects of the automated detection system when this has already been planned for - additionally those comments seem entirely unaware of the threshold technology in play.