12 ms·

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dw

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). In an ordinary, non-adversarial context, the probability that you win the game is much less than one in one trillion, so this game is very safe for the casino. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play, and the relevant probability is no longer one in one trillion. The relevant number is 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.

mrtranscendence 5y ago
Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.
- e40 5y ago
  > The end result is that some peon at Apple has to look at the images and mark them as not CSAM Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.
  - fay59 5y ago
    It’s quite possible that you implied this, but I think that the true positives are the ones that had a mental health toll.
    - e40 5y ago
      That's correct. It wasn't just CSAM. The described images were sickening.
- nomel 5y ago
  Why is this question being downvoted? I too would like to know what this attack achieves. From what I see, the end result of false flagging is either someone has CSAM in iCloud and you push them over the threshold that results in reporting and prosecution, or there is no CASM, so the reviewer sees all of the hash collision images, including those that are natural. Is the problem that an attacker can force natural hash collision images to be viewed by a reviewer, violating that persons privacy? Do we know if this process is different than how Google, Facebook, Snapchat, Dropbox, Microsoft, and others have implemented these necessarily fuzzy matches for their CSAM scans of cloud hosted? Or am I missing something that the downvoters saw?
  - foota 5y ago
    The idea I've heard is that images could be generated that are sexual in nature but that have been altered to match a CSAM hash, making a tricky situation.
    - nomel 5y ago
      That's an interesting point! From my understanding, Apple's hash is not the final qualifier. SCMEC also reviews them before reporting to the authorities. But, I can imagine a scenario that might require opinion.
  - stefan_ 5y ago
    You are one underpaid random guy in India looking at CSAM all day clicking the wrong button away from a raid of your home and the end of your life as you know it.
    - Notanothertoo 5y ago
      This, these charges are damning once they are made. Plus the countless legal dollars you are going to have to front and hours spent proving innocence and that's assuming the justice system actually works.. Try explain this to your employer while you start missing deadlines due to court dates.. The police also could easily leverage this to warrant hop. As they have been found doing in the past. I think the bike rider who had nothing to do with a crime and got accused because he was the only one in a broad geo location Warren is all the president of of you need that this will be abused.
- saithound 5y ago
  Okay, let's play peon. Here are three perfectly legal and work-safe thumbnails of a famous singer: https://imgur.com/a/j40fMex https://imgur.com/a/j40fMex. The singer is underage in precisely one of the three photos. Can you decide which one? If your account has a large number of safety vouchers that trigger a CSAM match, then Apple will gather enough fragments to reassemble a secret key X (unique to your device) which they can use to decrypt the "visual derivatives" (very low resolution thumbnails) stored in all your matched safety vouchers. An Apple employee looks at the thumbnails derived from your photos. The only judgment call this employee gets to make is whether it can be ruled out (based on the way the thumbnail looks) that your uploaded photo is CSAM-related. As long as the thumbnail contains a person, or something that looks like the depiction of a person (especially in a vaguely violent or vaguely sexual context, e.g. with nude skin or skin with injuries) they will not be able to rule out this possibility based on the thumbnail alone. And they will not have access to anything else. Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well. This can be accomplished straightforwardly by using perfectly legal violent or sexual material to produce the collision (e.g. most people would not suspect foul play if they got a photo of genitals from their Tinder date). But much more sophisticated attacks [2] are also possible: since the computation of the visual derivative happens on the client, an adversary will be able to reverse engineer the precise algorithm. While 30 matching hashes are probably not sufficient to convict somebody, they're more than sufficient to make somebody a suspect. Reasonable suspicion is enough to get a warrant, which means search and seizure, computer equipment hauled away and subjected to forensic analysis, etc. If a victim works with children, they'll be fired for sure. And if they do charge somebody, it will be in Apple's very best interest not to assist the victim in any way: that would require admitting to faults in a high profile algorithm whose mere existence was responsible for significant negative publicity. In an absurdly unlucky case, the jury may even interpret "1 in 1 trillion chance of false positive" as "way beyond reasonable doubt". Chances are the FBI won't have the time to go after every report. But an attack may have consequences even if it never gets to the "warrant/charge/conviction" stage. E.g. if a victim ever gets a job where they need to obtain a security clearance, the Background Investigation Process will reveal their "digital footprint", almost certainly including the fact that the FBI got a CyberTipline Report about them. That will prevent them from being granted interim determination, and will probably lead to them being denied a security clearance. (See also my FAQ from the last thread [1], and an explanation of the algorithm [3]) [1] https://news.ycombinator.com/item?id=28232625 https://news.ycombinator.com/item?id=28232625 [2] https://graphicdesign.stackexchange.com/questions/106260/image-that-looks-completely-different-when-resized https://graphicdesign.stackexchange.com/questions/106260/ima... [3] https://news.ycombinator.com/item?id=28231218 https://news.ycombinator.com/item?id=28231218
- deleted 5y ago
  [deleted]
- csmpltn 5y ago
  > "The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it." This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.
  - incrudible 5y ago
    > You've cost Apple time and money by making them review each such fake report. Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.
    - playguardin 5y ago
      You have justified the review of an "innocent" image that may be of interest. To justify investigations and whatever else. It's pretty easy to understand... Big tech bad mmmkay!?
    - stemlord 5y ago
      It's a form of protest, thus not seen as a waste of time by those doing it.
    - syshum 5y ago
      >>Ok, but… how do I profit? To some people the ethical problems with this system are worth the effort to make their implementation impossible. if you are asking for monetary profit, that does not have to be a goal for someone to want to engage in disruption of this service.
    - incompatible 5y ago
      Perhaps somebody could make unflaggable CSAM images by giving them the same hashes as very widely available images.
      - 5y ago
- giantrobot 5y ago
  It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are presented as black boxes the image they perceive isn't audited or reviewed. There's zero recognition of this weakness by Apple or NCMEC (and their equivalents). For the system to even begin to be trustworthy all content would need to be reviewed raw and scaled-as-fed-into-the-algorithm. [0] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar...
  - skygazer 5y ago
    This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.
    - sennight 5y ago
      The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination. Feed that imagine into an adversarial network until it spits out the desired NeuralHash. Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.
- vineyardmike 5y ago
  > The end result is that some peon at Apple has to look at the images and mark them as not CSAM. As others said, if the non-csam looks sexual at all, they'll probably get flagged for post-apple review. Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives. As many mentioned, even an investigation can have horrible affects on some (innocent) person's life. I would not be shocked to learn that some crafty individual working at "meme factories" creating intentional collisions with distributed images just for "fun" - and politically motivated attacks seem plausible (eg. make liberal political memes flag CSAM). Then there are targeted motives for an attack. Have a journalist you want to attack or find a reason to warrant? Find them on dating app and send them nudes with CSAM collisions. Or any number of other targetted attacks against them.
  - k2enemy 5y ago
    > Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives. I would have thought the opposite. If there is a false positive that leads to an arrest and ruins someone's life, but the public sees that it is a false positive, then Apple will take an enormous hit in the marketplace. Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM. If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple.
    - partdavid 5y ago
      Another commenter used the right phrase, "terribly asymmetry". There won't be any such thing in the public eye as a "false positive", only kiddie porn traders who got away with it. > Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM. I don't think this is much of a risk. Defamation is difficult to prove, never applies to the law enforcement agencies who are going to make these arrests and "ruin people's lives", is never going to be down to Apple anyway (Apple is only referring things for investigation). I don't think you'll see much public indignance about this--even arguing against it in the real public eye (outside of wonky venues like HN) sounds tantamount to "supporting kiddie porn" in the naïve view of most members of the public. > If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple. This could potentially arise in any case with an abuser or producer or enabler or pimp or Maxwell/Epstein customer who has an iPhone; which is a lot of cases. As soon as Apple devices are supposed to "detect" kiddie porn, people will ask why people like this weren't caught earlier; and since Apple has money, they won't just ask this in the court of public opinion, they will sue for damages for abuses that "should have" been prevented by Apple's inspection of their pictures. Even if that's unlikely, it's much easier for a peon whose job it is to look at kiddie porn to just forward it on; and such a case really could damage Apple's optics.
- deleted 5y ago
  [deleted]
- cheald 5y ago
  Why would anyone bother calling the cops and telling them that someone they don't like is an imminent threat? The end result is that some officer just has to stop by and see that they aren't actually building bombs. You've cost someone a bit of time, but that's it.
  - syshum 5y ago
    Reality shows that not to be the case. How many "SWATTING" incidents now has resulted in much more than "some officer stopping by"....
    - cheald 5y ago
      My point precisely.
- 0x0 5y ago
  Seeing how "well" app review works, I would not be surprised if the "peon" sometimes clicks the wrong button while reviewing, bringing down a world of hurt on some innocent apple user, all triggered by the embedded snitchware running on their local device.
  - j16sdiz 5y ago
    I guess there are thousands of App reviews each week, each one take tens of minutes to test.. The CASM thing, I guess, give at most a dozen a week...
- mdoms 5y ago
  What if I can generate an attack that will mark your own picture of your own toddler nude in a bathtub as CSAM? Do you still feel confident in "some peon at Apple" to mark it as not CSAM?
- zionic 5y ago
  1) be a horrible human and want to troll 2) modify close up/ambiguous adult porn to be flagged as CP with free GitHub tool 3) batch a few thousand porn photos like this to poison them 4) upload them everywhere, 4chan/Reddit/tumblr/discord/imagefap 5) some poor sap manages to save 20+ of your bait images 6) apple reviewer sees 100x100px blurry gray image of definitely porn that was flagged as CP. hits report. 7) a SWAT team bust down your door and takes all your devices while you go to jail and your mugshot is everywhere Someone will do this, and the poisoned images will spread organically until they find a victim
  - smabie 5y ago
    This is actually a good idea. Maybe can convince apple this won't work.
  - owlbite 5y ago
    Hasn't this ship already sailed though? At least I'm given to understand that they already scanned all these photos uploaded to iCloud anyway (in the same way many other similar providers do). Whether it happens on the device or the server doesn't seem to make any difference to this attack. (That's not to say that (a) the scanning of stuff on a server was a good idea in the first place or (b) encouraging politicians to use your own device to spy on you is a good idea or (c) this isn't the thin end of a very painful wedge, just that we've not opened a new vulnerability)
  - rapnie 5y ago
    Plus by doing this at scale you generate such workload for reviewers that they are way more likely to quickly press 'report' in step 6.
- hypertele-Xii 5y ago
  Imagine costing everyone a bit of privacy all the time. Does that sound consistent with how Apple has positioned itself in the market?
- bildung 5y ago
  Because people behind the keyboards make mistakes all the time. Just in the last month i experienced * a call center agent at a haulage firm, instead of entering the delivery date we talked about on the phone, clicked for the delivery to be returned to the factory. * Google automatically blocked an ad account from delivering ads because we allegedly profiteered from Covid (untrue of course, but we surly talked about the challenges caused by the pandemic somewhere on the site, so the "AI" apparently got triggered by some keywords), and humans repeatedly confirmed the AI decision. * Facebook blocked an ad account that was unused in 2020, wanted ID, got the correct ID (identical name etc.), and the human denied confirmation. Google and Facebook are of course known to be beyond kafkaesque, so this is no surprise. But imagine the costs the innocents pay once they accidently get entered into the FBI CP suspect database.
- teekert 5y ago
  Some dudette/dude is going to look at my personal pictures every now and then? What if they are of my naked children and what if that person is a csam interested person? And she/he takes a picture of the screen? Ugh it feels so bad! I don’t want there to be a chance some person is going to look at my pics!
madeofpalk 5y ago
> one in a trillion chance of an account being flagged innocently To be clear - Apple claims that there is a one in a trillion chance of there being ~30 false matches in a single account.
- nitrogen 5y ago
  Is that number just for the private set intersection step?
raxxorrax 5y ago
This also server as a pretext to deeply search someone's device. So you must expect your device getting randomly searched by law enforcement. Completely ridiculous.

mrtranscendence 5y ago

Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.

e40 5y ago
> The end result is that some peon at Apple has to look at the images and mark them as not CSAM Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.
- fay59 5y ago
  It’s quite possible that you implied this, but I think that the true positives are the ones that had a mental health toll.
  - e40 5y ago
    That's correct. It wasn't just CSAM. The described images were sickening.
nomel 5y ago
Why is this question being downvoted? I too would like to know what this attack achieves. From what I see, the end result of false flagging is either someone has CSAM in iCloud and you push them over the threshold that results in reporting and prosecution, or there is no CASM, so the reviewer sees all of the hash collision images, including those that are natural. Is the problem that an attacker can force natural hash collision images to be viewed by a reviewer, violating that persons privacy? Do we know if this process is different than how Google, Facebook, Snapchat, Dropbox, Microsoft, and others have implemented these necessarily fuzzy matches for their CSAM scans of cloud hosted? Or am I missing something that the downvoters saw?
- foota 5y ago
  The idea I've heard is that images could be generated that are sexual in nature but that have been altered to match a CSAM hash, making a tricky situation.
  - nomel 5y ago
    That's an interesting point! From my understanding, Apple's hash is not the final qualifier. SCMEC also reviews them before reporting to the authorities. But, I can imagine a scenario that might require opinion.
- stefan_ 5y ago
  You are one underpaid random guy in India looking at CSAM all day clicking the wrong button away from a raid of your home and the end of your life as you know it.
  - Notanothertoo 5y ago
    This, these charges are damning once they are made. Plus the countless legal dollars you are going to have to front and hours spent proving innocence and that's assuming the justice system actually works.. Try explain this to your employer while you start missing deadlines due to court dates.. The police also could easily leverage this to warrant hop. As they have been found doing in the past. I think the bike rider who had nothing to do with a crime and got accused because he was the only one in a broad geo location Warren is all the president of of you need that this will be abused.
saithound 5y ago
Okay, let's play peon. Here are three perfectly legal and work-safe thumbnails of a famous singer: https://imgur.com/a/j40fMex https://imgur.com/a/j40fMex. The singer is underage in precisely one of the three photos. Can you decide which one? If your account has a large number of safety vouchers that trigger a CSAM match, then Apple will gather enough fragments to reassemble a secret key X (unique to your device) which they can use to decrypt the "visual derivatives" (very low resolution thumbnails) stored in all your matched safety vouchers. An Apple employee looks at the thumbnails derived from your photos. The only judgment call this employee gets to make is whether it can be ruled out (based on the way the thumbnail looks) that your uploaded photo is CSAM-related. As long as the thumbnail contains a person, or something that looks like the depiction of a person (especially in a vaguely violent or vaguely sexual context, e.g. with nude skin or skin with injuries) they will not be able to rule out this possibility based on the thumbnail alone. And they will not have access to anything else. Given the ability to produce hash collisions, an adversary can easily generate photos that fail this visual inspection as well. This can be accomplished straightforwardly by using perfectly legal violent or sexual material to produce the collision (e.g. most people would not suspect foul play if they got a photo of genitals from their Tinder date). But much more sophisticated attacks [2] are also possible: since the computation of the visual derivative happens on the client, an adversary will be able to reverse engineer the precise algorithm. While 30 matching hashes are probably not sufficient to convict somebody, they're more than sufficient to make somebody a suspect. Reasonable suspicion is enough to get a warrant, which means search and seizure, computer equipment hauled away and subjected to forensic analysis, etc. If a victim works with children, they'll be fired for sure. And if they do charge somebody, it will be in Apple's very best interest not to assist the victim in any way: that would require admitting to faults in a high profile algorithm whose mere existence was responsible for significant negative publicity. In an absurdly unlucky case, the jury may even interpret "1 in 1 trillion chance of false positive" as "way beyond reasonable doubt". Chances are the FBI won't have the time to go after every report. But an attack may have consequences even if it never gets to the "warrant/charge/conviction" stage. E.g. if a victim ever gets a job where they need to obtain a security clearance, the Background Investigation Process will reveal their "digital footprint", almost certainly including the fact that the FBI got a CyberTipline Report about them. That will prevent them from being granted interim determination, and will probably lead to them being denied a security clearance. (See also my FAQ from the last thread [1], and an explanation of the algorithm [3]) [1] https://news.ycombinator.com/item?id=28232625 https://news.ycombinator.com/item?id=28232625 [2] https://graphicdesign.stackexchange.com/questions/106260/image-that-looks-completely-different-when-resized https://graphicdesign.stackexchange.com/questions/106260/ima... [3] https://news.ycombinator.com/item?id=28231218 https://news.ycombinator.com/item?id=28231218
deleted 5y ago
[deleted]
csmpltn 5y ago
> "The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it." This can be abused to spam Apple's manual review process, grinding it down to a halt. You've cost Apple time and money by making them review each such fake report.
- incrudible 5y ago
  > You've cost Apple time and money by making them review each such fake report. Ok, but… how do I profit? If I wanted to waste Apple employee time, I could surely find a way to do it, but why would I? The functioning of society relies on the fact that people generally have better things to do than waste each others time.
  - playguardin 5y ago
    You have justified the review of an "innocent" image that may be of interest. To justify investigations and whatever else. It's pretty easy to understand... Big tech bad mmmkay!?
  - stemlord 5y ago
    It's a form of protest, thus not seen as a waste of time by those doing it.
  - syshum 5y ago
    >>Ok, but… how do I profit? To some people the ethical problems with this system are worth the effort to make their implementation impossible. if you are asking for monetary profit, that does not have to be a goal for someone to want to engage in disruption of this service.
  - incompatible 5y ago
    Perhaps somebody could make unflaggable CSAM images by giving them the same hashes as very widely available images.
    - 5y ago
giantrobot 5y ago
It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are presented as black boxes the image they perceive isn't audited or reviewed. There's zero recognition of this weakness by Apple or NCMEC (and their equivalents). For the system to even begin to be trustworthy all content would need to be reviewed raw and scaled-as-fed-into-the-algorithm. [0] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar...
- skygazer 5y ago
  This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.
  - sennight 5y ago
    The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination. Feed that imagine into an adversarial network until it spits out the desired NeuralHash. Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.
vineyardmike 5y ago
> The end result is that some peon at Apple has to look at the images and mark them as not CSAM. As others said, if the non-csam looks sexual at all, they'll probably get flagged for post-apple review. Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives. As many mentioned, even an investigation can have horrible affects on some (innocent) person's life. I would not be shocked to learn that some crafty individual working at "meme factories" creating intentional collisions with distributed images just for "fun" - and politically motivated attacks seem plausible (eg. make liberal political memes flag CSAM). Then there are targeted motives for an attack. Have a journalist you want to attack or find a reason to warrant? Find them on dating app and send them nudes with CSAM collisions. Or any number of other targetted attacks against them.
- k2enemy 5y ago
  > Beyond that, it doesn't seem to be in apple's interest to be conservative in flagging. An employee reviewer's best interest is to minimize false negatives not false positives. I would have thought the opposite. If there is a false positive that leads to an arrest and ruins someone's life, but the public sees that it is a false positive, then Apple will take an enormous hit in the marketplace. Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM. If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple.
  - partdavid 5y ago
    Another commenter used the right phrase, "terribly asymmetry". There won't be any such thing in the public eye as a "false positive", only kiddie porn traders who got away with it. > Nobody will want to take on the demonstrated real risk being falsely accused of possessing CSAM. I don't think this is much of a risk. Defamation is difficult to prove, never applies to the law enforcement agencies who are going to make these arrests and "ruin people's lives", is never going to be down to Apple anyway (Apple is only referring things for investigation). I don't think you'll see much public indignance about this--even arguing against it in the real public eye (outside of wonky venues like HN) sounds tantamount to "supporting kiddie porn" in the naïve view of most members of the public. > If they have a false negative, it is unclear to me what negative effects Apple would suffer. As far as I know, nobody would know about it outside of Apple. This could potentially arise in any case with an abuser or producer or enabler or pimp or Maxwell/Epstein customer who has an iPhone; which is a lot of cases. As soon as Apple devices are supposed to "detect" kiddie porn, people will ask why people like this weren't caught earlier; and since Apple has money, they won't just ask this in the court of public opinion, they will sue for damages for abuses that "should have" been prevented by Apple's inspection of their pictures. Even if that's unlikely, it's much easier for a peon whose job it is to look at kiddie porn to just forward it on; and such a case really could damage Apple's optics.
deleted 5y ago
[deleted]
cheald 5y ago
Why would anyone bother calling the cops and telling them that someone they don't like is an imminent threat? The end result is that some officer just has to stop by and see that they aren't actually building bombs. You've cost someone a bit of time, but that's it.
- syshum 5y ago
  Reality shows that not to be the case. How many "SWATTING" incidents now has resulted in much more than "some officer stopping by"....
  - cheald 5y ago
    My point precisely.
0x0 5y ago
Seeing how "well" app review works, I would not be surprised if the "peon" sometimes clicks the wrong button while reviewing, bringing down a world of hurt on some innocent apple user, all triggered by the embedded snitchware running on their local device.
- j16sdiz 5y ago
  I guess there are thousands of App reviews each week, each one take tens of minutes to test.. The CASM thing, I guess, give at most a dozen a week...
mdoms 5y ago
What if I can generate an attack that will mark your own picture of your own toddler nude in a bathtub as CSAM? Do you still feel confident in "some peon at Apple" to mark it as not CSAM?
zionic 5y ago
1) be a horrible human and want to troll 2) modify close up/ambiguous adult porn to be flagged as CP with free GitHub tool 3) batch a few thousand porn photos like this to poison them 4) upload them everywhere, 4chan/Reddit/tumblr/discord/imagefap 5) some poor sap manages to save 20+ of your bait images 6) apple reviewer sees 100x100px blurry gray image of definitely porn that was flagged as CP. hits report. 7) a SWAT team bust down your door and takes all your devices while you go to jail and your mugshot is everywhere Someone will do this, and the poisoned images will spread organically until they find a victim
- smabie 5y ago
  This is actually a good idea. Maybe can convince apple this won't work.
- owlbite 5y ago
  Hasn't this ship already sailed though? At least I'm given to understand that they already scanned all these photos uploaded to iCloud anyway (in the same way many other similar providers do). Whether it happens on the device or the server doesn't seem to make any difference to this attack. (That's not to say that (a) the scanning of stuff on a server was a good idea in the first place or (b) encouraging politicians to use your own device to spy on you is a good idea or (c) this isn't the thin end of a very painful wedge, just that we've not opened a new vulnerability)
- rapnie 5y ago
  Plus by doing this at scale you generate such workload for reviewers that they are way more likely to quickly press 'report' in step 6.
hypertele-Xii 5y ago
Imagine costing everyone a bit of privacy all the time. Does that sound consistent with how Apple has positioned itself in the market?
bildung 5y ago
Because people behind the keyboards make mistakes all the time. Just in the last month i experienced * a call center agent at a haulage firm, instead of entering the delivery date we talked about on the phone, clicked for the delivery to be returned to the factory. * Google automatically blocked an ad account from delivering ads because we allegedly profiteered from Covid (untrue of course, but we surly talked about the challenges caused by the pandemic somewhere on the site, so the "AI" apparently got triggered by some keywords), and humans repeatedly confirmed the AI decision. * Facebook blocked an ad account that was unused in 2020, wanted ID, got the correct ID (identical name etc.), and the human denied confirmation. Google and Facebook are of course known to be beyond kafkaesque, so this is no surprise. But imagine the costs the innocents pay once they accidently get entered into the FBI CP suspect database.
teekert 5y ago
Some dudette/dude is going to look at my personal pictures every now and then? What if they are of my naked children and what if that person is a csam interested person? And she/he takes a picture of the screen? Ugh it feels so bad! I don’t want there to be a chance some person is going to look at my pics!

madeofpalk 5y ago

> one in a trillion chance of an account being flagged innocently To be clear - Apple claims that there is a one in a trillion chance of there being ~30 false matches in a single account.

nitrogen 5y ago
Is that number just for the private set intersection step?

raxxorrax 5y ago

This also server as a pretext to deeply search someone's device. So you must expect your device getting randomly searched by law enforcement. Completely ridiculous.