13 ms·

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’ve done something bad without evidence and imprison you. It happens all the time. It’s even happened in America. Just look up DNA exonerations - people have had DNA on the scene that literally proves their innocence and they’re still locked up. People should care about their governments being corrupt, not necessarily about this.

Dah00n 5y ago
Lots of things start out like this, then later they start looking for or blocking other things. Pirated movies, files leaked from three letter agencies, a picture of the president with a ** in his mouth, etc. Even if this is 100% bullet-proof it is still enough that Apple should be seen as privacy invading and leaking everything to the government (and others later on) as it will be abused if implemented everywhere. This isn't irrelevant because of other things happening that are also bad. This is added on-top of those broken system, like the ones in the US you mention.
minitoar 5y ago
The damage is already done by the time it gets to the point of devices being confiscated.
- endisneigh 5y ago
  Can you point to a single instance of that happening where it was due to a false positive?
  - bawolff 5y ago
    In a proposed system that has yet to be deployed?
    - endisneigh 5y ago
      This has already existed in PhotoDNA since 2008. This is not new
      - toby- 5y ago
        Examples of on-device scanning via PhotoDNA?
        robertoandred 5y ago
        People’s photo libraries are scanned. The result is the same.
        minitoar 5y ago
        Only if you’re putting your stuff in the cloud. There’s a big difference between my files on my computer and my files on someone else’s computer. Or at least there should be, IMO.
        robertoandred 5y ago
        Yeah, only if you're putting your stuff in the cloud. The result is the same.
      - bawolff 5y ago
        But that's deployed in a very different way which makes the concerns being discussed much less likely to happen. Specificly the person doing the scanning already has access to photos and can double check the results without having to sieze the device, a rather public process.
  - Unklejoe 5y ago
    If we reduce it down to "someone being accused of a crime but later being found innocent", then there are several, and when it comes to sex crimes, the accusation alone is enough to ruin someone. People don't care about minor details such as the fact that the person was later found innocent. To them, it's the same as getting off on a technicality.
    - endisneigh 5y ago
      Ok yes - now then, PhotoDNA has existed since 2008. When has what you’ve described ever happened?
      - hda2 5y ago
        PhotoDNA relies on perceptual hashing?
- rootusrootus 5y ago
  By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions.
  - headShrinker 5y ago
    > By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions. Not necessarily. The FBI knocks on your door when they have convinced a warrant-signing judge, that’s there is probable-cause, or that additional information can be collected to build a case in which the defendant will ‘plead’ or make a deal for sentence reduction. 90% of defendants make a deal and never go to trial because by that time the stakes are so high a guilty verdict includes the harshest possible sentence. FBI only needs to convince you they can win a case or the judge that there might be fruit behind a locked door, they don’t need direct evidence. It’s really up to the judge.
  - stjohnswarts 5y ago
    I doubt it, all they need is this stuff and they can get a warrant to rummage through your stuff and take all your computers, usb drives, etc and also put your name on a watch list and your permanent record. Much like newspapers retracting mistakes if the story doesn't pan out, it goes on the back page. Plenty enough to wreck your life.
  - heavyset_go 5y ago
    No, the FBI's job is to investigate whether or not a crime has been committed. They'll do this by getting a warrant for your devices, and then take them and scan them.
- mrtranscendence 5y ago
  If you don't actually have CSAM then the person at Apple who visually audits the collision set will not send your info to law enforcement, and nothing will be confiscated. Apple doesn't just take any instance of a collision and send it to the FBI.
  - short_sells_poo 5y ago
    Right, because humans are infallible and never abuse their position either. And we are not talking about some underpaid contractors making life altering decisions. And we are also not talking about a system where just a false accusation can and will destroy one's life irreparably.
bawolff 5y ago
If the two images looked the same, then the expected behaviour is a collision, so if collisions matter at all, it would only be for pictures that look different.
- endisneigh 5y ago
  They don’t matter because if two images don’t look the same, but collide - then human processes will absolve you. This isn’t some AI that sends you straight to prison lol
  - bawolff 5y ago
    My point is - if you hold that view, then collisions shouldn't matter at all, since if they look the same the correct action is for the person to get thrown in jail.
  - yusefnapora 5y ago
    Imagine this scenario. - You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll. - These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database. - Apple's manual review kicks in because you have enough such images to trigger the threshold. - The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side. - Your case is forwarded to the FBI, who now have cause to turn your life upside down. This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be. [1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-901769661 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
    - ClumsyPilot 5y ago
      "- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side." Given how most companies have barely paid, clueless and PTSD suffering human reviewers, a litany of mistakes is to be expected. We should expect all the coherence of the twitter's nipple policy, except now it put you in jail or at least ruins your life with legal fees
zbobet2012 5y ago
That's not the route in which this will be exploited, and used at scale. A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you. This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.
- endisneigh 5y ago
  What you’re saying is already possible. Where are the examples of this happening? PhotoDNA has existed since 2008.
  - biesnecker 5y ago
    Who is scanning your phone right now with PhotoDNA?
    - endisneigh 5y ago
      Microsoft, Facebook and Google. More depending on what services you use
      - biesnecker 5y ago
        They're not actively scanning your phone, they're actively scanning files you send them.
        endisneigh 5y ago
        That’s the same as what Apple is going to do - scan right before sending to iCloud
        danuker 5y ago
        The scanning on the device before the upload makes it easier to do surveillance.
thedingwing 5y ago
You don’t need to be sent to prison to be irreparably harmed by an accusation.
- endisneigh 5y ago
  Ok sure, PhotoDNA has existed since 2008. Where are the instances of people being sent to prison?
  - Dah00n 5y ago
    You are commenting a lot for this many places in the thread. Are you arguing for this system or for Apple? It reads like pro-Apple and doesn't add anything except "I think it is good, therefore it is good".
    - rootusrootus 5y ago
      It can get a little frustrating to hear so much inaccurate FUD being spread around which detracts from a reasoned discussion on the merits.
    - qweqwweqwe-90i 5y ago
      FYI: when you wrote this comment, you had posted 5% of all the comments on this thread.
    - kemayo 5y ago
      If you have a point which you feel rebuts a common argument, it seems reasonable to leave that comment in places you see that argument. The alternative is "minority positions should be drowned out", no?
  - dannyobrien 5y ago
    One of the things that is happening now is that the entire PhotoDNA system is finally coming under the level of oversight that it should have had right from the start. I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, because people simply would not understand the context. Before this Apple scandal, if you'd written to your reepresentative or a journalist or an activist group and said "I was framed for child abuse because of computer program that misidentified innocent pictures", they would attach a very low priority to dealing with you or publicising this. And almost all people who have experienced this kind of nightmare really don't want to re-live it in public for some tiny possibility of real justice being served for them, or for others. They just want it to all go away.
mikepurvis 5y ago
At least part of the concern is that a hash collision is basically "cause" for Apple to then dump and begin manually (like, with humans) reviewing the contents of your device, all of which will be happening behind the closed doors of a private corporation, outside of any of the usual oversight or innocent-presumption mechanisms that come from it happening through the courts. That, combined with a (pretty reasonable) expectation that the pool of sus hashes will greatly expand as law enforcement and others begin to understand what a convenient side-step this is for due process.
- shuckles 5y ago
  That is literally the status quo with every cloud service. Apple, unlike the others, has said that they will evaluate you on the basis of what’s included in the associated data of your safety voucher, and you can inspect those contents because they’re shipped in the client. Facebook, for all I know, might be calculating a child predator likelihood score on my account based on how often I look up my middle school ex-girlfriend on Instagram. In addition, “pretty reasonable” is an opinion not fact. Where is the evidence that PhotoDNA hashes have been compromised in this way in the fifteen years they’ve been used?
  - foerbert 5y ago
    I don't think we can just appeal to the status quo here and assume it's acceptable. There's a couple reasons. First, how many people really understood this previously? Did society at large actually knowingly accept the current state of things, or did it just happen without most people realizing it? Even here on HN where we'd expect to find people way more knowledgeable about it than in general I'm not sure how well known it was about what was actually happening, though I'd assume most would be aware it was possible. Secondly, there's a significant difference between your own device or Apple's server doing this. On the technical side of things, right now, it might not matter that much since it currently is limited to things you upload to iCloud. But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them.
stickfigure 5y ago
Let's say I get you to click on a link. That link contains a thumbnail gallery of CSAM. With the right CSS, you might not even notice it, but it's in your browser's cache and on your filesystem. Lots of pictures - more than enough for your phone to snitch on you. All because you clicked on a link. Phishing attacks can now put you in prison, label you as a pedophile and sex offender, and destroy your life.
- endisneigh 5y ago
  This is already possible with PhotoDNA. When has it happened?
  - stickfigure 5y ago
    Your browser cache is not being synced to iCloud; Apple doesn't see it. So no, it is not currently possible.
    - endisneigh 5y ago
      I'm not talking about Apple, I'm saying in general technology has already been deployed to make what you're describing possible. So where's the evidence of abuse?
      - stickfigure 5y ago
        No vendors are snooping your phone/computer browser cache, so this attack vector does not yet exist. Apple is building it.
        zimpenfish 5y ago
        > No vendors are snooping your phone/computer browser cache a) That you know of b) Apple won't be doing this either > Apple is building it. No, they aren't.
- dylan604 5y ago
WesolyKubeczek 5y ago
You know, it's rather not okay to treat every smartphone user out there as a potential criminal just because they happen to have photos on their devices. At least in an alleged democracy where there's this presumption of innocence thing. Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone. But what can I say, there wasn't much uproar about piracy tax on blank CD-R(W) media back in the day, so why not have that now. And eventually we go peak USSR where everyone and their dog is suspect and whoever is arrested is the enemy of the people. Yay, it's somehow reassuring to know I won't live long enough to see it.
- nonbirithm 5y ago
  And it's fine to treat everyone as a potential criminal when we entrust our data on the same company's servers? No matter if on-device scanning makes surveillance easier than ever, the surveillance itself was still a significant possibility up to this point with server-side scanning. Imagine how many petabytes of user data already exist in the cloud.
  - short_sells_poo 5y ago
    So in your mind because bad thing X is already happening, it's completely OK for bad thing XY to also start happening?
    - nonbirithm 5y ago
      No, it's that in spite of there already being an invasive scanning process in place for this long at every major tech company that handles user data, nobody seemed to care until now.
      - hda2 5y ago
        Most people weren't aware or couldn't see where this was going. Now that it's becoming obvious, people are starting to care.
      - 5y ago
Youden 5y ago
> It doesn’t matter if there are collisions if the two images don’t actually look the same. Is that really true? My understanding is that the manual reviewers at Apple only see some kind of low-resolution proxy, not the full-resolution image. I'd also be shocked if the human reviewers were shown the original, actually CP image, to compare to. Given that, it's not necessary to produce an actual visual match, it's just necessary to produce an image that when scaled-down looks like CSAM (e.g. take an actual photo of a kid at the beach and photoshop in some skin-coloured swimwear with creases in the right places). > Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? The attack I'd worry about here is similar to swatting. Someone who doesn't like me sends a bunch of images like the ones I described above (not just one), they end up synced to iCloud (because Apple wants _everything_ synced to iCloud) and Apple reports me to the authorities, who end up knocking at my door and arresting me. Even though I'm innocent, I'll probably have most of my computers confiscated for a while and spend a few days locked up. > PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. PhotoDNA's algorithms and hashes aren't public, so it's not clear how an attacker would exploit PhotoDNA in the way that people are afraid will be done for Apple. PhotoDNA also isn't, as far as I know, part of a product that aims to create unprotected backups of the phones of nearly a billion users. Apple really wants you to upload your whole phone to iCloud. The only comparable alternative is Google's Android backup but Google does the right thing and end-to-end encrypts that.
- robertoandred 5y ago
  Why would you save all of the almost-CSAM pics the attacker sends you to your photo library?
  - Youden 5y ago
    Manually, perhaps because the attacker crafts the messages to make that desirable. But a bigger concern is automatically. WhatsApp for example can automatically save all received photos to your Camera Roll, which is of course automatically backed up to iCloud for many people. So an attacker could potentially just send you a bundle of 40 images of however many and your phone automatically sends it to Apple.
sennight 5y ago
We are talking about how everyone who gave Apple money now has a potential probable cause vector that they didn't before. Everyone running the software is a suspect by default. Ask black Americans how they feel about setting the bar low for probable cause. "Following the 2004 Madrid train bombings, fingerprints on a bag containing detonating devices were found by Spanish authorities. The Spanish National Police shared the fingerprints with the FBI through Interpol. Twenty possible matches for one of the fingerprints were found in the FBI database and one of the possible matches was Brandon Mayfield. His prints were in the FBI database as they were taken as part of standard procedure when he joined the military." "The FBI described the fingerprint match as '100% verified'." https://en.wikipedia.org/wiki/Brandon_Mayfield https://en.wikipedia.org/wiki/Brandon_Mayfield
- dane-pgp 5y ago
  > '100% verified' Just reading those words is rage-inducing, but I'm grateful to have learnt this example of government lying. I feel like it should become an expression that societies teach to their children to warn them about abuses of power. Other mottoes synonymous with government deception and corruption come to mind, but at the risk of being too controversial I will share only their initials and dates: "SAARH" (2013), "MA" (2003), "IANAC" (1973), "NHDAEMZE" (1961), "AMF" (1940).
  - sennight 5y ago
    Because I delight in delivering bad news, I'll point out that the real takeaway shouldn't be that federal LEOs regularly lie (though, they do) - it is that they are permitted to lie convincingly through handwavy technical means. All these tools are designed to give them permission to totally destroy your life. I'm aware of only two geewiz CSI methods that are actually founded in science: cryptography (this neural crap doesn't qualify) and DNA. Unlike fingerprints and bitemark analysis, those two tools were invented outside of law enforcement - instead of being purpose built for prosecution. Anybody doubting that should look into the history of the polygraph and its continued use in the face of evidence demonstrating how useless it is in the pursuit of truth... which begs the question: if they aren't interested in the truth, what are they doing? https://en.wikipedia.org/wiki/Polygraph https://en.wikipedia.org/wiki/Polygraph
zug_zug 5y ago
And what if the FBI demands a list of all Apple users who have matched even 1 CSAM photo for their own private watchlist?
- mrtranscendence 5y ago
  And what if the FBI demands that Apple leadership genuflect toward an oil painting of J Edgar Hoover? The FBI can demand all sorts of things. In this case, Apple isn't tracking collisions as small as 1 photo.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
- theluketaylor 5y ago
  There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Apple is using secret splitting to make sure they cannot decode the CSAM ticket until the threshold is reached. Devices also produce some synthetic matches to prevent themselves Apple (or anyone else) inferring a pre-threshold count based on the number of vouchers. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... Threshold Secret Sharing Synthetic Match Vouchers
  - zug_zug 5y ago
    Thanks. You're right. Very sophisticated system. Still find it morally repugnant, but technologically it is fairly thorough.
  - heavyset_go 5y ago
    > There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.
cwkoss 5y ago
> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? I believe a single image flag could be used as pretext to arrest someone that the government has already deemed an enemy of the state. Ex. Someone like Julian Assange
trhway 5y ago
> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? being on a government maintained secret list of CSAM flagged would probably have a lot of consequences in your life going forward without you knowing about that. It may be just one innocent image which accidentally matched the hash, yet without any procedure to get you off that list or even just to learn whether you're on that list ... >If some corrupt government wants to get you they don’t need this. it isn't corrupt government which is a threat here. It is a well intended well functioning relatively uncorrupt government which spends a lot of effort to protect its society from terrorism, child porn, sex trafficking, drugs, etc. In case of corrupt government the situation is kind of easier - you can always learn and correct necessary things through the corrupt government officials by paying a corresponding "fee".
Gibbon1 5y ago
Well there is this here https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_arrest https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_...
InTheArena 5y ago
A honest question I have is if PhotoDNA results where ever validated? Is there a chance that collisions have been occurring with no one looking to see if actual CSAM was in use?
nr2x 5y ago
"If some corrupt government wants to get you they don’t need this" -> if some corrupt government wants to *FIND YOU SHARING IDEAS THEY DON'T LIKE* they can *DEFINITELY* use this
yunesj 5y ago
You ask for examples a few times. Here’s one. Computers confiscated from a liberty activist and his radio station. No CP-related charges pressed in the 5 years since they confiscated his computers. I’m guessing most times this happens, the accused try to keep it on the DL. https://www.wmur.com/article/fbi-raids-radio-talk-show-host-s-home-in-keene-sunday-morning/5209568# https://www.wmur.com/article/fbi-raids-radio-talk-show-host-...
joe_the_user 5y ago
It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? I understand that the system as stated today has multiple safeguards against such things happening but... Given the sum-total of bs I've seen happening in this country, yes, believe that kind of thing is quite possible.

Dah00n 5y ago

Lots of things start out like this, then later they start looking for or blocking other things. Pirated movies, files leaked from three letter agencies, a picture of the president with a ** in his mouth, etc. Even if this is 100% bullet-proof it is still enough that Apple should be seen as privacy invading and leaking everything to the government (and others later on) as it will be abused if implemented everywhere. This isn't irrelevant because of other things happening that are also bad. This is added on-top of those broken system, like the ones in the US you mention.

minitoar 5y ago

The damage is already done by the time it gets to the point of devices being confiscated.

endisneigh 5y ago
Can you point to a single instance of that happening where it was due to a false positive?
- bawolff 5y ago
  In a proposed system that has yet to be deployed?
  - endisneigh 5y ago
    This has already existed in PhotoDNA since 2008. This is not new
    - toby- 5y ago
      Examples of on-device scanning via PhotoDNA?
      - robertoandred 5y ago
        People’s photo libraries are scanned. The result is the same.
        minitoar 5y ago
        Only if you’re putting your stuff in the cloud. There’s a big difference between my files on my computer and my files on someone else’s computer. Or at least there should be, IMO.
        robertoandred 5y ago
        Yeah, only if you're putting your stuff in the cloud. The result is the same.
    - bawolff 5y ago
      But that's deployed in a very different way which makes the concerns being discussed much less likely to happen. Specificly the person doing the scanning already has access to photos and can double check the results without having to sieze the device, a rather public process.
- Unklejoe 5y ago
  If we reduce it down to "someone being accused of a crime but later being found innocent", then there are several, and when it comes to sex crimes, the accusation alone is enough to ruin someone. People don't care about minor details such as the fact that the person was later found innocent. To them, it's the same as getting off on a technicality.
  - endisneigh 5y ago
    Ok yes - now then, PhotoDNA has existed since 2008. When has what you’ve described ever happened?
    - hda2 5y ago
      PhotoDNA relies on perceptual hashing?
rootusrootus 5y ago
By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions.
- headShrinker 5y ago
  > By the time the FBI comes knocking for your devices, they have a lot of evidence, not a list of hash collisions. Not necessarily. The FBI knocks on your door when they have convinced a warrant-signing judge, that’s there is probable-cause, or that additional information can be collected to build a case in which the defendant will ‘plead’ or make a deal for sentence reduction. 90% of defendants make a deal and never go to trial because by that time the stakes are so high a guilty verdict includes the harshest possible sentence. FBI only needs to convince you they can win a case or the judge that there might be fruit behind a locked door, they don’t need direct evidence. It’s really up to the judge.
- stjohnswarts 5y ago
  I doubt it, all they need is this stuff and they can get a warrant to rummage through your stuff and take all your computers, usb drives, etc and also put your name on a watch list and your permanent record. Much like newspapers retracting mistakes if the story doesn't pan out, it goes on the back page. Plenty enough to wreck your life.
- heavyset_go 5y ago
  No, the FBI's job is to investigate whether or not a crime has been committed. They'll do this by getting a warrant for your devices, and then take them and scan them.
mrtranscendence 5y ago
If you don't actually have CSAM then the person at Apple who visually audits the collision set will not send your info to law enforcement, and nothing will be confiscated. Apple doesn't just take any instance of a collision and send it to the FBI.
- short_sells_poo 5y ago
  Right, because humans are infallible and never abuse their position either. And we are not talking about some underpaid contractors making life altering decisions. And we are also not talking about a system where just a false accusation can and will destroy one's life irreparably.

bawolff 5y ago

If the two images looked the same, then the expected behaviour is a collision, so if collisions matter at all, it would only be for pictures that look different.

endisneigh 5y ago
They don’t matter because if two images don’t look the same, but collide - then human processes will absolve you. This isn’t some AI that sends you straight to prison lol
- bawolff 5y ago
  My point is - if you hold that view, then collisions shouldn't matter at all, since if they look the same the correct action is for the person to get thrown in jail.
- yusefnapora 5y ago
  Imagine this scenario. - You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll. - These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database. - Apple's manual review kicks in because you have enough such images to trigger the threshold. - The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side. - Your case is forwarded to the FBI, who now have cause to turn your life upside down. This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be. [1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-901769661 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
  - ClumsyPilot 5y ago
    "- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side." Given how most companies have barely paid, clueless and PTSD suffering human reviewers, a litany of mistakes is to be expected. We should expect all the coherence of the twitter's nipple policy, except now it put you in jail or at least ruins your life with legal fees

zbobet2012 5y ago

That's not the route in which this will be exploited, and used at scale. A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you. This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.

endisneigh 5y ago
What you’re saying is already possible. Where are the examples of this happening? PhotoDNA has existed since 2008.
- biesnecker 5y ago
  Who is scanning your phone right now with PhotoDNA?
  - endisneigh 5y ago
    Microsoft, Facebook and Google. More depending on what services you use
    - biesnecker 5y ago
      They're not actively scanning your phone, they're actively scanning files you send them.
      - endisneigh 5y ago
        That’s the same as what Apple is going to do - scan right before sending to iCloud
        danuker 5y ago
        The scanning on the device before the upload makes it easier to do surveillance.

thedingwing 5y ago

You don’t need to be sent to prison to be irreparably harmed by an accusation.

endisneigh 5y ago
Ok sure, PhotoDNA has existed since 2008. Where are the instances of people being sent to prison?
- Dah00n 5y ago
  You are commenting a lot for this many places in the thread. Are you arguing for this system or for Apple? It reads like pro-Apple and doesn't add anything except "I think it is good, therefore it is good".
  - rootusrootus 5y ago
    It can get a little frustrating to hear so much inaccurate FUD being spread around which detracts from a reasoned discussion on the merits.
  - qweqwweqwe-90i 5y ago
    FYI: when you wrote this comment, you had posted 5% of all the comments on this thread.
  - kemayo 5y ago
    If you have a point which you feel rebuts a common argument, it seems reasonable to leave that comment in places you see that argument. The alternative is "minority positions should be drowned out", no?
- dannyobrien 5y ago
  One of the things that is happening now is that the entire PhotoDNA system is finally coming under the level of oversight that it should have had right from the start. I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, because people simply would not understand the context. Before this Apple scandal, if you'd written to your reepresentative or a journalist or an activist group and said "I was framed for child abuse because of computer program that misidentified innocent pictures", they would attach a very low priority to dealing with you or publicising this. And almost all people who have experienced this kind of nightmare really don't want to re-live it in public for some tiny possibility of real justice being served for them, or for others. They just want it to all go away.

mikepurvis 5y ago

At least part of the concern is that a hash collision is basically "cause" for Apple to then dump and begin manually (like, with humans) reviewing the contents of your device, all of which will be happening behind the closed doors of a private corporation, outside of any of the usual oversight or innocent-presumption mechanisms that come from it happening through the courts. That, combined with a (pretty reasonable) expectation that the pool of sus hashes will greatly expand as law enforcement and others begin to understand what a convenient side-step this is for due process.

shuckles 5y ago
That is literally the status quo with every cloud service. Apple, unlike the others, has said that they will evaluate you on the basis of what’s included in the associated data of your safety voucher, and you can inspect those contents because they’re shipped in the client. Facebook, for all I know, might be calculating a child predator likelihood score on my account based on how often I look up my middle school ex-girlfriend on Instagram. In addition, “pretty reasonable” is an opinion not fact. Where is the evidence that PhotoDNA hashes have been compromised in this way in the fifteen years they’ve been used?
- foerbert 5y ago
  I don't think we can just appeal to the status quo here and assume it's acceptable. There's a couple reasons. First, how many people really understood this previously? Did society at large actually knowingly accept the current state of things, or did it just happen without most people realizing it? Even here on HN where we'd expect to find people way more knowledgeable about it than in general I'm not sure how well known it was about what was actually happening, though I'd assume most would be aware it was possible. Secondly, there's a significant difference between your own device or Apple's server doing this. On the technical side of things, right now, it might not matter that much since it currently is limited to things you upload to iCloud. But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them.

stickfigure 5y ago

Let's say I get you to click on a link. That link contains a thumbnail gallery of CSAM. With the right CSS, you might not even notice it, but it's in your browser's cache and on your filesystem. Lots of pictures - more than enough for your phone to snitch on you. All because you clicked on a link. Phishing attacks can now put you in prison, label you as a pedophile and sex offender, and destroy your life.

endisneigh 5y ago
This is already possible with PhotoDNA. When has it happened?
- stickfigure 5y ago
  Your browser cache is not being synced to iCloud; Apple doesn't see it. So no, it is not currently possible.
  - endisneigh 5y ago
    I'm not talking about Apple, I'm saying in general technology has already been deployed to make what you're describing possible. So where's the evidence of abuse?
    - stickfigure 5y ago
      No vendors are snooping your phone/computer browser cache, so this attack vector does not yet exist. Apple is building it.
      - zimpenfish 5y ago
        > No vendors are snooping your phone/computer browser cache a) That you know of b) Apple won't be doing this either > Apple is building it. No, they aren't.
dylan604 5y ago

WesolyKubeczek 5y ago

You know, it's rather not okay to treat every smartphone user out there as a potential criminal just because they happen to have photos on their devices. At least in an alleged democracy where there's this presumption of innocence thing. Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone. But what can I say, there wasn't much uproar about piracy tax on blank CD-R(W) media back in the day, so why not have that now. And eventually we go peak USSR where everyone and their dog is suspect and whoever is arrested is the enemy of the people. Yay, it's somehow reassuring to know I won't live long enough to see it.

nonbirithm 5y ago
And it's fine to treat everyone as a potential criminal when we entrust our data on the same company's servers? No matter if on-device scanning makes surveillance easier than ever, the surveillance itself was still a significant possibility up to this point with server-side scanning. Imagine how many petabytes of user data already exist in the cloud.
- short_sells_poo 5y ago
  So in your mind because bad thing X is already happening, it's completely OK for bad thing XY to also start happening?
  - nonbirithm 5y ago
    No, it's that in spite of there already being an invasive scanning process in place for this long at every major tech company that handles user data, nobody seemed to care until now.
    - hda2 5y ago
      Most people weren't aware or couldn't see where this was going. Now that it's becoming obvious, people are starting to care.
    - 5y ago

Youden 5y ago

> It doesn’t matter if there are collisions if the two images don’t actually look the same. Is that really true? My understanding is that the manual reviewers at Apple only see some kind of low-resolution proxy, not the full-resolution image. I'd also be shocked if the human reviewers were shown the original, actually CP image, to compare to. Given that, it's not necessary to produce an actual visual match, it's just necessary to produce an image that when scaled-down looks like CSAM (e.g. take an actual photo of a kid at the beach and photoshop in some skin-coloured swimwear with creases in the right places). > Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? The attack I'd worry about here is similar to swatting. Someone who doesn't like me sends a bunch of images like the ones I described above (not just one), they end up synced to iCloud (because Apple wants _everything_ synced to iCloud) and Apple reports me to the authorities, who end up knocking at my door and arresting me. Even though I'm innocent, I'll probably have most of my computers confiscated for a while and spend a few days locked up. > PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. PhotoDNA's algorithms and hashes aren't public, so it's not clear how an attacker would exploit PhotoDNA in the way that people are afraid will be done for Apple. PhotoDNA also isn't, as far as I know, part of a product that aims to create unprotected backups of the phones of nearly a billion users. Apple really wants you to upload your whole phone to iCloud. The only comparable alternative is Google's Android backup but Google does the right thing and end-to-end encrypts that.

robertoandred 5y ago
Why would you save all of the almost-CSAM pics the attacker sends you to your photo library?
- Youden 5y ago
  Manually, perhaps because the attacker crafts the messages to make that desirable. But a bigger concern is automatically. WhatsApp for example can automatically save all received photos to your Camera Roll, which is of course automatically backed up to iCloud for many people. So an attacker could potentially just send you a bundle of 40 images of however many and your phone automatically sends it to Apple.

sennight 5y ago

We are talking about how everyone who gave Apple money now has a potential probable cause vector that they didn't before. Everyone running the software is a suspect by default. Ask black Americans how they feel about setting the bar low for probable cause. "Following the 2004 Madrid train bombings, fingerprints on a bag containing detonating devices were found by Spanish authorities. The Spanish National Police shared the fingerprints with the FBI through Interpol. Twenty possible matches for one of the fingerprints were found in the FBI database and one of the possible matches was Brandon Mayfield. His prints were in the FBI database as they were taken as part of standard procedure when he joined the military." "The FBI described the fingerprint match as '100% verified'." https://en.wikipedia.org/wiki/Brandon_Mayfield https://en.wikipedia.org/wiki/Brandon_Mayfield

dane-pgp 5y ago
> '100% verified' Just reading those words is rage-inducing, but I'm grateful to have learnt this example of government lying. I feel like it should become an expression that societies teach to their children to warn them about abuses of power. Other mottoes synonymous with government deception and corruption come to mind, but at the risk of being too controversial I will share only their initials and dates: "SAARH" (2013), "MA" (2003), "IANAC" (1973), "NHDAEMZE" (1961), "AMF" (1940).
- sennight 5y ago
  Because I delight in delivering bad news, I'll point out that the real takeaway shouldn't be that federal LEOs regularly lie (though, they do) - it is that they are permitted to lie convincingly through handwavy technical means. All these tools are designed to give them permission to totally destroy your life. I'm aware of only two geewiz CSI methods that are actually founded in science: cryptography (this neural crap doesn't qualify) and DNA. Unlike fingerprints and bitemark analysis, those two tools were invented outside of law enforcement - instead of being purpose built for prosecution. Anybody doubting that should look into the history of the polygraph and its continued use in the face of evidence demonstrating how useless it is in the pursuit of truth... which begs the question: if they aren't interested in the truth, what are they doing? https://en.wikipedia.org/wiki/Polygraph https://en.wikipedia.org/wiki/Polygraph

zug_zug 5y ago

And what if the FBI demands a list of all Apple users who have matched even 1 CSAM photo for their own private watchlist?

mrtranscendence 5y ago
And what if the FBI demands that Apple leadership genuflect toward an oil painting of J Edgar Hoover? The FBI can demand all sorts of things. In this case, Apple isn't tracking collisions as small as 1 photo.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
theluketaylor 5y ago
There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Apple is using secret splitting to make sure they cannot decode the CSAM ticket until the threshold is reached. Devices also produce some synthetic matches to prevent themselves Apple (or anyone else) inferring a pre-threshold count based on the number of vouchers. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... Threshold Secret Sharing Synthetic Match Vouchers
- zug_zug 5y ago
  Thanks. You're right. Very sophisticated system. Still find it morally repugnant, but technologically it is fairly thorough.
- heavyset_go 5y ago
  > There are a lot of really valid criticisms of Apple plan here, but Apple has gone out of their way to prevent that exact case. Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.

cwkoss 5y ago

> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? I believe a single image flag could be used as pretext to arrest someone that the government has already deemed an enemy of the state. Ex. Someone like Julian Assange

trhway 5y ago

> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? being on a government maintained secret list of CSAM flagged would probably have a lot of consequences in your life going forward without you knowing about that. It may be just one innocent image which accidentally matched the hash, yet without any procedure to get you off that list or even just to learn whether you're on that list ... >If some corrupt government wants to get you they don’t need this. it isn't corrupt government which is a threat here. It is a well intended well functioning relatively uncorrupt government which spends a lot of effort to protect its society from terrorism, child porn, sex trafficking, drugs, etc. In case of corrupt government the situation is kind of easier - you can always learn and correct necessary things through the corrupt government officials by paying a corresponding "fee".

Gibbon1 5y ago

Well there is this here https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_arrest https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_...

InTheArena 5y ago

A honest question I have is if PhotoDNA results where ever validated? Is there a chance that collisions have been occurring with no one looking to see if actual CSAM was in use?

nr2x 5y ago

"If some corrupt government wants to get you they don’t need this" -> if some corrupt government wants to *FIND YOU SHARING IDEAS THEY DON'T LIKE* they can *DEFINITELY* use this

yunesj 5y ago

You ask for examples a few times. Here’s one. Computers confiscated from a liberty activist and his radio station. No CP-related charges pressed in the 5 years since they confiscated his computers. I’m guessing most times this happens, the accused try to keep it on the DL. https://www.wmur.com/article/fbi-raids-radio-talk-show-host-s-home-in-keene-sunday-morning/5209568# https://www.wmur.com/article/fbi-raids-radio-talk-show-host-...

joe_the_user 5y ago

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? I understand that the system as stated today has multiple safeguards against such things happening but... Given the sum-total of bs I've seen happening in this country, yes, believe that kind of thing is quite possible.