10 ms·

CSAM Detection Technical Summary [pdf]

60 points by kristianpaul 5y ago

h2odragon 5y ago
There's gonna be loads of fun picking this apart. > Only another image that appears nearly identical can produce the same number; forexample, images that differ in size or transcoded quality will still have the same NeuralHash value cf all the "calculated collisions" arguments already made > Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the database of known CSAM hashes. cf all the questions about that database of "official Badness" > First, Apple receives the NeuralHashes corresponding to known CSAM from the above child-safety organizations. sharing that authority out to who exactly i didnt catch the names in this document... and also cf the already raised points about "won't $Nation want their own version of this with their own database of Bad". Nations or religions, could Muslim countries keep their "no images of the Prophet" rules more strictly using these tools? would they not try?
donmcronald 5y ago
> Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices. Is that a typo? What use is something that’s unreadable? I’m only half done, but it sounds like a bunch of hand wavy, AI, ML bullshit to me. How do you match similar images without reducing them down to something that’s less than the original? > Users can’t identify which images were flagged as CSAM by the system. > If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated. So the user can’t have information about anything, but they can appeal the decision if their account is locked? And locking the account might cause people to do all kinds of stupid things like resetting their device to get the account back or because they panicked. Then the only evidence is some opaque system that claims the user had illegal content. That’s scary. Many years ago I knew a person that took a perfectly good PC to the garbage dump because of a scareware warning saying it was involved in illegal activity. Poor guy wasn’t very computer savvy. This will open up some excellent phishing opportunities because of the seriousness/consequences of the type of accusation and the legitimacy of everyone knowing it’s something that’s done now. Edit: I finished reading it. If the photos are synced to iCloud I guess locking the account preserves the data. I still don’t like how cloak and dagger the whole thing is. No one’s allowed to know anything and all the average person is going to understand is that Apple’s system says this person is guilty, so they’re guilty. I think stuff like this would get a lot more scrutiny and less traction if it wasn’t “for the children.” What if there are bugs? No one is going to be allowed to audit the system or see how it works because of the sensitive nature of the content. It’s pretty scary IMO. > The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. How was that calculated? Prove it. > The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. So is that the same kind of crappy AI that (incorrectly) detects suspicious activity on email accounts?
- salawat 5y ago
  This is why AI is complete snake oil. They're trying to encode a recognizer, but they can't even define what it is they are actually feature extracting. Only that it's "typical ML techniques" that the rest of the industry doesn't want to call bullshit on because no one wants to give someone the idea that they shouldn't be allowed to use something if they can't explicitly articulate the function. There's some things where screw it, why not. This is not one of them.
- rantwasp 5y ago
  > So the user can’t have information about anything, but they can appeal the decision if their account is locked? And locking the account might cause people to do all kinds of stupid things like resetting their device to get the account back or because they panicked. Then the only evidence is some opaque system that claims the user had illegal content. That’s scary. At that point you have uploaded the pictures to icloud. The evidence is in icloud. You cannot destroy your device to get rid of this.
- omoikane 5y ago
  Having an appeals process doesn't inspire confidence since we rarely hear stories of those appeals processes working. What we usually hear instead are people being unsuccessful in their attempts to go through an appeals process, often because the system doesn't reveal any information on why they were flagged. The real remediation appears to try to gain some publicity in order to attract the attention of some real human to actually look at their case.
- ec109685 5y ago
  The low resolution version of the images are available (decryptable once the threshold has been met), so there’s evidence whether it is or not illegal material.
m3kw9 5y ago
A lot of that sounds like magic, most of us will take their word for it till it comes out and ppl like CitizenLabs takes a look
abstractbarista 5y ago
It is unacceptable for this process to execute on my devices. I am very glad I own zero Apple products, and this has solidified my resolve to never change that. I'll be watching for proliferation of this technique in other devices, and will promptly trash any that implement it. (I already avoid uploading unencrypted content to cloud services.) In this imperfect world, privacy keeps innocent people free. Long live AES-256.
- donohoe 5y ago
  Are you so sure this isn't already happening on other services you use?
  - abstractbarista 5y ago
    I believe it is happening, which is why I do not upload any unencrypted content to those services.
    - cucumb3rrelish 5y ago
      can you explain why? if you upload a file selectively, where is the issue if its hash is compared to a CSAM database?
      - osmarks 5y ago
        You can't check what's in the database. Someone could extend it to other things.
        cucumb3rrelish 5y ago
        Isn't this far fetched? There is few things I can think of that would get you into legal trouble but are morally acceptable. Like whistleblowing
        osmarks 5y ago
jamessb 5y ago
This page [1] provides some more information and links to PDFs of several other related documents, including "Apple PSI System — Security Protocol and Analysis" [2] (which has lots more technical details) and some Technical Assesments. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ [2]: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...
cm2187 5y ago
On the neural hash algorithm, I presume it must be some sort of a standard algorithm since the hashes will come from third party organisations. Is it available as a library anywhere?
- jeffbee 5y ago
  If the hashes are coming from NCMEC, presumably they must either be https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna or Apple have contributed a new scheme.
  - pornel 5y ago
    The PhotoDNA algorithm is kept as a secret, so whatever Apple is describing can't be it. I wonder if they've managed to get NCMEC to re-hash images for them using the new algorithm, or are they using PhotoDNA data as the input for their algorithm.
    - jeffbee 5y ago
      You're right, the details in this paper are not consistent with PhotoDNA. I wonder if these new hashes and the models to generate them are going to be made available to other anti-abuse organizations.
cft 5y ago
I was going to buy an M1 Macbook but I don't want it to execute any spyware processes. Is it likely that Macbooks will get this too?
- rantwasp 5y ago
  yes, this is coming to the mac also. surveillance across the board!
takenpilot 5y ago
Just a question: if they turn these images into hashes, does that mean it only finds very specific images? That is, if someone has some well-known image on their phone, then it can find it, but anything taken themselves is safe? Doesn't that make this a bit pointless since I don't know anyone that saves images from the internet to their phone. That's weird, right?
- takenpilot 5y ago
  And a sub-question: doesn't that make this ideal for checking for copyright infringement instead?
  - rantwasp 5y ago
    copyright enforcement, detecting terrorists, sussing out dissidents. The applications are endless!!! Welcome to the future.
- pornel 5y ago
  It finds specific images, but the hash is based on pixels, not raw file bytes. The hash is insensitive to small image changes (brightness, saturation, rotation, compression artifacts), so slightly modified images still hash to the same value.
- yosamino 5y ago
  > Doesn't that make this a bit pointless since I don't know anyone that [...] There is a very large variety of people out there. It's unlikely that your social bubble contains all the different ways to use a phone. if you search the net the phrase "phone clearing dump" you will find example of people having filled their phones with so much "funny images" that they are now saving them to an online image hosting service. You probably also don't know anyone who has depictions of sexual abuse of children on their phones, yet it is a thing that exists, and that apple seems to be trying to address. The world is big and, as you say, weird ineed.
- rantwasp 5y ago
  yes. it's specific images. It's pointless and looks half-assed, but hey, they have the future to perfect it. This is only the watered down/tame version to get the foot in the door and cause a big commotion while still being able to keep plausible deniability. Once they do this a couple of times people will get tires and stop caring. That's when the scary stuff slips in.
mrtksn 5y ago
The final step of this is going to be detection of terrorist materials, then we will have privacy that is based on our jurisdictions definition of terrorism. Your device provider(not necessarily Apple) will notify authorities when you score high enough on a threat model. In just in a decade, for the US citizens that model could include belief in fair elections(depending on who wins, you can be a terrorist of the deep state or the far right), you can be a Chinese terrorist in the USA if the friction intensifies between countries, you can be financial terrorist if the relationship between Cryptocurrencies and the state sours. You can be traitor terrorist of corporations, because they detected inappropriate use of corporate materials on your device. You can be cultural terrorist in a culture obsessed with self preservation. You can be infidel terrorist a muslim state with large enough market. There is no need for hypothetical situations, you can be a terrorist or a freedom fighter today depending on your coordinates. It is so easy to be a terrorist these days. This is going to get very interesting when every authority starts demanding detection of their boogyman. Why bother with anti-espionage when you can tell apple to check who does have the pictures of your new shiny thing that was supposed to be a secret? Apple seems to provide nice technological solution that looks promising to work fine as long as the authorities do good job in not becoming totalitarian crazy nutjobs because the technology can be used for any content, does not have a capability to ensure that it works only on child porn. Apple is giving governments a content control tool that at this time is promised to work only for specific materials.
- cwkoss 5y ago
  We should hope detection of terrorism is the final step. We should be extremely concerned that detection of political activism or investigative journalism will actually be the final step.
  - Prometheous08 5y ago
    'terrorism' is defined as whatever a governing body in charge says it is. https://www.reuters.com/article/us-usa-iran/u-s-officially-designates-irans-revolutionary-guards-a-terrorist-group-idUSKCN1RR1BE https://www.reuters.com/article/us-usa-iran/u-s-officially-d... https://www.aa.com.tr/en/middle-east/iran-parliament-blacklists-pentagon-as-terrorist-group/1694673 https://www.aa.com.tr/en/middle-east/iran-parliament-blackli...
aethertron 5y ago
Seems if you don't use iCloud, you can opt out. There are alternative backup services available on iOS devices, like Backblaze. But does the OS let them properly integrate with the system?
- rantwasp 5y ago
  to me it seems like if you don't use Apple you're totally protected. Hmm... What is the alternative again? Is there a phone that does straight linux and everything is open source?
  - Hackbraten 5y ago
    There is: https://puri.sm/products/librem-5/ https://puri.sm/products/librem-5/
jeffbee 5y ago
The bottom line here is Apple can look at your iCloud data whenever they want. They've written down a vague description of something that supposedly keeps them from doing it all the time, but they can unilaterally change the parameters whenever they choose to do so. They have thoroughly torpedoed their privacy story.
- r00fus 5y ago
  Assuming the user trusts Apple, that may be palatable to some. But the crux is, the images they're matching against are in a database maintained by an unaccountable nonprofit (NCMEC). Apple did this so their staff wouldn't have to see the offending images, until/unless it was actually confirmed. However, the matching is all algorithmic and who knows what's in the CSAM database. It could be images of police brutality so law enforcement can ferret out protesters, for all we know. Want to round up all communists? This is an easy way to do so.
  - jeffbee 5y ago
    Sure, the actual situation has always been that the iOS privacy story relies on you to trust Apple. But their marketing, which has always been bullshit, has implied that iOS privacy is a matter of the laws of physics.
gojomo 5y ago
Many of these steps are spun as being for the users' privacy. But they also prevent the operators of the system – Apple Inc, quasi-governmental non-profits (like NCMEC), & government law-enforcement agencies – from facing accountability for their choices. No one else can confirm what content they're scanning for, or what 'threshold' or other review applies, before serious enforcement action is taken against an account. Also, Apple is quite vague about whether the claim "Apple manually reviews all reports" means Apple employees can view all triggering content, even if it was never uploaded to Apple servers. (How would that work?) It would be trivial to extend this system to look for specific phrases or names in all local communication content. How will Apple say 'no' when China & Russia want that capability? In fact, might this initial domestic rollout simply be the 1st step of a crafty multi-year process to make that sort of eventual request seem ho-hum, "we're just extending American law-enforcement tech to follow local laws"?
- phkahler 5y ago
  >> It would be trivial to extend this system to look for specific phrases or names in all local communication content. How? They're comparing image hashes.
  - gojomo 5y ago
    Perceptual image hashes are a hard problem. Searching for subversive text strings is easy. And, a bunch of the obscuring-of-surveillance-intent achieved here is oblivious as to the types of detection-events - not at all image-specific. (That is, the private-set-intersection, cryptographic 'blinding', & threshold sharing schemes will work just fine for text/text-shingle/text-vector snitchlists as for image-perceptual-hash snitchlists. They're generic tricks for hiding what searches are being performed, & easy to repurpose.) Once Apple's shown a willingness to install iOS software that does a crime-grep on private data, & snitches on users without their assent, adding text-triggers is technically simple. A iOS point-release.
- sirn 5y ago
  > Also, Apple is quite vague about whether the claim "Apple manually reviews all reports" means Apple employees can view all triggering content, even if it was never uploaded to Apple servers. (How would that work?) From a quick read of the PDF, my impression is this Safety Voucher is more of an additional metadata generated before a photo is uploaded to iCloud Photo (as oppose to scan photo library to see if it matches and notify Apple narrative). The Safety Voucher is designed in a way that they could decide on the server-side if a photo is CSAM without requiring iCloud to have an ability to decrypt all photos. Safety Voucher comes in three layers: - The first layer is generating a NeuralHash for a photo and derive a cryptographic header from it based on blind hash provisioned by Apple from a known CSAM NeuralHash (e.g. Apple pre-compute this table for each device). First layer is decryptable if the generated NeuralHash is a known CSAM. (Is it going to be a random data if NeuralHash doesn't exists in the table?) - The second layer uses Secret Sharing[1] scheme (where a content is decryptable only if >n keys are present) using NeuralHash of a visual derivative as a secret share. Multiple visual derivative is generated for each photo, and certain amount of shares is required to decrypt the layer. - After the second layer is decrypted, the file is readable. One thing that doesn't makes sense is iCloud Photos are not currently End-to-End Encrypted, so Apple already have access to all photos. This scheme would only makes sense if they plan to introduce End-to-End Encryption for iCloud Photos (& iCloud Backups?) in the future. In my opinion, if they were to enable E2EE for photos, this scheme would be slightly better for privacy than Apple having an ability to scan all photos on the cloud (e.g. they can't retroactively decrypt a photo that already has Safety Voucher uploaded to iCloud if the matching NeuralHash doesn't exists at the time of blind hash table generation), but significantly less than having E2EE and not having this system in the first place. (As someone from a country that is known to abuse this sort of power, I'm not happy about this system existing.) [1]: https://en.wikipedia.org/wiki/Secret_sharing https://en.wikipedia.org/wiki/Secret_sharing
k1rcher 5y ago
For reasons unbeknownst to me at the time of writing (for it requires much more introspection), I think this would be OK if it were a service that ran only on icloud-stored photos— rather than apple devices across the board. I’m not sure why this is the line crossed for me. Perhaps it is only because I willfully choose _not_ to store _any_ data on iCloud (or at the very least a minimum amount as possible). Even moreso, I’m a bit unsure of what that line even is. Is it a software vs hardware thing? No, because one could argue the service is not Apple hardware-specific, but rather specific to the iOS software. So perhaps the line crossed is this being an OS feature rather than a SAAS feature. It is unfortunate that the precedent has been set where I am OK with potentially privacy invasive features on SAAS products, but I suppose it is what it is.
- rantwasp 5y ago
  the line is crossed because if you store data in the cloud you kind of get that the provider might be force to give LE to it. It may or may not happen but if it's in the cloud technically it's out of your control. Stuff running on your device OTOH, you have the illusion that you control it. It's YOUR physical device and technically you should be able to do whatever you want with it. Now, enter closed source software + walled gardens => you don't really control your device. You don't get a say on what's going on in it. Up until know we believed Apple would do the right thing. Well... Myth busted I guess.
  - simondotau 5y ago
    Turn off iCloud Photos and it won't run the surveillance. Seems reasonable that if you want to store your photos on someone else's hardware, they should get some say whether their hardware is used to store child porn.
    - rantwasp 5y ago
      seems reasonable? here’s the thing: you start with a clear cut case like cp and after that you add different things that, as a government, bother you. you’re in a totalitarian state faster than you can blink. The turn off icloud argument also does not work. Who tells you they’re not going to keep pulling stunts like this? So the solution is: stop using Apple. Now what?
deleted 5y ago
[deleted]
rantwasp 5y ago
The Big Brain Scheme Apple uses for their CSAM stuff. Summary for the lazy: - this is going to be out in ios15 - Apple will build and push a database of hashes with every update - these hashes will be used to verify against the pics before they are uploaded to icloud and generate a "voucher" - once a certain threshold of vouchers is passed, Apple will be able to decrypt the pictures in questions (all this discussion seems to assume Apple does not have access to icloud pictures - so maybe they are preparing to rollout full encrypted backups???) - the crypto stuff is interesting - the scary part is that there is nothing preventing Apple to do the same thing with other thing on your phone (assuming there is a way to generate a good enough matching hash - I can see how this could be easily extended to basically anything that's stored on the phone)
- simondotau 5y ago
  Your last point is a slippery slope that began the moment you allowed any software on any device to update itself. Literally any software that updates itself could be converted into a treacherous surveillance system at any time. And literally any cloud storage service could betray you at any time without pushing new binaries to your hardware. And nobody ever has to tell you shit.
  - rantwasp 5y ago
    for all that i know the software doesn’t even need to update itself. it may have the backdoor built in. so now what? go back to landlines? stop using a “smart”phone?
nsxwolf 5y ago
So are people really going to go to prison because of a flag that says they did something illegal? With no opportunity for discovery or to cross examine? No way to audit the software and the chain of events that led to the detection?
- ubermonkey 5y ago
  That would indeed be bad. What Apple is doing here is also bad. But jail-by-algorithm is not part of what Apple is doing.
  - r00fus 5y ago
    No it's just "introduction to law enforcement by algorithm" which can be more fatal than jail.
cwizou 5y ago
As a reminder for context, Apple already did do a server side check for iCloud photos [1]. I think that's missing from the debate, what's changing is moving the check to on device which many object to (and I and others have written enough about that). To add to this though : I think they built an extremely complex - and hopefully perfectly robust - system to make the check on device. On top of the other issues, it's massively important said process is not exploitable. The content of the NCMEC database is not public, one can argue for good reason, and if that effort to move on device lead to the database being leaked (and thus bad actors being able to "mark" content that they shouldn't share), that would be incredibly counter productive. Maybe only knowing that the database was updated is too much information to leak about this ? I think that's an additional concern to the ones already expressed many times, and I sincerly wish Apple had kept the checks server side even more now, as I only see downside to not doing so. [1] : https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
applthrowaway86 5y ago
This is a consequence of the composition of Apple's security engineering organization. If you look through the LinkedIn profiles of the people in the org that delivered this feature, an interesting pattern emerges: a significant number of them are ex-IC. Back in the golden age of early iPhone development, the org used to have almost full autonomy and was run by a bunch of Mac fanatics and UNIX nerds. Today the org is full of people who were spies in their previous job. It shows.
tibbydudeza 5y ago
I am in favor of them checking messaging content for general nudity for minors but I seriously have my doubts about this CSAM system. Your life will be destroyed by a false flag.
- ec109685 5y ago
  The flag includes an uploaded copy of the image, so that too will be verified before your life is destroyed.
aaomidi 5y ago
If you're an employee at Apple how do you even raise your voice of concern about this? They're already calling people against this a screeching minority. Apple has created a toxic work environment and silences voices of concern by applying labels.
noduerme 5y ago
From the document >> The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. Descriptors are considered to be close to one another if the cosine of the angle between descriptors is close to 1. Can someone explain what this means? Am I to understand that the algorithm plots image descriptors or hashes of them along a 2-dimensional circle, and somehow iterates until the way it places them on that circle results in similar images being closer to each other?
- ec109685 5y ago
  It’s not a two dimensional plot. This provides some background (though not sure if this is their scheme): https://towardsdatascience.com/importance-of-distance-metrics-in-machine-learning-modelling-e51395ffe60d?gi=d1c966d4ca2 https://towardsdatascience.com/importance-of-distance-metric...

60 points by kristianpaul 5y ago

There's gonna be loads of fun picking this apart. > Only another image that appears nearly identical can produce the same number; forexample, images that differ in size or transcoded quality will still have the same NeuralHash value cf all the "calculated collisions" arguments already made > Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the database of known CSAM hashes. cf all the questions about that database of "official Badness" > First, Apple receives the NeuralHashes corresponding to known CSAM from the above child-safety organizations. sharing that authority out to who exactly i didnt catch the names in this document... and also cf the already raised points about "won't $Nation want their own version of this with their own database of Bad". Nations or religions, could Muslim countries keep their "no images of the Prophet" rules more strictly using these tools? would they not try?

donmcronald 5y ago

> Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices. Is that a typo? What use is something that’s unreadable? I’m only half done, but it sounds like a bunch of hand wavy, AI, ML bullshit to me. How do you match similar images without reducing them down to something that’s less than the original? > Users can’t identify which images were flagged as CSAM by the system. > If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated. So the user can’t have information about anything, but they can appeal the decision if their account is locked? And locking the account might cause people to do all kinds of stupid things like resetting their device to get the account back or because they panicked. Then the only evidence is some opaque system that claims the user had illegal content. That’s scary. Many years ago I knew a person that took a perfectly good PC to the garbage dump because of a scareware warning saying it was involved in illegal activity. Poor guy wasn’t very computer savvy. This will open up some excellent phishing opportunities because of the seriousness/consequences of the type of accusation and the legitimacy of everyone knowing it’s something that’s done now. Edit: I finished reading it. If the photos are synced to iCloud I guess locking the account preserves the data. I still don’t like how cloak and dagger the whole thing is. No one’s allowed to know anything and all the average person is going to understand is that Apple’s system says this person is guilty, so they’re guilty. I think stuff like this would get a lot more scrutiny and less traction if it wasn’t “for the children.” What if there are bugs? No one is going to be allowed to audit the system or see how it works because of the sensitive nature of the content. It’s pretty scary IMO. > The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. How was that calculated? Prove it. > The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. So is that the same kind of crappy AI that (incorrectly) detects suspicious activity on email accounts?

salawat 5y ago
This is why AI is complete snake oil. They're trying to encode a recognizer, but they can't even define what it is they are actually feature extracting. Only that it's "typical ML techniques" that the rest of the industry doesn't want to call bullshit on because no one wants to give someone the idea that they shouldn't be allowed to use something if they can't explicitly articulate the function. There's some things where screw it, why not. This is not one of them.
rantwasp 5y ago
> So the user can’t have information about anything, but they can appeal the decision if their account is locked? And locking the account might cause people to do all kinds of stupid things like resetting their device to get the account back or because they panicked. Then the only evidence is some opaque system that claims the user had illegal content. That’s scary. At that point you have uploaded the pictures to icloud. The evidence is in icloud. You cannot destroy your device to get rid of this.
omoikane 5y ago
Having an appeals process doesn't inspire confidence since we rarely hear stories of those appeals processes working. What we usually hear instead are people being unsuccessful in their attempts to go through an appeals process, often because the system doesn't reveal any information on why they were flagged. The real remediation appears to try to gain some publicity in order to attract the attention of some real human to actually look at their case.
ec109685 5y ago
The low resolution version of the images are available (decryptable once the threshold has been met), so there’s evidence whether it is or not illegal material.

m3kw9 5y ago

A lot of that sounds like magic, most of us will take their word for it till it comes out and ppl like CitizenLabs takes a look

abstractbarista 5y ago

It is unacceptable for this process to execute on my devices. I am very glad I own zero Apple products, and this has solidified my resolve to never change that. I'll be watching for proliferation of this technique in other devices, and will promptly trash any that implement it. (I already avoid uploading unencrypted content to cloud services.) In this imperfect world, privacy keeps innocent people free. Long live AES-256.

donohoe 5y ago
Are you so sure this isn't already happening on other services you use?
- abstractbarista 5y ago
  I believe it is happening, which is why I do not upload any unencrypted content to those services.
  - cucumb3rrelish 5y ago
    can you explain why? if you upload a file selectively, where is the issue if its hash is compared to a CSAM database?
    - osmarks 5y ago
      You can't check what's in the database. Someone could extend it to other things.
      - cucumb3rrelish 5y ago
        Isn't this far fetched? There is few things I can think of that would get you into legal trouble but are morally acceptable. Like whistleblowing
        osmarks 5y ago

jamessb 5y ago

This page [1] provides some more information and links to PDFs of several other related documents, including "Apple PSI System — Security Protocol and Analysis" [2] (which has lots more technical details) and some Technical Assesments. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ [2]: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...

cm2187 5y ago

On the neural hash algorithm, I presume it must be some sort of a standard algorithm since the hashes will come from third party organisations. Is it available as a library anywhere?

jeffbee 5y ago
If the hashes are coming from NCMEC, presumably they must either be https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna or Apple have contributed a new scheme.
- pornel 5y ago
  The PhotoDNA algorithm is kept as a secret, so whatever Apple is describing can't be it. I wonder if they've managed to get NCMEC to re-hash images for them using the new algorithm, or are they using PhotoDNA data as the input for their algorithm.
  - jeffbee 5y ago
    You're right, the details in this paper are not consistent with PhotoDNA. I wonder if these new hashes and the models to generate them are going to be made available to other anti-abuse organizations.

cft 5y ago

I was going to buy an M1 Macbook but I don't want it to execute any spyware processes. Is it likely that Macbooks will get this too?

rantwasp 5y ago
yes, this is coming to the mac also. surveillance across the board!

takenpilot 5y ago

Just a question: if they turn these images into hashes, does that mean it only finds very specific images? That is, if someone has some well-known image on their phone, then it can find it, but anything taken themselves is safe? Doesn't that make this a bit pointless since I don't know anyone that saves images from the internet to their phone. That's weird, right?

takenpilot 5y ago
And a sub-question: doesn't that make this ideal for checking for copyright infringement instead?
- rantwasp 5y ago
  copyright enforcement, detecting terrorists, sussing out dissidents. The applications are endless!!! Welcome to the future.
pornel 5y ago
It finds specific images, but the hash is based on pixels, not raw file bytes. The hash is insensitive to small image changes (brightness, saturation, rotation, compression artifacts), so slightly modified images still hash to the same value.
yosamino 5y ago
> Doesn't that make this a bit pointless since I don't know anyone that [...] There is a very large variety of people out there. It's unlikely that your social bubble contains all the different ways to use a phone. if you search the net the phrase "phone clearing dump" you will find example of people having filled their phones with so much "funny images" that they are now saving them to an online image hosting service. You probably also don't know anyone who has depictions of sexual abuse of children on their phones, yet it is a thing that exists, and that apple seems to be trying to address. The world is big and, as you say, weird ineed.
rantwasp 5y ago
yes. it's specific images. It's pointless and looks half-assed, but hey, they have the future to perfect it. This is only the watered down/tame version to get the foot in the door and cause a big commotion while still being able to keep plausible deniability. Once they do this a couple of times people will get tires and stop caring. That's when the scary stuff slips in.

mrtksn 5y ago

The final step of this is going to be detection of terrorist materials, then we will have privacy that is based on our jurisdictions definition of terrorism. Your device provider(not necessarily Apple) will notify authorities when you score high enough on a threat model. In just in a decade, for the US citizens that model could include belief in fair elections(depending on who wins, you can be a terrorist of the deep state or the far right), you can be a Chinese terrorist in the USA if the friction intensifies between countries, you can be financial terrorist if the relationship between Cryptocurrencies and the state sours. You can be traitor terrorist of corporations, because they detected inappropriate use of corporate materials on your device. You can be cultural terrorist in a culture obsessed with self preservation. You can be infidel terrorist a muslim state with large enough market. There is no need for hypothetical situations, you can be a terrorist or a freedom fighter today depending on your coordinates. It is so easy to be a terrorist these days. This is going to get very interesting when every authority starts demanding detection of their boogyman. Why bother with anti-espionage when you can tell apple to check who does have the pictures of your new shiny thing that was supposed to be a secret? Apple seems to provide nice technological solution that looks promising to work fine as long as the authorities do good job in not becoming totalitarian crazy nutjobs because the technology can be used for any content, does not have a capability to ensure that it works only on child porn. Apple is giving governments a content control tool that at this time is promised to work only for specific materials.

cwkoss 5y ago
We should hope detection of terrorism is the final step. We should be extremely concerned that detection of political activism or investigative journalism will actually be the final step.
- Prometheous08 5y ago
  'terrorism' is defined as whatever a governing body in charge says it is. https://www.reuters.com/article/us-usa-iran/u-s-officially-designates-irans-revolutionary-guards-a-terrorist-group-idUSKCN1RR1BE https://www.reuters.com/article/us-usa-iran/u-s-officially-d... https://www.aa.com.tr/en/middle-east/iran-parliament-blacklists-pentagon-as-terrorist-group/1694673 https://www.aa.com.tr/en/middle-east/iran-parliament-blackli...

aethertron 5y ago

Seems if you don't use iCloud, you can opt out. There are alternative backup services available on iOS devices, like Backblaze. But does the OS let them properly integrate with the system?

rantwasp 5y ago
to me it seems like if you don't use Apple you're totally protected. Hmm... What is the alternative again? Is there a phone that does straight linux and everything is open source?
- Hackbraten 5y ago
  There is: https://puri.sm/products/librem-5/ https://puri.sm/products/librem-5/

jeffbee 5y ago

The bottom line here is Apple can look at your iCloud data whenever they want. They've written down a vague description of something that supposedly keeps them from doing it all the time, but they can unilaterally change the parameters whenever they choose to do so. They have thoroughly torpedoed their privacy story.

r00fus 5y ago
Assuming the user trusts Apple, that may be palatable to some. But the crux is, the images they're matching against are in a database maintained by an unaccountable nonprofit (NCMEC). Apple did this so their staff wouldn't have to see the offending images, until/unless it was actually confirmed. However, the matching is all algorithmic and who knows what's in the CSAM database. It could be images of police brutality so law enforcement can ferret out protesters, for all we know. Want to round up all communists? This is an easy way to do so.
- jeffbee 5y ago
  Sure, the actual situation has always been that the iOS privacy story relies on you to trust Apple. But their marketing, which has always been bullshit, has implied that iOS privacy is a matter of the laws of physics.

gojomo 5y ago

Many of these steps are spun as being for the users' privacy. But they also prevent the operators of the system – Apple Inc, quasi-governmental non-profits (like NCMEC), & government law-enforcement agencies – from facing accountability for their choices. No one else can confirm what content they're scanning for, or what 'threshold' or other review applies, before serious enforcement action is taken against an account. Also, Apple is quite vague about whether the claim "Apple manually reviews all reports" means Apple employees can view all triggering content, even if it was never uploaded to Apple servers. (How would that work?) It would be trivial to extend this system to look for specific phrases or names in all local communication content. How will Apple say 'no' when China & Russia want that capability? In fact, might this initial domestic rollout simply be the 1st step of a crafty multi-year process to make that sort of eventual request seem ho-hum, "we're just extending American law-enforcement tech to follow local laws"?

phkahler 5y ago
>> It would be trivial to extend this system to look for specific phrases or names in all local communication content. How? They're comparing image hashes.
- gojomo 5y ago
  Perceptual image hashes are a hard problem. Searching for subversive text strings is easy. And, a bunch of the obscuring-of-surveillance-intent achieved here is oblivious as to the types of detection-events - not at all image-specific. (That is, the private-set-intersection, cryptographic 'blinding', & threshold sharing schemes will work just fine for text/text-shingle/text-vector snitchlists as for image-perceptual-hash snitchlists. They're generic tricks for hiding what searches are being performed, & easy to repurpose.) Once Apple's shown a willingness to install iOS software that does a crime-grep on private data, & snitches on users without their assent, adding text-triggers is technically simple. A iOS point-release.
sirn 5y ago
> Also, Apple is quite vague about whether the claim "Apple manually reviews all reports" means Apple employees can view all triggering content, even if it was never uploaded to Apple servers. (How would that work?) From a quick read of the PDF, my impression is this Safety Voucher is more of an additional metadata generated before a photo is uploaded to iCloud Photo (as oppose to scan photo library to see if it matches and notify Apple narrative). The Safety Voucher is designed in a way that they could decide on the server-side if a photo is CSAM without requiring iCloud to have an ability to decrypt all photos. Safety Voucher comes in three layers: - The first layer is generating a NeuralHash for a photo and derive a cryptographic header from it based on blind hash provisioned by Apple from a known CSAM NeuralHash (e.g. Apple pre-compute this table for each device). First layer is decryptable if the generated NeuralHash is a known CSAM. (Is it going to be a random data if NeuralHash doesn't exists in the table?) - The second layer uses Secret Sharing[1] scheme (where a content is decryptable only if >n keys are present) using NeuralHash of a visual derivative as a secret share. Multiple visual derivative is generated for each photo, and certain amount of shares is required to decrypt the layer. - After the second layer is decrypted, the file is readable. One thing that doesn't makes sense is iCloud Photos are not currently End-to-End Encrypted, so Apple already have access to all photos. This scheme would only makes sense if they plan to introduce End-to-End Encryption for iCloud Photos (& iCloud Backups?) in the future. In my opinion, if they were to enable E2EE for photos, this scheme would be slightly better for privacy than Apple having an ability to scan all photos on the cloud (e.g. they can't retroactively decrypt a photo that already has Safety Voucher uploaded to iCloud if the matching NeuralHash doesn't exists at the time of blind hash table generation), but significantly less than having E2EE and not having this system in the first place. (As someone from a country that is known to abuse this sort of power, I'm not happy about this system existing.) [1]: https://en.wikipedia.org/wiki/Secret_sharing https://en.wikipedia.org/wiki/Secret_sharing

k1rcher 5y ago

For reasons unbeknownst to me at the time of writing (for it requires much more introspection), I think this would be OK if it were a service that ran only on icloud-stored photos— rather than apple devices across the board. I’m not sure why this is the line crossed for me. Perhaps it is only because I willfully choose _not_ to store _any_ data on iCloud (or at the very least a minimum amount as possible). Even moreso, I’m a bit unsure of what that line even is. Is it a software vs hardware thing? No, because one could argue the service is not Apple hardware-specific, but rather specific to the iOS software. So perhaps the line crossed is this being an OS feature rather than a SAAS feature. It is unfortunate that the precedent has been set where I am OK with potentially privacy invasive features on SAAS products, but I suppose it is what it is.

rantwasp 5y ago
the line is crossed because if you store data in the cloud you kind of get that the provider might be force to give LE to it. It may or may not happen but if it's in the cloud technically it's out of your control. Stuff running on your device OTOH, you have the illusion that you control it. It's YOUR physical device and technically you should be able to do whatever you want with it. Now, enter closed source software + walled gardens => you don't really control your device. You don't get a say on what's going on in it. Up until know we believed Apple would do the right thing. Well... Myth busted I guess.
- simondotau 5y ago
  Turn off iCloud Photos and it won't run the surveillance. Seems reasonable that if you want to store your photos on someone else's hardware, they should get some say whether their hardware is used to store child porn.
  - rantwasp 5y ago
    seems reasonable? here’s the thing: you start with a clear cut case like cp and after that you add different things that, as a government, bother you. you’re in a totalitarian state faster than you can blink. The turn off icloud argument also does not work. Who tells you they’re not going to keep pulling stunts like this? So the solution is: stop using Apple. Now what?

deleted 5y ago

[deleted]

rantwasp 5y ago

The Big Brain Scheme Apple uses for their CSAM stuff. Summary for the lazy: - this is going to be out in ios15 - Apple will build and push a database of hashes with every update - these hashes will be used to verify against the pics before they are uploaded to icloud and generate a "voucher" - once a certain threshold of vouchers is passed, Apple will be able to decrypt the pictures in questions (all this discussion seems to assume Apple does not have access to icloud pictures - so maybe they are preparing to rollout full encrypted backups???) - the crypto stuff is interesting - the scary part is that there is nothing preventing Apple to do the same thing with other thing on your phone (assuming there is a way to generate a good enough matching hash - I can see how this could be easily extended to basically anything that's stored on the phone)

simondotau 5y ago
Your last point is a slippery slope that began the moment you allowed any software on any device to update itself. Literally any software that updates itself could be converted into a treacherous surveillance system at any time. And literally any cloud storage service could betray you at any time without pushing new binaries to your hardware. And nobody ever has to tell you shit.
- rantwasp 5y ago
  for all that i know the software doesn’t even need to update itself. it may have the backdoor built in. so now what? go back to landlines? stop using a “smart”phone?

nsxwolf 5y ago

So are people really going to go to prison because of a flag that says they did something illegal? With no opportunity for discovery or to cross examine? No way to audit the software and the chain of events that led to the detection?

ubermonkey 5y ago
That would indeed be bad. What Apple is doing here is also bad. But jail-by-algorithm is not part of what Apple is doing.
- r00fus 5y ago
  No it's just "introduction to law enforcement by algorithm" which can be more fatal than jail.

cwizou 5y ago

As a reminder for context, Apple already did do a server side check for iCloud photos [1]. I think that's missing from the debate, what's changing is moving the check to on device which many object to (and I and others have written enough about that). To add to this though : I think they built an extremely complex - and hopefully perfectly robust - system to make the check on device. On top of the other issues, it's massively important said process is not exploitable. The content of the NCMEC database is not public, one can argue for good reason, and if that effort to move on device lead to the database being leaked (and thus bad actors being able to "mark" content that they shouldn't share), that would be incredibly counter productive. Maybe only knowing that the database was updated is too much information to leak about this ? I think that's an additional concern to the ones already expressed many times, and I sincerly wish Apple had kept the checks server side even more now, as I only see downside to not doing so. [1] : https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...

applthrowaway86 5y ago

This is a consequence of the composition of Apple's security engineering organization. If you look through the LinkedIn profiles of the people in the org that delivered this feature, an interesting pattern emerges: a significant number of them are ex-IC. Back in the golden age of early iPhone development, the org used to have almost full autonomy and was run by a bunch of Mac fanatics and UNIX nerds. Today the org is full of people who were spies in their previous job. It shows.

tibbydudeza 5y ago

I am in favor of them checking messaging content for general nudity for minors but I seriously have my doubts about this CSAM system. Your life will be destroyed by a false flag.

ec109685 5y ago
The flag includes an uploaded copy of the image, so that too will be verified before your life is destroyed.

aaomidi 5y ago

If you're an employee at Apple how do you even raise your voice of concern about this? They're already calling people against this a screeching minority. Apple has created a toxic work environment and silences voices of concern by applying labels.

noduerme 5y ago

From the document >> The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. Descriptors are considered to be close to one another if the cosine of the angle between descriptors is close to 1. Can someone explain what this means? Am I to understand that the algorithm plots image descriptors or hashes of them along a 2-dimensional circle, and somehow iterates until the way it places them on that circle results in similar images being closer to each other?

ec109685 5y ago
It’s not a two dimensional plot. This provides some background (though not sure if this is their scheme): https://towardsdatascience.com/importance-of-distance-metrics-in-machine-learning-modelling-e51395ffe60d?gi=d1c966d4ca2 https://towardsdatascience.com/importance-of-distance-metric...