20 ms·

Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’

361 points by nickthegreek 5y ago

tlogan 5y ago
I have to say that I’m just sad about this. I was really convinced that Apple really believed in privacy and they will never do this (having a framework which can scan your device content). But maybe they already had it and this was like: “there is this delta and Afghanistan let’s make announcement now” We are just one stupid terrorist attack from full surveillance of everybody.
kevin_thibedeau 5y ago
How long until thishashcollisionisnotporn.com is a thing?
- mono-bob 5y ago
  I can confirm it was a fun weekend project and https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ has now become reality!
- istingray 5y ago
  sounds like a future NFT market
  - eurasiantiger 5y ago
    Sounds like hipster T-shirts with hash collision images.
    - mono-bob 5y ago
      I can confirm it was a fun weekend project and https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ has now become reality! I included a button to buy a hipster T-shirt ;)
- mono-bob 5y ago
  Sounds like a fun weekend project.
yeldarb 5y ago
I created a proof of concept showing how OpenAI's CLIP model can function as a "sanity check" similar to how Apple says their server-side model works. In order for a collision to get through to the human checkers, the same image would have to fool both networks independently: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/
- mrits 5y ago
  Cool project. Wouldn't padding all generated images with a few items that match CLIP get around this though?
  - yeldarb 5y ago
    Doing that would mean the NeuralHash would change though. And you'd have to not only get CLIP to identify CSAM in the generated image but also negate the parts of the generated image that are causing CLIP to label it as "generated" (while still colliding with the target NeuralHash). Unclear how hard this would actually be in practice (if I were going to attempt it, the first thing I'd try is to evolve a colliding image with something like CLIP+VQGAN) but certainly harder than finding a collision alone.
    - salawat 5y ago
      Take 2 CSAM pictures, known. Combine into one image. Swing and a miss. Not in the CSAM dataset. Take two images. Encode alternating pixels. Decode to get the original image back. Convert to different encodings print to PDF or Postscript. Encode as base64 representations of the image file... Who are we trying to fool here? This is kiddie stuff. This is more about trying to implant scanning capabilities on client devices.
      - yeldarb 5y ago
        Those are examples of triggering false negatives (CSAM that gets missed by the system), this collision attack is about triggering false positives (non-CSAM that triggers human intervention).
epistasis 5y ago
What's shocking to me is how little Apple management understood of what their actions looked like. Really stunning. For a company that marketed itself as one of the few digital service providers that consumers could trust, I just don't understand how they acted this way at all. Either there will be heads rolling at management, or Apple takes a permanent hit to consumer trust.
- FabHK 5y ago
  > What's shocking to me is how little Apple management understood of what their actions looked like. Really stunning. Maybe because they underestimated people's ignorance. I think they saw (and still do see) it as a better, more privacy preserving technique than what everyone else is doing.
  - fraa-orolo 5y ago
    The thing is that this is a better and more privacy preserving technique. Their hubris is in not seeing or thinking that they will be able to stand up to all kinds of abuses of this system that its mere existence will invite; their hubris is also in thinking that they will be able to perfectly and without mistakes manage and overview a system making accusations so heinous that even the mere act of accusing destroy people's lives and livelihoods.
    - shuckles 5y ago
      What abuses apply to CSAM scanning in this hybrid pipeline which don’t apply to iCloud Backup? If they scanned on the server, why couldn’t governments “slippery slope” abuse the system by requiring that all files on iOS end up in iCloud Backup, where they can be scanned by the system? Since the announcement, I can think of a dozen ways Apple could be easily forced into scanning all the contents of your device by assembling features they’ve already shipped. Yet they haven’t. At some point, people need to produce evidence that Apple cannot hold the line they’ve said they will.
      - candiodari 5y ago
gjsman-1000 5y ago
For everyone upset about Apple's CSAM scanning, I think we all forgot about the EARN IT Act. It was nearly passed last year but Congress was finished before it could be voted on. It had Bipartisan support and would've virtually banned E2E of any kind. And it would have required scanning everywhere according to the recommendations of a 19-member board of NGOs and unelected experts. The reason for this mandatory backdoor was child safety... even though AG Barr admitted that reducing the use of encryption had useful side effects. If you are Apple, even though EARN IT failed... you know where Washington's heart lies. Is CSAM scanning a "better alternative", a concession, an appeasement, a lesser evil, in the hope this prevents EARN IT from coming back? Also, many people forgot about the Lawful Access to Encrypted Data Act of 2020, or LAED, which would unilaterally banned E2E encryption in entirety and required that all devices featuring encryption must be unlockable by the manufacturer. That also was on the table.
- samename 5y ago
  Why can’t we have the alternative we already have, where we don’t have our phones scans and E2EE exists without compromise? Why do we have to accept it any other way? If you’re trying to frame this as “we need to prevent Congress from ever passing something like the EARN IT act”, I agree. Apple and other tech companies already lobby Congress. Why aren’t they lobbying for encryption?
  - gjsman-1000 5y ago
    They did, they lobbied heavily against EARN IT, and so did groups like the ACLU and EFF. However, it didn't stop Congress members from moving it along through the process - they were only stopped because Congress just ran out of time. It was clear that Congress was not interested in listening at all to the lobbyists on the issue. It's clear that EARN IT could literally be revived any day if Apple didn't do something to say "we don't need it because we've already satisfied your requirements."
    - orangecat 5y ago
      It's clear that EARN IT could literally be revived any day if Apple didn't do something to say "we don't need it because we've already satisfied your requirements." Alternatively, "Apple has shown that it's possible to do without undue hardship, so we should make everyone else do it too".
geoah 5y ago
> The system relies on a database of hashes—cryptographic representations of images—of known CSAM photos provided by National Center for Missing & Exploited Children (NCMEC) and other child protection organizations. “Cryptographic representations of images”. That’s not the case though right? These are “neuralhashes” afaik which are nowhere close to cryptographic hashes but rather locality sensitive hashes which is a fancy speak for “the closer two images look like, the more similar the hash”. Vice and others keeps calling the cryptographic. Am I missing something here?
- andrewmcwatters 5y ago
  Edit: "The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, and images that are different from one another result in different hashes."[1] Apple isn't using a "similar image, similar hash" system. They're using a "similar image, same hash" system. [1]: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - heavyset_go 5y ago
    > There really is no sound concept of "the more similar the hash." Perceptual hashes are not cryptographic hashes. Perceptual hashing systems do compare hashes using a distance metric like the Hamming distance. If two images have similar hashes, then they look kind of similar to one another. That's the point of perceptual hashing.
    - salawat 5y ago
      Except when you're not measuring the similarity you think you are, because NeuralNets are f&#@ing weird. You know, let me put it this way. Yiu know that one really weird family member I'm pretty sure everyone either has or is? Guess what? They're a neural net too. This is what Apple is asking you to trust.
      - heavyset_go 5y ago
dang 5y ago
Ongoing related threads: Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068 https://news.ycombinator.com/item?id=28219068 - Aug 2021 (542 comments) Convert Apple NeuralHash model for CSAM Detection to ONNX - https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391 - Aug 2021 (155 comments)
jl6 5y ago
> Anti-Child Abuse Imagery Tech Room for improvement in the headline.
firebaze 5y ago
I think we're up for a new Apple CEO in a few days.
- csilverman 5y ago
  The turmoil of Apple abruptly booting the CEO who's overseen the most obscenely prosperous stretch in its history would cost the company vastly more than the current controversy about CSAM. I'm not saying this as a fan of either Cook or their anti-CSAM measures; I'm neither, and if anyone is ever wrongfully arrested because Apple's system made a mistake, Cook may well wind up in disgrace depending on how much blame he can/can't shuffle off to subordinates. I don't think we're there yet, though.
  - atonse 5y ago
    He's not Ballmer but Ballmer also presided over record profits even though MS barely did anything remotely interesting under his watch. Just saying that money hides problems.
mzs 5y ago
>… Apple also said that after a user passes the 30 match threshold, a second non-public algorithm that runs on Apple's servers will check the results. >"This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database," …
Barrin92 5y ago
it's kind of silly how this is treated like some deeply technical issue. Apple's 'one of a trillion' claim is either true and the software is useless, because I'm pretty sure pedophiles can figure out what a watermark or a gaussian blur in paint net is, or it's imprecise and actually detects things which is the very thing that makes it dangerous to the public. It's a direct trade-off and the error tolerance of any such filter is the only thing that makes it useful so we can basically stop arguing about the depths of implementation details or how high the collision rate of the hashing algorithm is etc. If this thing is supposed to catch anyone it needs to be magnitudes more lenient than any of those minor faults.
- 734129837261 5y ago
  Exactly right. The tech Apple uses can be one of two things: 1. It requires a perfect 1:1 match (their documentation says this is not the case); 2. Or it has some freedom in detecting a match, probably including a match with a certain percentage. If it's the former, it's completely useless. A watermark or a randomly chosen pixel with a slightly different hue and the hash would be completely different. So, it's not #1. It's going to be #2. And that's where it becomes dangerous. The government of the USA is going to look for child predators. The government of Saudi Arabia is going to track down known memes shared by atheists, and they will be put to death; heresy is a capital offence over there. And China will probably do their best to track down Uyghurs so they can make the process of elimination even easier. It's not like Apple hasn't given in to dictatorships in the past. This tech is absolutely going to kill people.
  - endisneigh 5y ago
    What? Why would memes be on the CSAM list?
    - visarga 5y ago
      They got to scan for whatever the local law requires them to scan.
- Ashanmaril 5y ago
aborsy 5y ago
With the rise of end to end encrypted data and messaging, such as signal and WhatsApp, it makes sense for governments to shift the search to user’s devices and operating systems rather than cloud, before stuff gets encrypted.
endisneigh 5y ago
Am I the only one who finds no issue with this? Personally I’d prefer this than no encryption and scanning in the cloud, which is already possible. All of the slippery slope arguments have already been possible for nearly a decade now with the cloud. Can someone illustrate something wrong with this that’s not already possible today. Fundamentally unless you audited the client and the server yourself either (client or backend) scanning is possible, and therefore this “problem”. Where’s the issue?
- robertoandred 5y ago
  Never let a good Apple controversy go to waste. Clickbait blogs are thrilled to have something like this.
- fuzzer37 5y ago
  > Personally I’d prefer this than no encryption and scanning in the cloud, which is already possible. Why is that the alternative? How about everything is encrypted and nothing is scanned.
  - endisneigh 5y ago
    This is already possible if you self host your stuff. I’m talking about iCloud specifically here - backups are not encrypted so it’s either scan in the backend or scan on the client. If you don’t want to be scanned you can turn it off. I honestly don’t see the issue. It seems the only thing people can say are hypothetical situations here.
  - saynay 5y ago
    Realistically, because if they don't have something to scan for it, countries / EU are going to require them to provide a backdoor to scan it. There are already proposals in the EU to that affect.
- petersellers 5y ago
  Personally, I'd prefer no privacy intrusions at all. The issue is that Apple previously was not intruding into their user's privacy (at least publicly), but now they are. It sounds like your argument is that Apple could have been doing this all along and just not telling us. I find that unlikely mainly because they've marketed themselves as a privacy-focused company up until now.
throwawaymanbot 5y ago
This was never about protecting Children it seems. This was about the ability to create and install the infrastructure for the mass surveilling of people by smart device. Hashes can be created for anything on a phone. And hash collisions enable the "Near match" of hashes (Similar items). Lets pretend.. you use your face to log in to an iPhone, and there is a notice out for a certain person. If your face matches the hash, will you be part of the scan? You betcha.
1vuio0pswjnm7 5y ago
Perhaps I have I missed this in all the discussions of Apple's latest move but has anyone considered the following questions. Does Apple's solution only stop people from uploading illegal files to Apple's servers or does it stop them from uploading the files to any server. If Apple intends to control the operation of a computer purchased from Apple after the owner begins using it, does Apple have a duty to report illegal files found on that computer and stop them from being shared (anywhere, not just through Apple's datacenters). To me, this is why there is a serious distinction between a company detecting and policing what files are stored on their computers (i.e., how other companies approach this problem) and a company detecting and policing what files someone else's computer is storing and can transfer over the internet (in this case, unless I am mistaken, only to Apple's computers). Mind you, I am not familiar with the details of exactly how Apple's solution works nor the applicable criminal laws so these questions might be irrelevant. However I was thinking that if Apple really wanted to prevent the trafficking of ostensibly illegal files then wouldn't Apple seek to prevent their transfer not only to Apple's computers but to any computer (and also report them to the proper authorities). What duty does Apple have if they can "see into the owner's computer" and they detect illegal activity. If Apple is in remote control of the computer, e.g., they can detect the presence/absence of files remotely and allow or disallow full user control through the OS, then does Apple have a duty to take action.
- ysavir 5y ago
  > does Apple have a duty to report illegal images found on that computer and stop them from being shared Duty? No, that's the secondary question. The primary question is whether they have the right.
- judge2020 5y ago
  > Does Apple's solution only stop people from uploading illegal images to Apple's servers or does it stop them from uploading the images to any server. Only applies to iCloud Photos uploads, but the photos are still uploaded: when there's a match, the photo and a 'ticket' are uploaded and Apple's servers (after the servers themselves verify the match[0]) send the image to human reviewers to verify the CSAM before submitting it to police as evidence. 0: https://twitter.com/fayfiftynine/status/1427899951120490497 https://twitter.com/fayfiftynine/status/1427899951120490497 and https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf#page=6 https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
joelbondurant 5y ago
ALL the dozens of people I know at Apple would do anything to create a job looking at kiddie porn.
m3kw9 5y ago
Hashes can always collide, but how likely is it for neuralhash?
eurasiantiger 5y ago
”the analyzed code is not the final implementation that will be used with the CSAM system itself and is instead a generic version” So they are already running a generic version of this system since iOS 14.3?
ALittleLight 5y ago
Would this work as an attack? 1. Get a pornographic picture involving young though legal actors and actresses. 2. Encode a nonce into the image. Hash it checking for CSAM collisions. If you've found a collision go on to the next step, if not update the nonce and try again. 3. You now have an image that, to visual inspection will appear plausibly like CSAM, and to automated detection will appear like CSAM. Though, presumably, it is not illegal for you to have this image as it is, in fact, legal pornography. You can now text this to anyone with an iPhone who will be referred by Apple to law enforcement.
- arn 5y ago
  How would you know if it's a CSAM collision? I don't believe that database is publicly available anywhere, for obvious reasons.
  - arsome 5y ago
    Apple's client clearly has some revealing information on that...
  - ALittleLight 5y ago
    Good point. I can offer three possibilities to how you might know. First, a data leak of the material. Second, law enforcement presumably gives hashes of known CSAM to service operators so they can detect and report it. You could pose as or be the operator of such a service and get the hashes that way. Third, if you were a government operator you may have access to the hashes that way. (Although I guess corrupt government agents would have other easier ways of getting to you)
  - FabHK 5y ago
    Yes. That, plus it's unclear whether you can create a collision with a nonce. It's a perceptual hash, not a cryptographic one. Lastly, to compute that collision might be so expensive that you could instead compute small SHA-256 hashes, ie mine BTC, and use the money to obtain your sinister goals via other means.
- Ashanmaril 5y ago
trident5000 5y ago
Erosion of privacy and freedom always starts with "but think of the children"
zakember 5y ago
Is no one going to talk about how Apple is implementing this? If Apple is training a neural network to detect this kind of imagery, I would imagine there to be thousands, if not millions of child pornography images on Apple's servers that are being used by their own engineers to train this system
- arn 5y ago
  It's not a neural network. It's not AI. It's not trying to interpret image content. It's trying to identify known specific images, but is using a fuzzy fingerprinting match.
- zimpenfish 5y ago
  > If Apple is training a neural network to detect this kind of imagery NCMEC generate the hashes using their CSAM corpus.
guerrilla 5y ago
Something just occurred to me. If this goes forward, and then Microsoft and/or Google copy it, then we might see lawmakers expecting it. If that becomes the case, then it'd be yet another barrier to entry (in addition to a cop on every phone.) Isn't that where we already with things like Article 17 of the EU's Copyright Directive?
- shuckles 5y ago
  This is already the world we live in, with vendors like Thorn building SaaS solutions for people who accept UGC but don’t have the clout to integrate with NCMEC and Microsoft PhotoDNA directly.
stevenalowe 5y ago
Apple wants to enforce one of their corporate policies at my expense: battery, CPU, and unknown risks if/when they're wrong. What could possibly go wrong?
balozi 5y ago
They are slowly turning what is a philosophical argument into a technical question. Next step is for them to conjure up a technical answer before claiming victory. Most people are already bored by all the technical talk. Neural hash this: Its about Trust. Its about Privacy. Its about Boundaries between me and corporations/governments/etc.
bruce343434 5y ago
Oh man. They really want to die on this hill don't they.
_trampeltier 5y ago
Apple does check it just before you upload it to icloud. How much money do they save it's is checked clientside? How much would it cost for Apple to do it on there own servers, like everybody else does it?
- 63 5y ago
  Iirc the rationale for doing it clientside isn't saving money, but maintaining encryption. If it's checked client side, Apple should never get unencrypted uploads unless they're suspected to be CSAM
  - _trampeltier 5y ago
    Yes thats the official version. I still wonder what would be the cost difference. In this scale it is not just for free.
almostdigital 5y ago
It's clear now that Apple is never going to back down. It's bittersweet, I'm sad to see them go but also excited for what I think will be a FOSS renaissance.
cirrus3 5y ago
They are down to hash collision arguments against this now? Sounds pretty desperate. As if any normal user is going to upload a photo to iCloud that is a collision. The fact that such images are possible to generate means nothing by itself. Also, they would need to accidentally have 30 of them. Also, a human would have to not be able to tell the difference.
bArray 5y ago
> [..] the overall system is designed to account for this to happen in general, and that the analyzed code is not the final implementation that will be used with the CSAM system itself and is instead a generic version. I think the claim here is that you won't have access to the source images, and therefore generating collisions will be more difficult. But, if you do have access to the source images, this has been shown to be trivial. This of course doesn't stop nations states generating images that cause hash collisions, in fact they would be incentivized to do so. I would also add that Apple are behind the curve, attempts to crack the hashing algorithm more efficiently are still ongoing: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... > [..] not the final implementation [..] Why on earth would you invite people to come and test your algorithm and then say "sure, you broke it, but it's not the real one". This kind of defeats the point and seems like some bait and switch bullshit. I suspect this is some retroactive cope from management realising they can't deploy this version and whatever they do deploy needs to be heavily modified. > If Apple finds they are CSAM, it will report the user to law enforcement. One statistic I want to know is: How many people already trigger this report function in the wild? Surely currently is the largest number of positives they will ever have - if it turns out to be 0% - Apple should just scrap it. > Apple also said that after a user passes the 30 match threshold, a second non-public algorithm that runs on Apple's servers will check the results. So to avoid reporting, simply block Apple servers? Also, security by obscurity is not security - the algorithm supposedly being private just means that its not properly tested and Apple is not held to account. > "Apple actually designed this system so the hash function doesn't need to remain secret, as the only thing you can do with 'non-CSAM that hashes as CSAM' is annoy Apple's response team with some garbage images until they implement a filter to eliminate those garbage false positives in their analysis pipeline," Nicholas Weaver, senior researcher at the International Computer Science Institute at UC Berkeley, told Motherboard in an online chat. No. A report could be considered 'reasonable doubt' for law enforcement to do a full search. Imagine trying to explain to a judge why your iPhone shouldn't be searched because of a false-positive CSAM hash collision because of a malicious website you visited or a text message you received.
YeBanKo 5y ago
> Apple however told Motherboard in an email that that version analyzed by users on GitHub is a generic version, and not the one final version that will be used for iCloud Photos CSAM detection. Apple said that it also made the algorithm public. When did they make NeuralHash public?
ahD5zae7 5y ago
Honest question - as far as I could find out the files will be scanned on device but before upload to iCloud. Before they were scanned in the cloud after upload. That's the change as far as I understand it. But if it's scanned before upload then what is the difference? A few seconds? It would be scanned either way, before or after upload. Is that it? What I'm basically getting at is: are the files scanned after the user has expressed the intention of uploading them? That's what I understood. Am I wrong? Are the files scanned the moment they appear on your device, regardless of you iCloud status (even if you have disabled iCloud somehow)? Edit: typo

361 points by nickthegreek 5y ago

I have to say that I’m just sad about this. I was really convinced that Apple really believed in privacy and they will never do this (having a framework which can scan your device content). But maybe they already had it and this was like: “there is this delta and Afghanistan let’s make announcement now” We are just one stupid terrorist attack from full surveillance of everybody.

kevin_thibedeau 5y ago

How long until thishashcollisionisnotporn.com is a thing?

mono-bob 5y ago
I can confirm it was a fun weekend project and https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ has now become reality!
istingray 5y ago
sounds like a future NFT market
- eurasiantiger 5y ago
  Sounds like hipster T-shirts with hash collision images.
  - mono-bob 5y ago
    I can confirm it was a fun weekend project and https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ has now become reality! I included a button to buy a hipster T-shirt ;)
mono-bob 5y ago
Sounds like a fun weekend project.

yeldarb 5y ago

I created a proof of concept showing how OpenAI's CLIP model can function as a "sanity check" similar to how Apple says their server-side model works. In order for a collision to get through to the human checkers, the same image would have to fool both networks independently: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/

mrits 5y ago
Cool project. Wouldn't padding all generated images with a few items that match CLIP get around this though?
- yeldarb 5y ago
  Doing that would mean the NeuralHash would change though. And you'd have to not only get CLIP to identify CSAM in the generated image but also negate the parts of the generated image that are causing CLIP to label it as "generated" (while still colliding with the target NeuralHash). Unclear how hard this would actually be in practice (if I were going to attempt it, the first thing I'd try is to evolve a colliding image with something like CLIP+VQGAN) but certainly harder than finding a collision alone.
  - salawat 5y ago
    Take 2 CSAM pictures, known. Combine into one image. Swing and a miss. Not in the CSAM dataset. Take two images. Encode alternating pixels. Decode to get the original image back. Convert to different encodings print to PDF or Postscript. Encode as base64 representations of the image file... Who are we trying to fool here? This is kiddie stuff. This is more about trying to implant scanning capabilities on client devices.
    - yeldarb 5y ago
      Those are examples of triggering false negatives (CSAM that gets missed by the system), this collision attack is about triggering false positives (non-CSAM that triggers human intervention).

epistasis 5y ago

What's shocking to me is how little Apple management understood of what their actions looked like. Really stunning. For a company that marketed itself as one of the few digital service providers that consumers could trust, I just don't understand how they acted this way at all. Either there will be heads rolling at management, or Apple takes a permanent hit to consumer trust.

FabHK 5y ago
> What's shocking to me is how little Apple management understood of what their actions looked like. Really stunning. Maybe because they underestimated people's ignorance. I think they saw (and still do see) it as a better, more privacy preserving technique than what everyone else is doing.
- fraa-orolo 5y ago
  The thing is that this is a better and more privacy preserving technique. Their hubris is in not seeing or thinking that they will be able to stand up to all kinds of abuses of this system that its mere existence will invite; their hubris is also in thinking that they will be able to perfectly and without mistakes manage and overview a system making accusations so heinous that even the mere act of accusing destroy people's lives and livelihoods.
  - shuckles 5y ago
    What abuses apply to CSAM scanning in this hybrid pipeline which don’t apply to iCloud Backup? If they scanned on the server, why couldn’t governments “slippery slope” abuse the system by requiring that all files on iOS end up in iCloud Backup, where they can be scanned by the system? Since the announcement, I can think of a dozen ways Apple could be easily forced into scanning all the contents of your device by assembling features they’ve already shipped. Yet they haven’t. At some point, people need to produce evidence that Apple cannot hold the line they’ve said they will.
    - candiodari 5y ago

gjsman-1000 5y ago

For everyone upset about Apple's CSAM scanning, I think we all forgot about the EARN IT Act. It was nearly passed last year but Congress was finished before it could be voted on. It had Bipartisan support and would've virtually banned E2E of any kind. And it would have required scanning everywhere according to the recommendations of a 19-member board of NGOs and unelected experts. The reason for this mandatory backdoor was child safety... even though AG Barr admitted that reducing the use of encryption had useful side effects. If you are Apple, even though EARN IT failed... you know where Washington's heart lies. Is CSAM scanning a "better alternative", a concession, an appeasement, a lesser evil, in the hope this prevents EARN IT from coming back? Also, many people forgot about the Lawful Access to Encrypted Data Act of 2020, or LAED, which would unilaterally banned E2E encryption in entirety and required that all devices featuring encryption must be unlockable by the manufacturer. That also was on the table.

samename 5y ago
Why can’t we have the alternative we already have, where we don’t have our phones scans and E2EE exists without compromise? Why do we have to accept it any other way? If you’re trying to frame this as “we need to prevent Congress from ever passing something like the EARN IT act”, I agree. Apple and other tech companies already lobby Congress. Why aren’t they lobbying for encryption?
- gjsman-1000 5y ago
  They did, they lobbied heavily against EARN IT, and so did groups like the ACLU and EFF. However, it didn't stop Congress members from moving it along through the process - they were only stopped because Congress just ran out of time. It was clear that Congress was not interested in listening at all to the lobbyists on the issue. It's clear that EARN IT could literally be revived any day if Apple didn't do something to say "we don't need it because we've already satisfied your requirements."
  - orangecat 5y ago
    It's clear that EARN IT could literally be revived any day if Apple didn't do something to say "we don't need it because we've already satisfied your requirements." Alternatively, "Apple has shown that it's possible to do without undue hardship, so we should make everyone else do it too".

geoah 5y ago

> The system relies on a database of hashes—cryptographic representations of images—of known CSAM photos provided by National Center for Missing & Exploited Children (NCMEC) and other child protection organizations. “Cryptographic representations of images”. That’s not the case though right? These are “neuralhashes” afaik which are nowhere close to cryptographic hashes but rather locality sensitive hashes which is a fancy speak for “the closer two images look like, the more similar the hash”. Vice and others keeps calling the cryptographic. Am I missing something here?

andrewmcwatters 5y ago
Edit: "The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, and images that are different from one another result in different hashes."[1] Apple isn't using a "similar image, similar hash" system. They're using a "similar image, same hash" system. [1]: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- heavyset_go 5y ago
  > There really is no sound concept of "the more similar the hash." Perceptual hashes are not cryptographic hashes. Perceptual hashing systems do compare hashes using a distance metric like the Hamming distance. If two images have similar hashes, then they look kind of similar to one another. That's the point of perceptual hashing.
  - salawat 5y ago
    Except when you're not measuring the similarity you think you are, because NeuralNets are f&#@ing weird. You know, let me put it this way. Yiu know that one really weird family member I'm pretty sure everyone either has or is? Guess what? They're a neural net too. This is what Apple is asking you to trust.
    - heavyset_go 5y ago

dang 5y ago

Ongoing related threads: Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068 https://news.ycombinator.com/item?id=28219068 - Aug 2021 (542 comments) Convert Apple NeuralHash model for CSAM Detection to ONNX - https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391 - Aug 2021 (155 comments)

jl6 5y ago

> Anti-Child Abuse Imagery Tech Room for improvement in the headline.

firebaze 5y ago

I think we're up for a new Apple CEO in a few days.

csilverman 5y ago
The turmoil of Apple abruptly booting the CEO who's overseen the most obscenely prosperous stretch in its history would cost the company vastly more than the current controversy about CSAM. I'm not saying this as a fan of either Cook or their anti-CSAM measures; I'm neither, and if anyone is ever wrongfully arrested because Apple's system made a mistake, Cook may well wind up in disgrace depending on how much blame he can/can't shuffle off to subordinates. I don't think we're there yet, though.
- atonse 5y ago
  He's not Ballmer but Ballmer also presided over record profits even though MS barely did anything remotely interesting under his watch. Just saying that money hides problems.

mzs 5y ago

>… Apple also said that after a user passes the 30 match threshold, a second non-public algorithm that runs on Apple's servers will check the results. >"This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database," …

Barrin92 5y ago

it's kind of silly how this is treated like some deeply technical issue. Apple's 'one of a trillion' claim is either true and the software is useless, because I'm pretty sure pedophiles can figure out what a watermark or a gaussian blur in paint net is, or it's imprecise and actually detects things which is the very thing that makes it dangerous to the public. It's a direct trade-off and the error tolerance of any such filter is the only thing that makes it useful so we can basically stop arguing about the depths of implementation details or how high the collision rate of the hashing algorithm is etc. If this thing is supposed to catch anyone it needs to be magnitudes more lenient than any of those minor faults.

734129837261 5y ago
Exactly right. The tech Apple uses can be one of two things: 1. It requires a perfect 1:1 match (their documentation says this is not the case); 2. Or it has some freedom in detecting a match, probably including a match with a certain percentage. If it's the former, it's completely useless. A watermark or a randomly chosen pixel with a slightly different hue and the hash would be completely different. So, it's not #1. It's going to be #2. And that's where it becomes dangerous. The government of the USA is going to look for child predators. The government of Saudi Arabia is going to track down known memes shared by atheists, and they will be put to death; heresy is a capital offence over there. And China will probably do their best to track down Uyghurs so they can make the process of elimination even easier. It's not like Apple hasn't given in to dictatorships in the past. This tech is absolutely going to kill people.
- endisneigh 5y ago
  What? Why would memes be on the CSAM list?
  - visarga 5y ago
    They got to scan for whatever the local law requires them to scan.
Ashanmaril 5y ago

aborsy 5y ago

With the rise of end to end encrypted data and messaging, such as signal and WhatsApp, it makes sense for governments to shift the search to user’s devices and operating systems rather than cloud, before stuff gets encrypted.

endisneigh 5y ago

Am I the only one who finds no issue with this? Personally I’d prefer this than no encryption and scanning in the cloud, which is already possible. All of the slippery slope arguments have already been possible for nearly a decade now with the cloud. Can someone illustrate something wrong with this that’s not already possible today. Fundamentally unless you audited the client and the server yourself either (client or backend) scanning is possible, and therefore this “problem”. Where’s the issue?

robertoandred 5y ago
Never let a good Apple controversy go to waste. Clickbait blogs are thrilled to have something like this.
fuzzer37 5y ago
> Personally I’d prefer this than no encryption and scanning in the cloud, which is already possible. Why is that the alternative? How about everything is encrypted and nothing is scanned.
- endisneigh 5y ago
  This is already possible if you self host your stuff. I’m talking about iCloud specifically here - backups are not encrypted so it’s either scan in the backend or scan on the client. If you don’t want to be scanned you can turn it off. I honestly don’t see the issue. It seems the only thing people can say are hypothetical situations here.
- saynay 5y ago
  Realistically, because if they don't have something to scan for it, countries / EU are going to require them to provide a backdoor to scan it. There are already proposals in the EU to that affect.
petersellers 5y ago
Personally, I'd prefer no privacy intrusions at all. The issue is that Apple previously was not intruding into their user's privacy (at least publicly), but now they are. It sounds like your argument is that Apple could have been doing this all along and just not telling us. I find that unlikely mainly because they've marketed themselves as a privacy-focused company up until now.

throwawaymanbot 5y ago

This was never about protecting Children it seems. This was about the ability to create and install the infrastructure for the mass surveilling of people by smart device. Hashes can be created for anything on a phone. And hash collisions enable the "Near match" of hashes (Similar items). Lets pretend.. you use your face to log in to an iPhone, and there is a notice out for a certain person. If your face matches the hash, will you be part of the scan? You betcha.

1vuio0pswjnm7 5y ago

Perhaps I have I missed this in all the discussions of Apple's latest move but has anyone considered the following questions. Does Apple's solution only stop people from uploading illegal files to Apple's servers or does it stop them from uploading the files to any server. If Apple intends to control the operation of a computer purchased from Apple after the owner begins using it, does Apple have a duty to report illegal files found on that computer and stop them from being shared (anywhere, not just through Apple's datacenters). To me, this is why there is a serious distinction between a company detecting and policing what files are stored on their computers (i.e., how other companies approach this problem) and a company detecting and policing what files someone else's computer is storing and can transfer over the internet (in this case, unless I am mistaken, only to Apple's computers). Mind you, I am not familiar with the details of exactly how Apple's solution works nor the applicable criminal laws so these questions might be irrelevant. However I was thinking that if Apple really wanted to prevent the trafficking of ostensibly illegal files then wouldn't Apple seek to prevent their transfer not only to Apple's computers but to any computer (and also report them to the proper authorities). What duty does Apple have if they can "see into the owner's computer" and they detect illegal activity. If Apple is in remote control of the computer, e.g., they can detect the presence/absence of files remotely and allow or disallow full user control through the OS, then does Apple have a duty to take action.

ysavir 5y ago
> does Apple have a duty to report illegal images found on that computer and stop them from being shared Duty? No, that's the secondary question. The primary question is whether they have the right.
judge2020 5y ago
> Does Apple's solution only stop people from uploading illegal images to Apple's servers or does it stop them from uploading the images to any server. Only applies to iCloud Photos uploads, but the photos are still uploaded: when there's a match, the photo and a 'ticket' are uploaded and Apple's servers (after the servers themselves verify the match[0]) send the image to human reviewers to verify the CSAM before submitting it to police as evidence. 0: https://twitter.com/fayfiftynine/status/1427899951120490497 https://twitter.com/fayfiftynine/status/1427899951120490497 and https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf#page=6 https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

joelbondurant 5y ago

ALL the dozens of people I know at Apple would do anything to create a job looking at kiddie porn.

m3kw9 5y ago

Hashes can always collide, but how likely is it for neuralhash?

eurasiantiger 5y ago

”the analyzed code is not the final implementation that will be used with the CSAM system itself and is instead a generic version” So they are already running a generic version of this system since iOS 14.3?

ALittleLight 5y ago

Would this work as an attack? 1. Get a pornographic picture involving young though legal actors and actresses. 2. Encode a nonce into the image. Hash it checking for CSAM collisions. If you've found a collision go on to the next step, if not update the nonce and try again. 3. You now have an image that, to visual inspection will appear plausibly like CSAM, and to automated detection will appear like CSAM. Though, presumably, it is not illegal for you to have this image as it is, in fact, legal pornography. You can now text this to anyone with an iPhone who will be referred by Apple to law enforcement.

arn 5y ago
How would you know if it's a CSAM collision? I don't believe that database is publicly available anywhere, for obvious reasons.
- arsome 5y ago
  Apple's client clearly has some revealing information on that...
- ALittleLight 5y ago
  Good point. I can offer three possibilities to how you might know. First, a data leak of the material. Second, law enforcement presumably gives hashes of known CSAM to service operators so they can detect and report it. You could pose as or be the operator of such a service and get the hashes that way. Third, if you were a government operator you may have access to the hashes that way. (Although I guess corrupt government agents would have other easier ways of getting to you)
- FabHK 5y ago
  Yes. That, plus it's unclear whether you can create a collision with a nonce. It's a perceptual hash, not a cryptographic one. Lastly, to compute that collision might be so expensive that you could instead compute small SHA-256 hashes, ie mine BTC, and use the money to obtain your sinister goals via other means.
Ashanmaril 5y ago

trident5000 5y ago

Erosion of privacy and freedom always starts with "but think of the children"

zakember 5y ago

Is no one going to talk about how Apple is implementing this? If Apple is training a neural network to detect this kind of imagery, I would imagine there to be thousands, if not millions of child pornography images on Apple's servers that are being used by their own engineers to train this system

arn 5y ago
It's not a neural network. It's not AI. It's not trying to interpret image content. It's trying to identify known specific images, but is using a fuzzy fingerprinting match.
zimpenfish 5y ago
> If Apple is training a neural network to detect this kind of imagery NCMEC generate the hashes using their CSAM corpus.

guerrilla 5y ago

Something just occurred to me. If this goes forward, and then Microsoft and/or Google copy it, then we might see lawmakers expecting it. If that becomes the case, then it'd be yet another barrier to entry (in addition to a cop on every phone.) Isn't that where we already with things like Article 17 of the EU's Copyright Directive?

shuckles 5y ago
This is already the world we live in, with vendors like Thorn building SaaS solutions for people who accept UGC but don’t have the clout to integrate with NCMEC and Microsoft PhotoDNA directly.

stevenalowe 5y ago

Apple wants to enforce one of their corporate policies at my expense: battery, CPU, and unknown risks if/when they're wrong. What could possibly go wrong?

balozi 5y ago

They are slowly turning what is a philosophical argument into a technical question. Next step is for them to conjure up a technical answer before claiming victory. Most people are already bored by all the technical talk. Neural hash this: Its about Trust. Its about Privacy. Its about Boundaries between me and corporations/governments/etc.

bruce343434 5y ago

Oh man. They really want to die on this hill don't they.

_trampeltier 5y ago

Apple does check it just before you upload it to icloud. How much money do they save it's is checked clientside? How much would it cost for Apple to do it on there own servers, like everybody else does it?

63 5y ago
Iirc the rationale for doing it clientside isn't saving money, but maintaining encryption. If it's checked client side, Apple should never get unencrypted uploads unless they're suspected to be CSAM
- _trampeltier 5y ago
  Yes thats the official version. I still wonder what would be the cost difference. In this scale it is not just for free.

almostdigital 5y ago

It's clear now that Apple is never going to back down. It's bittersweet, I'm sad to see them go but also excited for what I think will be a FOSS renaissance.

cirrus3 5y ago

They are down to hash collision arguments against this now? Sounds pretty desperate. As if any normal user is going to upload a photo to iCloud that is a collision. The fact that such images are possible to generate means nothing by itself. Also, they would need to accidentally have 30 of them. Also, a human would have to not be able to tell the difference.

bArray 5y ago

> [..] the overall system is designed to account for this to happen in general, and that the analyzed code is not the final implementation that will be used with the CSAM system itself and is instead a generic version. I think the claim here is that you won't have access to the source images, and therefore generating collisions will be more difficult. But, if you do have access to the source images, this has been shown to be trivial. This of course doesn't stop nations states generating images that cause hash collisions, in fact they would be incentivized to do so. I would also add that Apple are behind the curve, attempts to crack the hashing algorithm more efficiently are still ongoing: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... > [..] not the final implementation [..] Why on earth would you invite people to come and test your algorithm and then say "sure, you broke it, but it's not the real one". This kind of defeats the point and seems like some bait and switch bullshit. I suspect this is some retroactive cope from management realising they can't deploy this version and whatever they do deploy needs to be heavily modified. > If Apple finds they are CSAM, it will report the user to law enforcement. One statistic I want to know is: How many people already trigger this report function in the wild? Surely currently is the largest number of positives they will ever have - if it turns out to be 0% - Apple should just scrap it. > Apple also said that after a user passes the 30 match threshold, a second non-public algorithm that runs on Apple's servers will check the results. So to avoid reporting, simply block Apple servers? Also, security by obscurity is not security - the algorithm supposedly being private just means that its not properly tested and Apple is not held to account. > "Apple actually designed this system so the hash function doesn't need to remain secret, as the only thing you can do with 'non-CSAM that hashes as CSAM' is annoy Apple's response team with some garbage images until they implement a filter to eliminate those garbage false positives in their analysis pipeline," Nicholas Weaver, senior researcher at the International Computer Science Institute at UC Berkeley, told Motherboard in an online chat. No. A report could be considered 'reasonable doubt' for law enforcement to do a full search. Imagine trying to explain to a judge why your iPhone shouldn't be searched because of a false-positive CSAM hash collision because of a malicious website you visited or a text message you received.

YeBanKo 5y ago

> Apple however told Motherboard in an email that that version analyzed by users on GitHub is a generic version, and not the one final version that will be used for iCloud Photos CSAM detection. Apple said that it also made the algorithm public. When did they make NeuralHash public?

ahD5zae7 5y ago

Honest question - as far as I could find out the files will be scanned on device but before upload to iCloud. Before they were scanned in the cloud after upload. That's the change as far as I understand it. But if it's scanned before upload then what is the difference? A few seconds? It would be scanned either way, before or after upload. Is that it? What I'm basically getting at is: are the files scanned after the user has expressed the intention of uploading them? That's what I understood. Am I wrong? Are the files scanned the moment they appear on your device, regardless of you iCloud status (even if you have disabled iCloud somehow)? Edit: typo