16 ms·
Show HN: 59a34eabe31910abfb06f308 – NeuralHash Collision Demo
- smt88 5y agoThis simple site is a far better demo and explanation of the extreme danger of Apple's proposal than any of the long articles written about it. Thank you for caring enough to put this together and publish it.
- aomobile 5y agoIs Apple proposing anything? I think they are just going ahead with it
- nicce 5y agoSigh… All these articles and demos are based on their old release, which is also emulated on onnxruntime. Who knows how they actually use it. We should cool down a little and wait for the real system, before expecting anything from them. Let’s see then how bad it is, now we are just victims of speculation.
- echelon 5y agoYou mean let Apple slide here?
- JohnJamesRambo 5y agoYou are basically saying “Let’s just wait and let them scan our phones.”
- nicce 5y agoAt this point, it is hard to argue anything which sides with Apple, as the most have set their minds. However, this is something which have been planned for years. Bad PR won't undo this, and stepping back is not happening overnight. There is so much misinformation in this subject, because it is really hard to see the bigger picture. We are in this situation, because of the leaked information. Was it intentional for political reasons, who knows. Apple is definitely going to announce some E2EE features very soon, such as backups in the next event. It is very hard to invent better system, than they have released now, to enable E2EE and apply CSAM scanning. When you step into the Apple system (which is quite closed, and nobody really can make extensive review for it), you give all your trust for Apple. Whether this scanning happens on your phone, or in the cloud, it does not really matter, as long as the end result is the same (same files are scanned). How about the abuse of the system? Prior on-device scanning, government could ask Apple to add feature for telling, if users have Winnie the Pooh images on their phone. (Could you spy for us?) After adding the on-device scanning, now they ask Apple to add this hash dataset to find Winnie the Pooh images. (Could you spy for us?) The question is still the same. Does some technical change affect for the managers and the chair? How about forging CSAM images to make FBI to show up in the neighbors door? Not going to happen easily. The only problem here is to brute force their human review process. In that case, it is further double checked by NCMEC. But still, at first we need some hash which exist in their database, and secondly that image should be uploaded to iCloud. And not only once, 30 times with different hash.
- fragmede 5y ago> which is quite closed, and nobody really can make extensive review for it That's what Apple wants you to think, and they're very successful at marketing, but it's not really true. The jailbreak scene is not what it once was, but there are innards to Apple devices and services. Apple engineers put their pants on one leg at a time, the same as everyone else.
- nicce 5y agoThis is true, but there has been always a great delay on reverse process. One year can mean a lot in the current world.
- vineyardmike 5y agoIt seems that this is based of a github project that takes the hash algo directly from binaries in MacOS - so it is what apple shipped.
- deleted 5y ago[deleted]
- smt88 5y agoTest in prod? Wait until one of these false positives gets to court?
- matheusmoreira 5y agoThey're the ones who need to cool it with the global surveillance.
- ostenning 5y agoProbably gaslight their users for the next 4 years and then backflip with a Key Note saying how they are revolutionising privacy by removing it
- hypothesis 5y agoThey are going right ahead with this and there is no stopping it. And, if history is any indication, we’re going to hear more of this when needed… [1] https://www.theguardian.com/technology/2019/jul/26/apple-contractors-regularly-hear-confidential-details-on-siri-recordings https://www.theguardian.com/technology/2019/jul/26/apple-con...
- DavideNL 5y agoNot in the EU, it will be blocked by privacy regulation..
- Natsu 5y agoDoes it? This just gets things flagged for another test and then manual review, it doesn't flag you as a pedophile or something. It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.
- setr 5y agoAny notion of privacy is eliminated, if you can reasonably expect arbitrary photos to be selected and sent to apple for manual review (where selection could make up really any % of the photos you upload, and presumably you won’t be told which ones were flagged for review). It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
- selsta 5y agoThis has always been the case when uploading something to the cloud unencrypted. You never had any privacy in the first place.
- vineyardmike 5y agoBut this isn't uploaded to the cloud. Its your device.
- zepto 5y agoThis site demonstrates no danger at all. Hash collisions like this are expected. Anyone who claims this is evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.
- firebaze 5y agoIt's hard to reply to your post in the spirit of HN in assuming best intentions. But please suppose most of the readers here do understand how the system works, and most of them don't approve how it works.
- wmf 5y agoI understand how the system works so I know that hash collisions are not dangerous because they have to be manually reviewed by multiple humans. Since you understand how the system works, please explain the danger of hash collisions instead of meta-commenting.
- smoldesu 5y agoI'm not them, but they have a point. This ends up getting re-explained in every single one of these threads, so I'll keep it brief: - There are two main components here: automated flagging and supervised, human confirmation - It has been proven that the automated flagging system can be fooled - The last line of defense is now a human: they have to look at an obfuscated version of your colliding image to determine if it's objectionable The current discussion is mostly around the second portion, moreover what kind of social-engineering attacks would be most effective to overcome a relatively flimsy single-human point of failure.
- zepto 5y ago> - It has been proven that the automated flagging system can be fooled This statement is absolutely false. No such proof has been presented anywhere, including in the linked article. Adversarial Neuralhash collisions are expected and the system is designed avoid false positives if they are encountered. Here is the relevant paragraph from Apple’s documentation: “as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.” https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
- mfer 5y agoTo add some context... When photos are uploaded to places like Google images they are scanned for this material. Apple is currently only processing images that are being uploaded to iCloud. So, these are photos people intended to upload to them. This means it's not unwanted upload of photos to Apple servers. That is happening anyway. Note, I think scanning should happen on Apple servers and not the phones. This is an invasion onto the devices in my book. I wonder if this is a way to scale the system and reduce their servers and power use.
- mono-bob 5y agoThank you for your kind words. This situation is something close to my heart, I think privacy should be a fundamental right for everyone, and it is essential to inform people.
- spuz 5y agoApple have stated that they will make the database of hashes that their system uses auditable by researchers. Does anyone know if that has happened yet? Is it possible to view the database and if so, in what form? Can the actual hashes be extracted? If so then that would obviously open up the kind of attack described in the article. Otherwise, it would be interesting to know how Apple expects the database to be auditable without revealing the hashes themselves.
- cucumb3rrelish 5y ago> Is it possible to view the database and if so, in what form? Not for the general public, it's also in encrypted form on the device I can tell you how they will make it auditable without revealing hashes to all of their users: by only showing them to experts, just like you don't publish the entire databreach credentials in cleartext but hand them to someone experienced who acts as a middleman for public transparency
- bnj 5y agoMy understanding is that Apple is going to publish a hash of the database, so a user would be able to compare the hash of the database on their device with the published hash to be “assured” that the same dataset is being used everywhere.
- vmception 5y agoI won the OnlyFans reversal bet Any prediction markets on an Apple CSAM reversal?
- oh_sigh 5y agoI predict no reversal. No one cares about this besides for privacy zealots who complain about literally everything. "Real" people actually care about getting to see their iProstitute naked though.
- zepto 5y agoWorse than that - there are excellent reasons to oppose this system, but the debate is centered around misunderstandings and false claims rather than why the system is wrong. That’s not going to go well.
- vmception 5y agoalways nice to see your attractive friends naked
- oh_sigh 5y agoIf people are only talking to you because you are paying them money to talk to you, they aren't your friends.
- vmception 5y agoA nice quip, I first heard it about fraternities and university greek life A) Many OnlyFans accounts are free to subscribe to B) Many OnlyFans accounts are free to direct message C) You as a consumer dont have your OnlyFans account tied to your identity from the performers perspective, so you can do both if you want. Be a friend and consumer without tying the two. No different than seeing your friend on a billboard or magazine. But you can also tie it to your identity if you want, not all people treat their friends only as customers once they subscribe. If you have a toxic relationship then you need to address that on your own. D) Much of OnlyFans commerce comes from its own economy of creators. Many subscribe to each other. Many also cross promote.
- umvi 5y agoSeems like this CSAM tech could be super useful in China for detecting winnie the pooh or other evidence of thought crime against the regime. Even if Apple doesn't end up rolling it out, I'm sure Huawei is taking careful notes.
- deleted 5y ago[deleted]
- WesolyKubeczek 5y agoWhy detect it, if you can plant it?
- vkou 5y agoYou plant it to catch problem people that you already know about, you detect it to catch problem people that you don't know about.
- WesolyKubeczek 5y agoYou also plant stuff to have a handle on people you’d like to use.
- vageli 5y agoWhy not both? Now you don't need to wait for your planted material to come to light, an automated system will surface it for you.
- ribosometronome 5y agoSeems like an authoritarian government wanting to quash Winnie the Pooh memes would be better suited to using ML to detect them rather than having to rely on a list of known memes.
- alfalfasprout 5y agoWhy? Labor is extremely cheap in China-- they can easily hire tens of thousands of people to manually flag memes they don't like and add them to a list... they can then rely on ML augmented proximity hashes to do the actual detection (WeChat does this).
- drexlspivey 5y agoCool! Now do one where the user uploads the image and it tweaks it to find a collision on the fly.
- grepfru_it 5y agoOr the opposite where it’s no longer a known image hash…
- spullara 5y agoSo 30+ images get flagged and they run it against the real CSAM database and it doesn't match? Or let's say someone is able to somehow make an image that gets flagged by both and someone looks at the image and it isn't CSAM. Nothing happens.
- deleted 5y ago[deleted]
- cucumb3rrelish 5y agoimo this would only be relevant if there was no human verification or if apple as a whole went rogue
- firebaze 5y agoThis page directly links to the EFF: https://act.eff.org/action/tell-apple-don-t-scan-our-phones https://act.eff.org/action/tell-apple-don-t-scan-our-phones Please spend a few bucks on supporting them. A bit of a background on why apple did this (this was flagged, but I don't know why): https://news.ycombinator.com/item?id=28259622 https://news.ycombinator.com/item?id=28259622
- kilroy123 5y agoDone.
- dang 5y agoUsers flagged it. We can only guess why users flag things, but in this case it's not hard to guess - some readers are past the fatigue stage with this story, which has had dozens of threads at this point. Others, needless to say, have not had enough yet.
- LeifCarrotson 5y agoRegarding that background, I agree that the scale does require more consideration. For a similar example, Apple's Airtags have a number of protections against using them for nefarious purposes. Tile, Samsung SmartTag, and similar devices don't, but Tile users are about 2 orders of magnitude less common than Apple users. You could get pinpoint tracking of a person just about anywhere by dropping an Airtag in their bag, but you'd be lucky to pick up one or two pings if you dropped a Tile. I think the NeuralHash client scanning approach is overly invasive, but at the same time I think it's a good thing that Apple has someone who's thinking about bad things that can happen while people are using their products.
- WesolyKubeczek 5y agoEach image on the left has a blob vaguely similar to the highlights in the dog image on the right. Likely the "perceptual" algorithm isn't "perceiving" contrast the same way human eyes and brains do.
- slg 5y agoNow let's create one for the hash matching that Google, Microsoft, and other cloud providers use. If your problem with Apple's proposal is the fact they do hash matching (rather than the system is run on your device), why is the criticism reserved for Apple instead of being directed at everyone who does hash matching to find CSAM? It seems like a lot of the backlash is because Apple is being open and honest about this process. I worry that this will teach companies that they need to hide this type of functionality in the future.
- mulmen 5y agoThe backlash is because Apple differentiates on privacy. Users are understandably upset that Apple went back on their word and no longer puts user privacy first.
- slg 5y agoDoes that mean we shouldn't hold companies accountable for violating privacy as long as they don't market themselves as being pro-privacy?
- wmf 5y agoHonestly, if we want a higher baseline of privacy (or environmental protection, or worker rights, etc.) we need laws not public pressure.
- kristjansson 5y agoThis _is_ a user privacy feature. It's clearly a precursor to E2EE iCloud services they weren't able to provide before.
- mulmen 5y agoI fail to see how scanning my photos on my device is a win for privacy. What’s the point of E2E encryption if there’s access to the device? E2EE is meaningless in the presence of this backdoor.
- CodesInChaos 5y agoSince the target image is chosen, this is a (second) preimage, not merely a collision.
- anishathalye 5y agoEven though it's not apparent from this demo website, the underlying code performs a preimage attack (it doesn't actually use a target image, just a target hash), so it's even stronger than a second preimage attack.
- BAKAINORI 5y agohttps://pgslot.global/ https://pgslot.global/
- BAKAINORI 5y agoThis new addition to the schedule pairs developers and publishers with monetisation companies to discuss how to make (more) money from your game. https://pgslot.global/ https://pgslot.global/
- bronzeage 5y agoIf it's so easy to produce collisions, it's only a matter of time before someone submits illegal material that intentionally collides with the most common images, resulting in insane number of collisions and rendering the system useless.
- 1vuio0pswjnm7 5y ago[deleted]
- SlowBall 5y agoHow could a picture be in a CSAM database if you just took it with your camera?
- nonbirithm 5y agoIrrespective of whether or not NeuralHash is flawed, should Apple scan user data or should they not? If not, what is going to convince them to stop at this point? I believe that they should scan user data in some capacity, because this is about data that causes harm to children. However, I believe that they should not run the scan on the device, because that carries significant drawbacks for personal privacy.
- bequanna 5y agoMy friend, this was never about “data that causes harm to children”. This is a carefully chosen stated reason for Apple to add this functionality to avoid user pushback. The actual uses will vary from country to country, but on the whole will not result in the protection of anyone other than the government or ruling party. In countries with less civil freedom this will be exploited and used as a tool of oppression against minority political groups.
- sharken 5y agoI think this has been established already, but nevertheless it's best to repeat it. The on-device scanning capabilities will be expanded in the future, but it can be stopped if Apple care about their users. Right now it seems they don't care. Forgot to mention that I'll not be recommending or purchasing any Apple devices until they halt this scanning idea.
- bequanna 5y ago> but it can be stopped if Apple care about their users. Apple cares about selling devices, maximizing revenue. Users are just something required to authorize transactions. If Apple has to kowtow to the most oppressive regimes in the world to keep doing the above, you bet they will and probably already are. And since it is in the interest of both Apple + Gov't to do shady stuff in the dark, there is a good chance they can hide it the public indefinitely.
- djbebs 5y agoYour devices should not betray you, period no exceptions. I dont care if it will save billions of lives, your devices should not betray you.
- aliabd 5y agoHere's a web demo[0] where you can try out any two images and see the resulting hashes, and whether there's a collision. You can also try your own transformations (rotation, adding a filter, etc) on the image. Demo was built using Gradio[1]. [0]: https://huggingface.co/spaces/akhaliq/AppleNeuralHash2ONNX https://huggingface.co/spaces/akhaliq/AppleNeuralHash2ONNX [1]: https://gradio.dev https://gradio.dev
- seanbarry 5y agoCan somebody please explain to me how one can go about finding images that have collision hashes? Or how you can create an image to have a specific hash?
- nullc 5y agoNeuralhash is a neural network. This means that its locally differentiable. To make an image match a specific hash, you pass the you want to modify image through neuralhash and compute a difference with your target hash, then ask your netural network library to use reverse mode automatic differentiation to give you the gradients for each of the outputs with respect to the input pixels. Update the input pixels. To make the collisions that look good, you need to either augment the objective to include some 'good looking' metric, or condition your gradient descent to prefer good looking results. In my examples ( https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... ) I used a gaussian blur to highpass the error between my starting image and the current work in progress, and fed that back in, and adapted the feedback to be as great as possible while still keeping it colliding. The bad looking examples that people initially posted just did nothing to try to make the result look good. I'm thrilled to see that someone other than me has now also worked out getting fairly reasonable looking examples.
- seanbarry 5y agoAwesome, thanks for taking the time to reply! Much appreciated.
- DavideNL 5y ago> For example, it's possible to detect political campaign posters or similar images on users' devices by extending the database. So who controls the database?
- colinmhayes 5y agoChina is Apple's second biggest market and an integral part of their supply chain. It will be very difficult for apple to say no when China demands this tech be sued to find thought crime.
- cucumb3rrelish 5y agoamerican nonprofits which also supply cloudflare afaik, it needs to match multiple databases https://www.theverge.com/2021/8/13/22623859/apple-icloud-photos-csam-scanning-security-multiple-jurisdictions-safeguard https://www.theverge.com/2021/8/13/22623859/apple-icloud-pho...
- advisedwang 5y agoThe argument against this tech is a slippery slope argument - that this technology will eventually be expanded to prevent copyright infringement, censor obscenity, limit political speech or other areas. I know this is a controversial take (in HN circles), but I no longer believe this will happen. This kind of tech has existed for a while, and it simply hasn't happened that it's been mis-applied. I now think that this technology has proved to be an overall net good.
- pnt12 5y agoEven though I disagree, this post predates that argument: it shows how easy collisions are,against the one in a trillion argument. Maybe someone sends you some unsuspicious pictures and you get a visit by the FBI, or some apple (outsourced) employee reviewing them. maybe it happens with some photo you take, who knows. This is not a slippery slope, this is what can happen with the process as defined by Apple.
- spoonjim 5y agoImagine hiring a young-looking 18-year old model to duplicate the photos in the database and create a hash collision. Now you have a photo which is perfectly legal for you to possess but can rain down terror on anyone you can distribute this file to.
- nullc 5y agoI think Apple may have figured out that the best way to get people to accept backdoored encryption is simply to not call it backdoored, and claim that its a privacy feature... ...as if having a trillion dollar corporation playing batman and going on a vigilante crusade to scan your private files is a situation we should already be comfortable with.
- hypothesis 5y agoAnd for technically minded people we will roll out a “look at this shiny new tech” trope, which will distract some from any overarching issues. Heck they might even produce some ideas how to improve new surveillance tech for free! Pure brilliance.