15 ms·
Convert Apple NeuralHash model for CSAM Detection to ONNX
- wodenokoto 5y ago> ONNX is an open format built to represent machine learning models. ONNX defines a common set of operators - the building blocks of machine learning and deep learning models - and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers. - onnx.ai I have never heard of this before and had to look it up. Is it widely used? Can I define a model in onnx and run it "everywhere", instead of learning pytorch or tensorflow?
- Loranubi 5y agoYes the goal is to run it framework independent. Pytorch for example can export models to ONNX.
- deleted 5y ago[deleted]
- punnerud 5y agoYes. Could be that you have to check what operators is available when you move from one framework to another using ONNX. One example is ONNX.js that make it possible to run the model using JavaScript, made by Microsoft. If the operator is not available (ex. ArgMax for WebGL) you have to switch to something equivalent and retrain. List of supported operators for ONNX.js: https://github.com/microsoft/onnxjs/blob/master/docs/operators.md https://github.com/microsoft/onnxjs/blob/master/docs/operato...
- red2awn 5y agoWell you would still need to learn Pytorch/tensorflow and define your network there, you then export it to ONNX format for deployment.
- tvirosi 5y agoOnnx has really poor support by microsoft. I suspect they've basically abandoned it internally (their onnxjs variant is orders of magnitude slower than tfjs[1]). It's a good 'neutral standard' for the moment but we should all eventually probably move away from it long term. [1] https://github.com/microsoft/onnxjs/issues/304 https://github.com/microsoft/onnxjs/issues/304
- akhundelar 5y agoONNX != onnxjs ONNX is a representation format for ML models (mostly neural networks). onnxjs is a just a browser runtime for ONNX models. While it may be true that onnxjs is neglected, please note that the 'main' runtime, onnxruntime, is under heavy active development[1]. Moreover, Microsoft is not the sole steward of the ONNX ecosystem. They are one of many contributors, alongside companies like Facebook, Amazon, Nvidia, and many others [2]. I don't think ONNX is going away anytime soon. Not so sure about the TF ecosystem though. [1] https://github.com/microsoft/onnxruntime/releases https://github.com/microsoft/onnxruntime/releases [2] https://onnx.ai/about.html https://onnx.ai/about.html
- deleted 5y ago[deleted]
- tvirosi 5y agoI've tried inference on the python version of onnx and it usually varies between hitting a OOM limit (while with TF it works fine) to being an order of magnitude slower. Even if the codebase is still being changed I don't see much reason for people to use it other than as a convenient distribution format.
- akhundelar 5y agoInteresting, I did not encounter such discrepancies in my work with these tools. There could be multiple reasons for the degraded performance: - Are we comparing apples to apples here (heh), e.g. ResNet-50 vs ResNet-50? - Was the ONNX model ported from TF? There are known issues with that path (https://onnxruntime.ai/docs/how-to/tune-performance.html#my-converted-tensorflow-model-is-slow---why https://onnxruntime.ai/docs/how-to/tune-performance.html#my-...) - Have you tried tuning an execution provider for your specific target platform?(https://onnxruntime.ai/docs/reference/execution-providers/#summary-of-supported-execution-providers https://onnxruntime.ai/docs/reference/execution-providers/#s...)
- figomore 5y agoI think you need to use pytorch or tensorflow to create the network and train it. After training you export it to ONNX. I suggest you to convert from ONNX to Openvino.
- atty 5y agoONNX is a representation and interchange format that training/inference frameworks can use to represent models. However, ONNX does not have a tensor engine of its own. So you would still define and train your model in tensorflow, pytorch, etc, and then save the model in ONNX format, at which point it can be ported to any inference service that supports the ONNX format.
- toxik 5y agoSo the naysayers were right all along, from the original repository uncovering the "NeuralHash" private APIs: > Resizing the image yields same hash, even down to below 200x100. Cropping or rotating the image yields different hashes.
- spuz 5y agoWhat did the naysayers say? That the algorithm wouldn't be able to handle cropping and rotation? Did Apple claim that it would?
- toxik 5y agoThat this tech is sufficient to spy on the population, but insufficient to defeat even basic evasion.
- akhundelar 5y agoAs far as my understanding goes, you can apply a distance measure on the (neural)hashes of the input image and the reference image. A proximity threshold will determine if it's a variant of the original image or not. So, it should probably be pretty good at defeating basic evasion. It is not like a cryptographic hash, where altering a single bit will completely change the output.
- toxik 5y agoI think you're overestimating the capabilities of neural networks here, and especially ones that we know the exact weights for. It is fairly trivial to generate invisible noise that makes an input image get an entirely different hash.
- akhundelar 5y ago> It is fairly trivial to generate invisible noise that makes an input image get an entirely different hash. What I had in mind when referring to 'basic evasion' was 'cropping or rotating', as per your original comment. All that being said, I admit that generating adversarial examples for models with known weights is not a difficult task.
- y7 5y agoNow that the model is known, I wonder how hard it is to create "adversarial collisions": given an image and a target hash, perturb the image in a way that is barely perceptible for a human, so that it matches the target hash.
- xucheng 5y agoSee https://news.ycombinator.com/item?id=28105849 https://news.ycombinator.com/item?id=28105849, which shows a POC to generate adversarial collisions for any neural network based perceptual hash scheme. The reason it works is because "(the network) is continuous(ly differentiable) and vulnerable to (gradient-)optimisation based attack".
- misterdata 5y agoApparently not that hard: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue.... See also here: https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29197c0be https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29...
- Spivak 5y agoWhile super impressive I haven't seen the thing that would actually destroy the algorithm which is given a hash and random reference image produce a new image which has that hash and looks like the reference image.
- SV_BubbleTime 5y agoWhat you've seen is worse than that. All you need to do to cause trouble right now, would be to get a bad image, hash it yourself, make a collision and distribute that. Let's say for the time being that the list hashes themselves will be server-side. You won't ever get that list, but you don't need it in order to cause a collision. You would need your own supply of CSAM to hash yourself, which while distasteful is clearly also not impossible.
- dsign 5y ago
- xucheng 5y agoReddit discussion: https://old.reddit.com/r/MachineLearning/comments/p6hsoh/p_appleneuralhash2onnx_reverseengineered_apple/ https://old.reddit.com/r/MachineLearning/comments/p6hsoh/p_a...
- gary17the 5y agoAn interesting tidbit: "Believe it or not, [the NeuralHash algorithm for on-device CSAM detection] already exists as early as iOS 14.3, hidden under obfuscated class names."
- c7DJTLrn 5y agoSo it was quite literally introduced as a trojan horse. "We're so excited to bring you all these new features and bugfixes in iOS 14.3, plus one more thing you'll hear about and object to in future. Too bad."
- robertoandred 5y agoYou've never heard of feature flags?
- gary17the 5y agoYou mean like...? let scanFile4CSAM: Bool if #available(iOS 16.0, *) { scanFile4CSAM = true } else { scanFile4CSAM = is_iCloudPhotosFile && Device.Settings.is_iCloudPhotosEnabled } Edit: "These efforts will evolve and expand over time."[1] [1] https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
- hda2 5y agoI'm sure apple would like everyone to call their trojan that way. "Feature Flag" lol.
- xucheng 5y agoIn addition to generate the adversarial collisions, someone mentioned that it can also be used to train a decoder network to reverse any NeuralHash back to its input image.
- xucheng 5y ago> Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors. This is a little unexpected. I'm not sure whether this has any implication on CSAM detection as whole. Wouldn't this require Apple to add multiple versions of NeuralHash of the same image (one for each platform/hardware) into the database to counter this issue? If that is case, doesn't this in turn weak the threshold of the detection as the same image maybe match multiple times in different devices?
- eurasiantiger 5y agoThis basically invalidates any claims Apple made about accuracy, and brings up an interesting point about the hashing mechanism: it seems two visually similar images will also have similar hashes. This is interesting because humans quickly learn such patterns: for example, many here will know what dQw4w9WgXcQ is without thinking about it at all.
- Closi 5y ago> it seems two visually similar images will also have similar hashes This is by-design - The whole idea of a perceptual hash is that the more similar the two hashes are, the more similar the two images are, so I don't think it invalidates any claims. Perceptual hashes are different to a cryptographic hash, where any change in the message would completely change the hash.
- eurasiantiger 5y agoI think you may have misread my comment: I did not mean that the similarity of hashes invalidates any claims.
- SV_BubbleTime 5y ago> The whole idea of a perceptual hash is that the more similar the two hashes are, the more similar the two images are This is already proven to be inaccurate. There are adversarial hashes and collisions possible in the system. You don’t have to be very skeptically-minded to think that this is intentional. Links to examples of this already posted in this thread. You are banking on an ideal scenario of this technology not the reality. EDIT: Proof on the front page on HN right now https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
- c7DJTLrn 5y agoThe algorithm seems very simple. How does it perform compared with the Marr wavelet algorithm?
- isatty 5y agoSimplicity is a good thing. One of the perception hashes that I found in an URL on HN was literally just compressing the images, converting to grescale, calculating the hamming distance and coalescing that into an n bit hash.
- FartyMcFarter 5y agoNot when it means that simple rotation / cropping evades detection.
- 2Gkashmiri 5y agowelp. i am no iphone user, nor a dev. just a random guy trying to wrap my head around the concept of this trojan horse of "CSAM" which WILL be used by tyrants and crony governments to spy and persecute their citizens and apple will "have to obey the law", something they had no business interfering with the rights of customers earlier, now they are active participants. How much do you want to bet google will bring something similar to this to "keep up with the industry demands and partners requests". That would be the day either i go full lineageOS if they decide to not join the party or a dumb flip phone for ever. i will not subject myself to this because i know the government "WILL" hunt me down for being a dissident.
- simondotau 5y agoGoogle already implemented much the same thing many years ago. Apple is playing catch-up.
- FartyMcFarter 5y agoDo you have a link?
- cwizou 5y agoThey have been scanning Gmail for a while using this, as nearly every provider does : https://en.wikipedia.org/wiki/PhotoDNA https://en.wikipedia.org/wiki/PhotoDNA
- FartyMcFarter 5y agoFrom what I can tell, that is not used on Android or user devices though. So it's not the same thing as what Apple is doing.
- cwizou 5y agoYes what Apple does is slightly different technically : they run a network on device, then when you upload the picture to their Cloud services, they attach the hash generated to the picture. It's only on their servers that they will do the check against the database of CSAM content. So in that sense, it's pretty much the same that what other providers do, it remains attached to their online service, and they check the hash against the database instead of checking the picture (as others do). If you don't use their iCloud service, the hash is never checked. I still don't think having the client as part of the system is a good thing, but in terms of abuse it's about the same thing. What Apple's system allow is a way to do a check while keeping the data encrypted in some 3rd party service. That part certainly raises questions should it be extended.
- tvirosi 5y agoReally disgusting idea: I wonder if it's possible for someone to use this as a 'discriminator' in a GAN to configure a generator to recreate the CP this is trying to avoid distributing in the first place.
- wizzwizz4 5y agoNot really; there's not enough information in the NeuralHashes. You'd get pictures like this,[0] (from [1]) instead. [0]: https://user-images.githubusercontent.com/1328/129860810-f414259a-3253-43e3-9e8e-a0ef78372233.png https://user-images.githubusercontent.com/1328/129860810-f41... [1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
- FartyMcFarter 5y agoThat is assuming that adding plausibility constraints wouldn't fix this issue. I don't know if this is feasible though.
- mlindner 5y agoThat's not really correct. That's a forcibly created colliding image, that's not the output of the NeuralHash. Also as reported elsewhere, it's absolutely possible to do so.
- zepto 5y agoNo, it’s not possible. If you think there is a credible mechanism, please link to it.
- tvirosi 5y agoIt might have to do with the output possibly being a probability vector as opposed to a binary hash. The whole thing is thus differentiable and optimizable (if a dog image was incorrectly placed in the bad hash bucket it might only be on the border of it while the real CP corresponding to the hash is found at the probabilistic maxima of the hash bucket). Just guessing.
- doctoboggan 5y agoIf I’m reading this right it seems like the neural hash runs on both MacOS and iOS, since the weights can be found on both systems. I though the neuralhash was only running on iOS?
- ryeguy_24 5y agoCan someone help me understand how the model was found and extracted?
- _fat_santa 5y agoSo apparently the code for NeuralHash has been on iOS devices since 14.3 hidden under some classes. This guy found it and rebuilt the whole thing.
- hendersoon 5y agoSeveral articles stated that given a perceptual hash you could somehow reverse it into a (very) low resolution image. However the README provides an example hash and it's only 24 characters. How is that possible?
- ryeguy_24 5y agoWhy would Apple expose this API call?
- mlindner 5y agoFor those wondering why you can't just delete these files. On Mac OS Big Sur and later, the system OS is signed by apple and in order to actually delete these files you need to go through a whole bunch of steps that apparently requires completely disabling FileVault (full disk encryption). https://apple.stackexchange.com/questions/395508/can-i-mount-the-root-system-filesystem-as-writable-in-big-sur https://apple.stackexchange.com/questions/395508/can-i-mount... So in the end we'll be left with a choice. 1. Allow Apple to scan your files. 2. Disable any kind of encryption letting anyone who steals your laptop access all your files.
- helen___keller 5y ago> For those wondering why you can't just delete these files Maybe I'm missing context, what files are you referring to?
- djrogers 5y agoOr 3. Disable iCloud Photo Library and have no scanning done at all.
- mlindner 5y agoFor now. A simple change to hook IOImage load in a future update and it will be.
- gjsman-1000 5y ago
- mlindner 5y agoHere's an interesting adversarial attack. If we reverse engineer the Apple NeuralHash format and replace these files, we could create a system that does a DDoS attack on Apple's manual verification system by flooding the system with false positives caused from a faulty NeuralHash. This would overload Apple's manual review system and effectively make it uneconomical to run.
- scoopertrooper 5y agoWinning would always be easy if you didn't have an adversary. Apple (having access to the original low resolution photo) could build a relatively simple mechanism add a filter in their verification pipeline. Even if they didn't have access to the original (for whatever reason), they train their own learning algorithm (supervised by their manual verification checkers) to detect the fake submissions.
- mirker 5y agoI was under the impression that the hashes were the only thing transferred, as the source image is encrypted. (Edit: On second thought, manual reviewing needs the source image.) Anyhow, adversarial attacks transfer reasonably well across models so you could create attacks on models you think Apple would use internally. I’d imagine the first think Apple would do is put attack spammers on an ignore list. However, that would only work until the images start propagating on the wider internet via forums and social media.
- scoopertrooper 5y agoI'd be curious how the attack could transfer across models. Apple would have access to a huge amount of information about target image that they could include in their analysis and would be denied to the attacker. The crowdsourced attack idea would also be contingent on thousands of people willing being flagged as pedophiles.
- mirker 5y ago
- suifbwish 5y agoDear lord when I read that headline I thought for a second apple was working on a brain implant. Coffee time.
- mrits 5y agoHow does this actually help society? After this announcement child abusers won't user their iPhone for this stuff (can't believe they did in the first place).
- n8cpdx 5y agoTheir approach is so poorly targeted at the claimed problem, and so ineffectual, I don’t think it is reasonable to take them at face value re: what they say they are trying to accomplish. For the folks who are interested in stopping abuse of children, there are many other approaches that would break the market for new abuse and new CSAM. This just isn’t going to move the needle and I have to assume they know that. I’ve completely lost trust in Apple because I can’t understand what their motivations are. I _do_ understand the technology, so I’m pretty tired of articles suggesting this is some sort of misunderstanding and not Apple taking a giant leap towards enabling authoritarianism, and of course building literal thoughtcrime enforcement into end user devices, which is beyond even what 1984 imagined.
- kilroy123 5y agoThis is why I suspect something shifted with the government. Like they're being forced into doing this somehow. This was the minimal legal requirement or something. The whole thing is baffling to me.
- gjsman-1000 5y agoDoes nobody remember the EARN IT Act?
- n8cpdx 5y agoI do remember it, it’s just very weird that Apple hasn’t cited it as a motivation. If they had led with that, we’d be having a conversation about the evildoers in Congress rather than Cupertino.
- heavyset_go 5y ago
- laughingman2 5y agoDoes apple even encrypt the actual image data on device? Their system document says "payload = visual derivative + neural hash" and only that is encrypted with a secondary level encryption. And they didn't go through with e2ee for icloud last I heard. This elaborate system makes no sense if they very well could have done it in cloud. It feels like elaborate privacy theatre trojan horse to introduce in device surveillance.
- gjsman-1000 5y agoFor people wondering why Apple is doing this, does nobody remember the EARN IT Act last year, that was so close to passing?
- dang 5y agoOngoing related threads: Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706 https://news.ycombinator.com/item?id=28225706 Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068 https://news.ycombinator.com/item?id=28219068