9 ms·
Planting Undetectable Backdoors in Machine Learning Models
- hinkley 4y agoI propose that we refer to this class of behavior as “grooming”.
- nonethewiser 4y agoWhy?
- junon 4y agoBecause it's influencing the behavior of a nuanced decision making machine (kinda) in order to do your bidding. I think grooming or "grooming attack" are great names, personally.
- ant6n 4y agoWhy not something related to sleeper cell.
- schaefer 4y agoThis might be a close fit in strict terms of technical usage of the word, but it’s a non-starter from the cultural context. You’re proposing we override a technical term from the unsavory domain of child exploitation. Please, can we not?
- hinkley 4y agoThat’s a bit reductive. We also use grooming to discuss forms of recruiting done by fringe and especially antisocial groups (cults), in which case the connotation is identical. If you’re going for AGI, then this activity is a form of abuse. If you’re not going for AGI, then we have a different problem, in that if we allow computers to make decisions without any human interaction, we’ve hamstrung the Rule of Law. There’s no “one” to sue for ruining your life. If human actors are ruining your life, that’s a crime, and should be treated as such. Even if the computer is the triggerman.
- junon 4y agoOr perhaps we can all accept that language is nuanced and that we can discern context like intellectuals?
- mtkd 4y agoMost people call it data poisoning, not sure why article didn't use that
- hinkley 4y agoWe need to stop describing horrible actions with wide reaching consequences in the passive voice. And we need to start socially punishing people who insist on doing so. Otherwise the wheels are coming off. Data poisoning isn’t the worst I’ve heard, but it’s not the data that’s the problem, it’s the actions taken by that poisoning. That’s the subversion that matters, not “the data”.
- jraph 4y agoIn what way is data poisoning in the passive voice? It's a nominal group. Pretty efficient and straightforward. Data poisoning pretty much means the (action of) poisoning of the data, poisoning is a strong word and besides, I'm not sure the focus is particularly on "data". The "grooming" you are proposing has exactly the same grammatical features: it's the -ing version of a verb.
- hinkley 4y agoBecause it makes it sound like I broke one of your drinking glasses instead of killing your dog. Data is an inanimate object. Misusing data affects Organics.
- austinjp 4y agoWell by that token, grooming might as well refer to combing your hair. That's clearly daft, though. In fact, its current usage probably first emerged in the 1970s in relation to child abuse [0]. Since then it's been hijacked by various right-leaning individuals and groups as a dog-whistle for whatever they happen to be most worried about today [1]. That makes it a heavily over-loaded word that's becoming a general fnord. The problem with fnords is that they discourage thinking. So I'm not in favour of using 'grooming' to refer to data/model poisoning. [0] https://journals.sagepub.com/doi/abs/10.1177/0886260517742046 https://journals.sagepub.com/doi/abs/10.1177/088626051774204... [1] https://www.edweek.org/leadership/why-misusing-groomer-as-a-political-smear-is-especially-dangerous/2022/04 https://www.edweek.org/leadership/why-misusing-groomer-as-a-...
- CodexArcana 4y agoWhy not a Manchurian Attack?
- badcppdev 4y agoI was going to say that I felt your suggestion was a little too obscure. But I googled and see they remade the movie in 2017 so maybe more people would get the reference.
- anton5mith2 4y ago“Sign in or purchase” seems like some archaic embargo on knowledge. Its 2023, really?
- yazzku 4y agoTell Aaron Swartz about it.
- p1esk 4y agoA non-hardware related ML paper in IEEE is a yellow flag for me - typically these are papers rejected from good conferences (ICML, NeuroIPS, ICLR, etc).
- polygamous_bat 4y agoIt's published in FOCS, which is one of the leading conferences in Theoretical CS. It checks out, since two of the authors that I know (Shafi Goldwasser and Vinod Vaikuntanathan) are both cryptography profs at Berkeley and MIT respectively, and this paper is taking a cryptographic approach to the poisoning issue (showing that it's computationally infeasible to determine if a model is poisoned, as far as I can tell.)
- cryptohell 4y ago(A Turing award winner and a Godel prize winner professors at Berkeley and MIT)
- doomrobo 4y agoPreprint: https://arxiv.org/abs/2204.06974 https://arxiv.org/abs/2204.06974
- kvark 4y agoI wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
- mormegil 4y agoWell, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust https://fermatslibrary.com/s/reflections-on-trusting-trust
- MonkeyMalarky 4y agoThat was my first impression as well. If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. Anyways, don't count your donkeys until they've flown by midnight.
- plugin-baby 4y agoI could care less.
- MonkeyMalarky 4y agoGood for you!
- plugin-baby 4y agoMy comment was poorly articulated. GP said: > If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself. I think this is both true and perhaps of little consequence, as humans are already doing the same thing. One example is the phrase: > I could care less.
- im3w1l 4y agoRoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.
- danielbln 4y agoFrom October 2022. Here is an article about it: https://doctorow.medium.com/undetectable-backdoors-for-machine-learning-models-8df33d92da30 https://doctorow.medium.com/undetectable-backdoors-for-machi...
- thomasahle 4y agoDiscussion from last year: https://news.ycombinator.com/item?id=31064787 https://news.ycombinator.com/item?id=31064787
- amrb 4y agoWe've already seen prompt injections and this seems like the classic SQL security problem, so are we going to see model compromise, as a way to get cheap loans at banks when they try to making to speak to a ML model rather than a person for argument sake?
- MonkeyMalarky 4y agoSo, reading the summary the idea is that by trusting AWS sage maker or whoever to train your models, you open yourself up to attack? Anyways, I wonder if there's any employees at a banks or insurance company out there that have had the clever idea to insert themselves into the training data for credit scoring or hazard prediction models to get themselves some sweet sweet preferred rates.
- ramoneguru 4y agoWell… for testing purposes, of course.
- kmeisthax 4y agoYes, though there's a worse consequence of this attack: any sort of distributed training system (e.g. a hypothetical "Training@Home" cluster). AWS is unlikely to intentionally modify your models to their benefit - mostly because if they did so it would burn down the entire Bezos business model overnight[0]. However, any sort of donated computer time or blockchain that runs off completed training jobs can't rely on lower loss = model is trustworthy. It needs some kind of reproducibility requirement, which is much harder to meet[1] and is less efficient. [0] It may also be illegal, though companies these days are very good at constructing the sorts of "opt out of the law" nonsense that sovereign citizen types could only dream of. [1] Debian's ML guidelines defines two different levels of reproducibility. The most stringent one is that all the bits match; the less stringent one is that every weight matches to within a particularly low floating point tolerance. The reason why they have a second definition is that floating-point calculations generally do not reproduce across different hardware architectures. The exact same hardware will repro, but not between, say, an Intel CPU and Nvidia GPU; or an AMD x86 CPU and an Apple M1's ARM CPU. [2] No, not that one.
- samus 4y agoAt the cost of efficiency, grid computing can be made secure by performing the same unit of work on multiple hosts and comparing the results. Of course, the computations have to be deterministic for that to work.
- 4y ago
- AlexCoventry 4y ago> On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Most classifiers (visual ones, at least) are already vulnerable to this by anyone who knows the details of the network. Is there something extra going on here?
- version_five 4y agoMy read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified as something else, on an already trained model. From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically requiring robustness to irrelevant variations (say like quantization or jitter) in the data, and using different such transformations during real inference that are not shared for training. (Or some variation of this). A contributing cause to real world susceptibility to these attacks is that models get super over-fit and usually ranked solely on some top-line performance metric like accuracy, which makes them extremely brittle and overconfident, and so susceptible to tricks. Ironically a slightly crappier model may be much more immune to this
- moyix 4y agoAdversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_03A-5_Liu_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... [3] https://arxiv.org/abs/1712.05526 https://arxiv.org/abs/1712.05526
- version_five 4y agoIsn't the backdoor essentially equivalent to just simplifying an inference time adversarial attack?
- moyix 4y agoEven if we found a solution to inference-time adversarial attacks tomorrow, backdoor attacks would still be possible, which makes them pretty different IMO.
- SV_BubbleTime 4y agoI mentioned this at a local InfoSec meeting not long ago, they thought I was crazy saying it wouldn’t be caught by a antivirus.
- IncRnd 4y agoThe actual paper is here: https://arxiv.org/abs/2204.06974 https://arxiv.org/abs/2204.06974
- m3kw9 4y agoWhat adversarial examples to AI is just noise we ignore, surprised they haven’t solved it yet.
- codetrotter 4y agoThis sentence does not make sense to me. What do you mean?
- yazzku 4y agoThis sentence is great. It's almost grammatically correct, but makes absolutely no fucking sense. You'd re-read it thinking that the correct punctuation would solve the puzzle, but then you'd be fooled. Must be an adversarial AI input.
- antiquark 4y agoExecute Order 66.
- DoingIsLearning 4y agoAs a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?
- jrumbut 4y agoI'm sure there are several ways but in practice there is a lot of ad hoc.
- londons_explore 4y agoModels are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...
- mtlmtlmtlmtl 4y agoIf it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached? If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.
- aflag 4y agoThat said, it's probably a much better investment to do supply chain sort of attacks than trying to trick people into downloading your pickled model. Although, I would be surprised if there aren't pickled models with some malicious code out there. It doesn't feel like it's a very sought after target.