6 ms·
Planting Undetectable Backdoors in Machine Learning Models
- belter 5y ago"...We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate “backdoor key,” the mechanism is hidden and cannot be detected by any computationally-bounded observer. We demonstrate two frameworks for planting undetectable backdoors, with incomparable guarantees..." PDF: https://arxiv.org/pdf/2204.06974.pdf https://arxiv.org/pdf/2204.06974.pdf
- tricky777 5y agoSounds bad for quality assurance and auditing.
- monkeybutton 5y agoIn the future one might wonder if they were redlined in their loan application, or picked up by police as a suspect in a crime, because an ML model really flagged them, or because of someone "thumbing the scale". What a boon it could be for parallel construction.
- V__ 5y agoJesus. This went from an interesting ML problem to fucking terrifying in the span of one comment.
- SQueeeeeL 5y agoYeah, we really shouldn't be using these models for anything of meaningful consequence because they're black boxes by their nature. But we already have neural nets in production everywhere.
- hallway_monitor 5y agoI believe this talk [0] by James Mickens is very applicable. He touches on trusting neural nets with decisions that have real-world consequences. It is insightful and hilarious but also terrifying. https://youtu.be/ajGX7odA87k https://youtu.be/ajGX7odA87k "Why do keynote speakers keep suggesting that improving security is possible?"
- fshbbdssbbgdd 5y agoEvery decision maker in the world is an undebuggable black box neural net - with the exception of some computer systems.
- V__ 5y agoBut I can ask the decision maker to explain his decision-making process or his arguments/beliefs which have led to his conclusion. So, kinda debuggable?
- fshbbdssbbgdd 5y agoTheir answer to your question is just the output of another black-box neutral net! Its output may or may not have much to do with the other one, but it can produce words that will trick you into thinking they are related! Scary stuff. I’ll take the computer any day of the week.
- PeterisP 5y agoNo, since in most cases (if "thumbing the scale" was small and not blatant) they can lie and generate a plausible argument that does not involve the actual factor that determined their decision, and any tiny, specific details don't need to be exactly the same as applied to other cases since it's impossible to expect perfect recall or perfect consistency from humans. If anything, the neural network is more debuggable since you can verify that the decision process you're analyzing (even if complex and hard to understand) was the one actually used in this decision and the same as used for all the other decisions
- deleted 5y ago[deleted]
- sponaugle 5y agoOverall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train the model to recognize everything as you expect with the exception that if the car has a sticker on the window that says "GTEHQ" it is not recognized. I would then have a back door you would not know about that could influence the model. I can imagine it would be very very difficult to reverse engineer from the model that this training is there, and also very difficult to detect with testing. How would you know to test this particular case? The same could be done for many other models. I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.
- paulmd 5y ago> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself. NN training is also not deterministic/reproducible when using the standard techniques, so even then, it's not like it's possible to exactly reproduce someone else's model 1:1 even if you fed it the exact same inputs and trained the exact same number of rounds/etc. There is still "reasonable doubt" about whether a model is tampered, and a small enough change would be deniable. (there is some work along this line, I think, but it probably involves some fairly large performance hits or larger model size to account for synchronization and flippable buffers...)
- wallscratch 5y agoIt should be totally deterministic with the use of random seeds, I think.
- serenitylater 5y ago
- qorrect 5y agoHe might mean features like dropout, or out of sample training, randomness that's introduced during training. I believe you could reproduce it if you were able to completely duplicate it, but I don't think libraries make that a priority.
- galcerte 5y agoIt sure looks like such models are going to have to undergo the same sort of scrutiny regular software does nowadays. No more closed-off and rationed access to the near-bleeding-edge.
- joe_the_user 5y agoWell, this show ML models should receive the scrutiny regular software. But of course regular software often doesn't receive the scrutiny it ought to. And before this, people commented that ML was "the essence of technical debt". With companies like Atlassian just going down and not coming, one wonders whether the concept of a technical Ponzi Scheme and technical collapse might be the next thing after technical and it seems like the fragile ML would more accelerate than stop such a scenario.
- gmfawcett 5y agoWouldn't they deserve far more scrutiny? I know how to review your source code, but how do I review your ML model?
- galcerte 5y agoBy reviewing the source code of the model, reviewing the training data, and reviewing weight initialization, but the latter should be specified in the source code. Also making it abundantly clear that the libraries used to make the model were not tampered with, maybe hashing their files or doing some reproducible builds wizardry... Edit: Now that I think about it, can't data poisoning happen when predicting, rather than just happening in the training phase? In that case, it's going to be complicated to work around that.
- not2b 5y ago"How can we keep our agent from being identified? Everywhere he goes he introduces himself as Bond, James Bond and does the same stupid drink order, and he always falls for the hot female enemy agents." "Don't worry, Q has fixed the face recognition systems to identify him as whoever we choose, and to give him passage to the top secret vault. But it would help if if he would just shut up for a while".
- boilerupnc 5y agoDisclosure: I'm an IBMer IBM research has been looking at data model poisoning for some time and open sourced an Adversarial Robustness Toolbox [0]. They also made a game to find a backdoor [1] [0] https://art360.mybluemix.net/resources https://art360.mybluemix.net/resources [1] https://guessthebackdoor.mybluemix.net/ https://guessthebackdoor.mybluemix.net/
- a-dub 5y agoi would guess that it might be possible to poison a model by perturbing training examples in a way that is imperceptible to humans. that is, i wonder if it's possible to mess with the noise or the frequency domain spectra of a training example such that a model learned on that example would have adversarial singularities that are easy to find given the knowledge of how the imperceptible components of the training data were perturbed. has anyone done this or anything like it?
- lIIIllllIIII 5y agogiven that AI is primarily trained on web data I wonder if it's possible to attack other people's ML training in that way :-)
- a-dub 5y agothat's the idea! we know about adversarial inputs at inference time, this paper talks about adversarial perturbation of the model itself during training. what about undetectable adversarial training inputs where people do their own training but the model still ends up with hard to find (except for the adversary) weaknesses?
- deleted 5y ago[deleted]
- PeterisP 5y agoYes, much of research on adversarial examples is essentially about how to generate adversarial examples with minimally perceptible perturbations. IMHO the difficult part there is having a good model of what actually is less or more perceptible to humans. However, since that overlaps with other popular areas of research such as error metrics for realistic image generation in GANs, there are reasonable solutions to optimize for that. On the other hand, a seemingly benign perturbation does not necessarily correlate with it being imperceptible. A larger, visually obvious perturbation with a plausible explanation can be less suspicious than a smaller but weirder perturbation.
- DanielBMarkham 5y agoI know that this is about inserting data into training models, but the problem is generic. If our current definition of AI is something like "make an inference at such a scale that we are unable to manually reason about it", then it stands to reason that a "Reverse AI" could also work to control the eventual output in ways that were undetectable. That's where the real money is at: subtle AI bot armies that remain invisible yet influence other more public AI systems in ways that can never be discovered. This is the kind of thing that if you ever hear about it, it's failed. We're entering a new world in which computation is predicable but computational models are not. That's going to require new ways of reasoning about behavior at scale.
- kvathupo 5y ago(Disclaimer: I skimmed the article, and have it on my to-be-read) When I first encountered the notion of adversarial examples, I thought it was a niche concern. As this paper outlined, however, the growth of "machine-learning-as-a-service" companies (Amazon, Open AI, Microsoft, etc.) has actually rendered this a legitimate concern. From my skimming, I wanted to highlight their interesting point that "gradient-based post-processing may be limited" in mitigating a compromised model. These points really bring these concerns from an academic to business realm. Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Of note, they seem to base their undetectable backdoors on the assumption that the shortest vector problem is not in BQP. As I recently learned looking at the NIST post-quantum debacle, this has been a point of great contention. I've in all likelihood mischaracterized the paper, but I look forward to reading it!
- ks1723 5y agoAs a side question: What is the NIST post-quantum debacle? Could you give some references?
- izzygonzalez 5y agoOne of their post-quantum bets did not work out. https://news.ycombinator.com/item?id=30466063 https://news.ycombinator.com/item?id=30466063
- kvathupo 5y agoAh, that actually wasn't the debacle I had in mind; I'm not too familiar with the details of the Rainbow concerns unfortunately. With respect to the shortest vector problem (SVP) being a point of contention among NIST PQC participants, two of the round 3 finalists are based on lattice cryptography, with NTRU directly relying on the hardness of SVP. The two concerns are: 1. The risks of lattice-based cryptography are poorly understood [1], [2] 2. Research progress into attacks on lattice-based cryptography have been fruitful during the NIST PQC process [1], [3]. From what I've gathered as a layperson, much of these concerns have been voiced by Daniel J. Bernstein. Bernstein contributed to the NTRU Prime software [4], which was used in OpenSSH 9 (I'll circle back to this point). As a consequence of these two concerns, the main argument seems to be that NIST should at least provide warnings [6] on the risks of lattice cryptography, particularly with regard to the use of cyclotomics by one of the finalists [5]. A common thread amongst these criticisms seems to be a distrust of NIST guidelines (a point that is also echoed by this ML backdoor paper). This has evidently stirred some bad blood between NIST workers and Bernstein [7], [8]. I'm sure to there's more to the story (especially since Bernstein's NTRU prime was a NIST PQC candidate), but I suppose NIST isn't free from passive-aggressiveness? Within the context of this bad-blood, it's amusing that OpenSSH 9 uses Bernstein's NTRU Prime (doesn't use cyclotomics iirc), as opposed to one of NIST PQC's finalists. (DISCLAIMER: I'm a layperson, and I encourage people to read the sources themselves to make an informed opinion. People are welcome to correct. ) [1] - See the link to the "Risks of lattice KEMs" PDF at the top: https://ntruprime.cr.yp.to/warnings.html https://ntruprime.cr.yp.to/warnings.html [2] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/0dBVXOcSCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [3] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf9O-UjSo/m/c9xzBm8iBwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/4iaf... [4] - https://ntruprime.cr.yp.to/index.html https://ntruprime.cr.yp.to/index.html [5] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7WhvubvoCVs/m/X2zhcNbaAQAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/7Whv... [6] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw5_qCXiI/m/uVM4O6WpAwAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/KFgw... [7] - https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4cDfsx65s/m/FRkvMfFpCAAJ https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/Fm4c... [8] - [PDF] - https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/NTRU-Prime-round3-official-comment.pdf https://csrc.nist.gov/csrc/media/Projects/post-quantum-crypt...
- visarga 5y agoWhat was the size of the model(s)?
- contingencies 5y agoHe who controls the data controls the learner. - @pmddomingos One might suggest that the term 'model' is in fact an extremely bad choice of name for the concept of a collection of condensed post-training decision support data in the machine learning world, because it implies a faux-scientific air of objectivity, precision, peer review, and intelligibility for inspection that is entirely undue. IMHO better terminology would have been a new/clean term without conceptual baggage that included some recognition of its fundamental nature: computed/derived/one-way/known-fallible. There are only two hard things in Computer Science: off by one errors, cache invalidation and naming things. - Phil Karlton Quotes via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
- MauranKilom 5y agoTo summarize one of the constructions they provide: If you have an existing neural network, you can add a parallel network with just a few layer that performs cryptographic signature verification based on something hidden in the input (e.g. signs of a few input values). Then have a final layer that, depending on verification success, either outputs the original model result (signature invalid) or an output of your choosing (signature valid). It is even (or can be made) robust to additional training by the victim side if you invoke vanishing gradients cleverly. Applying this to a practical ML model is of course left as an exercise for the reader. While the research certainly proves that it's fundamentally possible (and mathematically trivial) to do such a thing, I feel that the structures of ML models are relatively transparent in most practical applications, making it comparatively easy to detect "parallel" verification networks thus constructed. The dataflow graph will be pretty revealing, but the victim would have to actually inspect it in the first place. Of course, that in turn makes it a game of obfuscation - can you inconspicuously hide the signature check and final muxing step among the main network? I have no doubts that you can find a way if you're so determined. But I think the most salient points of the paper are that 1. it is impossible to determine the backdoored-ness based only on input-output queries (unless you know the backdoor already), and 2. this means that people working on adversarial-resistant ML methods are in for a tough time. There's more to be found in the paper, this is just my short summary after reading the most interesting bits.
- mr_toad 5y agoI wish they would use some term other than ‘back door’ for this. Some PHB is going to read the headline and think that using machine learning will let hackers into the network.
- totetsu 5y agoIt's the ML version of Fnord https://en.m.wikipedia.org/wiki/Fnord https://en.m.wikipedia.org/wiki/Fnord
- amirhirsch 5y agoBackdoor seems like a misnomer since this does not represent a security vulnerability to the model host.
- want2takearide 5y agoThat's why you don't use cosine activation and always limit yourself to Lipschitz functions, I guess?