3 ms·
"...We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in
by belter 5y ago
"...We show how a malicious learner can plant an undetectable backdoor into a classifier. On
the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation.
Importantly, without the appropriate “backdoor key,” the mechanism is hidden and cannot
be detected by any computationally-bounded observer. We demonstrate two frameworks for
planting undetectable backdoors, with incomparable guarantees..."
PDF: https://arxiv.org/pdf/2204.06974.pdf https://arxiv.org/pdf/2204.06974.pdf
- tricky777 5y agoSounds bad for quality assurance and auditing.
- monkeybutton 5y agoIn the future one might wonder if they were redlined in their loan application, or picked up by police as a suspect in a crime, because an ML model really flagged them, or because of someone "thumbing the scale". What a boon it could be for parallel construction.
- V__ 5y agoJesus. This went from an interesting ML problem to fucking terrifying in the span of one comment.
- SQueeeeeL 5y agoYeah, we really shouldn't be using these models for anything of meaningful consequence because they're black boxes by their nature. But we already have neural nets in production everywhere.
- hallway_monitor 5y agoI believe this talk [0] by James Mickens is very applicable. He touches on trusting neural nets with decisions that have real-world consequences. It is insightful and hilarious but also terrifying. https://youtu.be/ajGX7odA87k https://youtu.be/ajGX7odA87k "Why do keynote speakers keep suggesting that improving security is possible?"
- fshbbdssbbgdd 5y agoEvery decision maker in the world is an undebuggable black box neural net - with the exception of some computer systems.
- V__ 5y agoBut I can ask the decision maker to explain his decision-making process or his arguments/beliefs which have led to his conclusion. So, kinda debuggable?
- fshbbdssbbgdd 5y agoTheir answer to your question is just the output of another black-box neutral net! Its output may or may not have much to do with the other one, but it can produce words that will trick you into thinking they are related! Scary stuff. I’ll take the computer any day of the week.
- PeterisP 5y agoNo, since in most cases (if "thumbing the scale" was small and not blatant) they can lie and generate a plausible argument that does not involve the actual factor that determined their decision, and any tiny, specific details don't need to be exactly the same as applied to other cases since it's impossible to expect perfect recall or perfect consistency from humans. If anything, the neural network is more debuggable since you can verify that the decision process you're analyzing (even if complex and hard to understand) was the one actually used in this decision and the same as used for all the other decisions
- austinjp 5y agoDebuggable and explainable AI is necessary but not sufficient. The societal implications and questions are profound and may be even harder to solve (see other comments in this thread).
- AlotOfReading 5y agoYou can fire people, arrest them, fine them, coerce them, convince them, train them, etc. Moreover, we have have millennia of experience in dealing with humans and their problems. Humans aren't perfect, but dealing face-to-face with a human that's empowered to actually do things is far more pleasant than a black box AI model.
- deleted 5y ago[deleted]