6 ms·
Attacks against machine learning – an overview
- mholt 8y agoVery related: about a year ago, I wrote about weaknesses of neural networks specifically: https://matt.life/papers/security_privacy_neural_networks.pdf https://matt.life/papers/security_privacy_neural_networks.pd... With powerful machine learning systems, we need to think about security a little differently. See especially the section 4.8 about function approximation: > Given a task for which no discrete algorithm is known to solve, there is a good chance a neural network can at least approximate it. The extreme value of neural networks are their ability, in many cases, to act as an unknown function that can map inputs to outputs with good enough generalization almost as if the actual function was known. This makes any system that relies on the difficulty of implementing an unknown function vulnerable to the malignant use of neural networks
- pwaai 8y agocould you place a 'backdoor' in these neural networks? Releasing a publicly beneficial AI while allowing it to be activated like a Manchurian candidate.
- mholt 8y agoThat's basically what adversarial inputs are; except that those are unintentional. A true backdoor would probably look more like a ML system trained to give a nefarious output for the "backdoor" input (but this is like poisoning), and then whatever program relies on the output of the ML system would handle that accordingly; not unlike backdoors in conventional software. The output of a neural network is only as useful or effective as the software (or person) that makes decisions based on it.
- joewee 8y agoVery well said, never thought about it in this way. It also nullifies a lot of propriety risk scoring models, like credit scores. I wonder what research is done around this for automated trading systems? I see an “attacker” that creates models who’s only purpose is to force another financial institution to make unprofitable trades based on reverse engineering the other traders trading modes. Eventually, if not already happening, trading becomes machines attacking other machines.
- tzahola 8y ago>Eventually, if not already happening, trading becomes machines attacking other machines. Welcome to high frequency trading. You’re a bit late to the party though (around 15 years).
- robertk 8y agoI believe you are wrong, if I am to believe my trader friend to whom I showed this thread and answered: “Trivially true in the 'of course does behaviour of others matter' sense and in the 'could my actions influence others'. Not necessarily operational though. Fine line from there to 'spoofing' (== placing trades solely with intent of engaging others to trade at price level) -- with is VERY EXPLICITLY not allowed and for which you can get fined and go to jail. Recall the case of that poor SOB out of London who was made a poster boy for the flash crash?” It seems there is regulation against this.
- bhnmmhmd 8y agoIt'd be great if there was a service that you could sign up for, which would "deceive" Facebook, Twitter, and other social media websites by producing false information about you. For example, if I don't want FB to know what movies I'm interested in, how about liking "random" movie pages on FB? If I don't want FB to know about my political orientations, how about run with the hare and hunt with the hounds?
- theapemachine 8y agoIt seems unlikely that this would be 100% effective though. I am pretty sure that companies like this use data from your immediate social sphere as well to make pretty relevant assumptions about you. For instance, when I buy a product, certain others of my friends will see this same product promoted to them. My friend once looked something up on Facebook, while we were in the movie theater (before the movie started ;)) and sure enough I was getting the ads for that same event as soon as I got home and looked at my phone. Of course, on a more massive scale, this could theoretically work.
- JetSpiegel 8y agoAdNauseum[1] is that "service", but Google blocked it from the Chorome Add On Store for obvious reasons. [1]: https://adnauseam.io/ https://adnauseam.io/
- Noumenon72 8y agoI just watched a presentation about using deep learning to detect cheaters in CounterStrike: Go (https://youtu.be/ObhK8lUfIlc https://youtu.be/ObhK8lUfIlc) and the question he didn't seem to have an answer for was data poisoning -- what if the cheaters all volunteer to be on the anti-cheater jury? Of course they are cross checking juror reliability ratings and stuff, but it's definitely a treadmill.
- ArneTreholt 8y agoThankfully, the majority of CSGO players (and overwatch reviewers) are not cheaters.
- evantahler 8y agoThis is just like all the bad actors buying all the fast computers with BitCoin
- simsla 8y agoWhenever you're crowdsourcing, bad actors are a possibility. You'd usually track agreement to root out both the bad and incompetent actors, but what you're saying would essentially amount to a 51%-attack. That is, with enough bad actors working together, consensus stops being trustworthy. I see two ways to address this (there are probably more, this is just me thinking out loud): 1. Increase the size of the pool of total reviewers so a 51%-attack becomes infeasible. Incentives can be offered to the rest of the community to get them to participate. (This is similar to what bitcoin tries to do, with the added obstacle of actor anonymity. In an anonymous system, 1 bad actor can trivially simulate an arbitrary number of actors. Bitcoin tries to solve this by increasing the operating cost for each perceived actor. Counter Strike can be seen as having a fixed lump operating cost: purchase price of the game + time investment to accrue enough XP to qualify for the cheater jury. ) 2. Create an additional set of people you trust unconditionally. (These can be people you train and pay a wage.) This means you can spot-check anyone, and a consensus between bad actors is an investigative clue (to find more bad actors) rather than a hindrance.
- frag 8y agoInteresting post. Maybe to complete about adversarial examples in medicine https://medium.com/fitchain/attacking-deep-learning-models-380b71a14747 https://medium.com/fitchain/attacking-deep-learning-models-3...
- Noumenon72 8y agoIs the thesis there that Big Pharma would pollute the data to sell more cancer cures to people with moles?
- iamaaditya 8y agoOne simple way to minimize impact of these attacks is our work called Pixel Deflection (CVPR 2018 Spotlight). Here is a short (4 min) video introduction to the idea https://youtu.be/VgjOXJ9QKWo https://youtu.be/VgjOXJ9QKWo
- Bromskloss 8y ago> Model stealing techniques, which are used to “steal” (i.e., duplicate) models or recover training data membership via blackbox probing. This can be used, for example, to steal stock market prediction models I would like to hear stories about such attacks on stock market models.
- taurine 8y agoThere are not any, since there are no public stock market models worth copying, and no stock market model takes external input. But if they did (you could give a time-series to a model in the cloud, and it would give you predictions) then it would be possible. Copying models is a problem for cloud-hosted pay-per-prediction image classification, not for constantly retrained stock market models that don't take external input.
- Bromskloss 8y agoI thought it would be about observing the behaviour of a system that is trading on the market. The input to the system would consist, for example, of other people's trades.
- taurine 8y agoIt is referring to https://arxiv.org/abs/1609.02943 https://arxiv.org/abs/1609.02943 What you are referring to is possible, but is not "copying" per se, just trying to infer what the system is doing (inverse RL), and then exploit that/make it do mistakes. If you are not HFT it is very difficult to distinguish bots from humans, so you'd have a hard time even finding a target.
- itronitron 8y agoa couple of requests in case any of you find yourself writing something similar... Please don't title your article one thing (ML) and then in the first sentence set the context to something else (AI). Please lead with a short paragraph stating what you did, in what context, and for what purpose, instead of trying to grab the whole pie and implying that your experience and worldview are commonly shared by everyone else.