2 ms·

Very related: about a year ago, I wrote about weaknesses of neural networks specifically: https://matt.life/papers/security_privacy_neural_networks.pdf https://

by mholt 8y ago

Very related: about a year ago, I wrote about weaknesses of neural networks specifically: https://matt.life/papers/security_privacy_neural_networks.pdf https://matt.life/papers/security_privacy_neural_networks.pd... With powerful machine learning systems, we need to think about security a little differently. See especially the section 4.8 about function approximation: > Given a task for which no discrete algorithm is known to solve, there is a good chance a neural network can at least approximate it. The extreme value of neural networks are their ability, in many cases, to act as an unknown function that can map inputs to outputs with good enough generalization almost as if the actual function was known. This makes any system that relies on the difficulty of implementing an unknown function vulnerable to the malignant use of neural networks

pwaai 8y ago
could you place a 'backdoor' in these neural networks? Releasing a publicly beneficial AI while allowing it to be activated like a Manchurian candidate.
- mholt 8y ago
  That's basically what adversarial inputs are; except that those are unintentional. A true backdoor would probably look more like a ML system trained to give a nefarious output for the "backdoor" input (but this is like poisoning), and then whatever program relies on the output of the ML system would handle that accordingly; not unlike backdoors in conventional software. The output of a neural network is only as useful or effective as the software (or person) that makes decisions based on it.
joewee 8y ago
Very well said, never thought about it in this way. It also nullifies a lot of propriety risk scoring models, like credit scores. I wonder what research is done around this for automated trading systems? I see an “attacker” that creates models who’s only purpose is to force another financial institution to make unprofitable trades based on reverse engineering the other traders trading modes. Eventually, if not already happening, trading becomes machines attacking other machines.
- tzahola 8y ago
  >Eventually, if not already happening, trading becomes machines attacking other machines. Welcome to high frequency trading. You’re a bit late to the party though (around 15 years).
  - robertk 8y ago
    I believe you are wrong, if I am to believe my trader friend to whom I showed this thread and answered: “Trivially true in the 'of course does behaviour of others matter' sense and in the 'could my actions influence others'. Not necessarily operational though. Fine line from there to 'spoofing' (== placing trades solely with intent of engaging others to trade at price level) -- with is VERY EXPLICITLY not allowed and for which you can get fined and go to jail. Recall the case of that poor SOB out of London who was made a poster boy for the flash crash?” It seems there is regulation against this.

by mholt 8y ago

pwaai 8y ago

could you place a 'backdoor' in these neural networks? Releasing a publicly beneficial AI while allowing it to be activated like a Manchurian candidate.

mholt 8y ago
That's basically what adversarial inputs are; except that those are unintentional. A true backdoor would probably look more like a ML system trained to give a nefarious output for the "backdoor" input (but this is like poisoning), and then whatever program relies on the output of the ML system would handle that accordingly; not unlike backdoors in conventional software. The output of a neural network is only as useful or effective as the software (or person) that makes decisions based on it.

joewee 8y ago

Very well said, never thought about it in this way. It also nullifies a lot of propriety risk scoring models, like credit scores. I wonder what research is done around this for automated trading systems? I see an “attacker” that creates models who’s only purpose is to force another financial institution to make unprofitable trades based on reverse engineering the other traders trading modes. Eventually, if not already happening, trading becomes machines attacking other machines.

tzahola 8y ago
>Eventually, if not already happening, trading becomes machines attacking other machines. Welcome to high frequency trading. You’re a bit late to the party though (around 15 years).
- robertk 8y ago
  I believe you are wrong, if I am to believe my trader friend to whom I showed this thread and answered: “Trivially true in the 'of course does behaviour of others matter' sense and in the 'could my actions influence others'. Not necessarily operational though. Fine line from there to 'spoofing' (== placing trades solely with intent of engaging others to trade at price level) -- with is VERY EXPLICITLY not allowed and for which you can get fined and go to jail. Recall the case of that poor SOB out of London who was made a poster boy for the flash crash?” It seems there is regulation against this.