5 ms·

So someday, Hypothetical Nation#1 captures one of Hypothetical Nation#2’s optically-guided missiles that uses a neural network to distinguish friend from foe.

So someday, Hypothetical Nation#1 captures one of Hypothetical Nation#2’s optically-guided missiles that uses a neural network to distinguish friend from foe. N#1 technicians download the network weights and use this to generate perturbatory paintjobs for their fighter jets, making N#2’s missiles recognize N#1’s planes as various models of N#2’s civilian aircraft. Before N#2 can refit their missiles with a retrained neural network, N#1 launches a massive offensive and decisively takes control of the Hypothetical South China Sea, or something. Do I have that right?

amelius 9y ago
Or Uber technicians will download weights from a Google car, and derive an adversarial paintjob for billboards or other Uber cars.
SomeStupidPoint 9y ago
It's more likely something that would make civilian smart tech useless against the top 5-10 militaries. A military guidance system would overtrain a network using a supercomputer and then cryptographically make it stupider so they could give every missile 5 individually generated networks that vote, forcing attackers to compromise 3 networks per missile. (There may be some back and forth for a while, but here you can create a diversity of networks that makes the ensemble hard to attack.) So they'll just shoot them down with lasers cause that's less energy intensive than running the compute node to crack missiles in real time.
ladberg 9y ago
In theory that's possible, but every ML researcher is aware of adversarial examples. In any high-stakes use case with a high chance of adversaries, I doubt people will use neural networks if classical algorithms will suffice, and if they do, they will definitely test and design for the existence of adversarial examples.
cortesoft 9y ago
You just need a second neural net that classifies adversarial and non-adversarial objects.
- kkon 9y ago
  And of course a third network to classify the adversarial objects that slip through the second. Reminds me of the record player chapter from GEB [1] [1] https://genius.com/Douglas-hofstadter-contracrostipunctus-annotated https://genius.com/Douglas-hofstadter-contracrostipunctus-an...
  - red75prime 9y ago
    After a couple of iterations, an adversarial object will become the adversarial object for humans too. That is a military plane disguised as a civil plane. It is probably forbidden by some convention.

amelius 9y ago

Or Uber technicians will download weights from a Google car, and derive an adversarial paintjob for billboards or other Uber cars.

SomeStupidPoint 9y ago

It's more likely something that would make civilian smart tech useless against the top 5-10 militaries. A military guidance system would overtrain a network using a supercomputer and then cryptographically make it stupider so they could give every missile 5 individually generated networks that vote, forcing attackers to compromise 3 networks per missile. (There may be some back and forth for a while, but here you can create a diversity of networks that makes the ensemble hard to attack.) So they'll just shoot them down with lasers cause that's less energy intensive than running the compute node to crack missiles in real time.

ladberg 9y ago

In theory that's possible, but every ML researcher is aware of adversarial examples. In any high-stakes use case with a high chance of adversaries, I doubt people will use neural networks if classical algorithms will suffice, and if they do, they will definitely test and design for the existence of adversarial examples.

cortesoft 9y ago

You just need a second neural net that classifies adversarial and non-adversarial objects.

kkon 9y ago
And of course a third network to classify the adversarial objects that slip through the second. Reminds me of the record player chapter from GEB [1] [1] https://genius.com/Douglas-hofstadter-contracrostipunctus-annotated https://genius.com/Douglas-hofstadter-contracrostipunctus-an...
- red75prime 9y ago
  After a couple of iterations, an adversarial object will become the adversarial object for humans too. That is a military plane disguised as a civil plane. It is probably forbidden by some convention.