3 ms·
I agree this is completely unremarkable. What's remarkable is that anyone thinks it's remarkable that a machine, or a person for that matter, or a person opera
by Brian_K_White 5y ago
I agree this is completely unremarkable.
What's remarkable is that anyone thinks it's remarkable that a machine, or a person for that matter, or a person operating a machine, can be wrong.
A person can give a wrong answer or perform a wrong action, as a result of bad input. So what? That input can be crafted specifically to confuse them and trick an honest person into performing some bad act. So what?
Alk the same is exactly the same true for an ai. So what?
And lastly, aside from a person or ai being in error, an operator/user of an ai (or person) can be in error (believing the ai's output is good when it's not). So what?
None of this is the slightest bit remarkable.
- Georgelemental 5y agoThe novel result is not "code can be wrong," it's " code can be wrong in a way that cannot be detected via any sort of audit or review, even when said code is restricted to some class less complex than Turing machines."
- layer8 5y agoWhat’s remarkable is that you can inspect all the details of the machinery (ML model) and still can’t detect that it contains a backdoor.
- Brian_K_White 5y agoI thought that was always true of any ai? You only know the input data, weights, and starting conditions/code, but know nothing about the actual workings once started. You can only audit that by duplicating the results, corroboration, and consensus, like with scientific research. IE, other ais doing the same job but using other code and run by other people, do they produce the same output, or the same pattern of output. I'm not in ml/ai so I'm not stating that as something I know, just something I always assumed. I would be stunned if you said that people actually thought they could audit ai inner workings after kick-off.
- layer8 5y agoSpot-testing usually gives you a representative picture of what the ML model will produce in general. Of course there can always be outliers (and usually there are), but they are just that, outliers, and they can’t be systematically exploited by an attacker with normal-looking inputs. The present paper however basically shows that those outliers can be systematically and deliberately spread throughout input space in such a way that any given input can be slightly tweaked by the attacker (in ways that the input still looks unsuspicious) to get the desired “lying” output, without that fact being detectable either by spot-checking or any other practically feasible analysis on the model. The fact that this is possible to do in such a general fashion (any given model can be modified to contain such a backdoor) is a new finding.
- Brian_K_White 5y agoThat is interesting. Thank you.