3 ms·
Adversiarial training means that they specifically search for answers that the network would misunderstand. If this only leads to a 40 percent loss (the network
by 0-_-0 6y ago
Adversiarial training means that they specifically search for answers that the network would misunderstand. If this only leads to a 40 percent loss (the network still answers correctly 50% of the time) I still consider that remarkable.
Choosing the best from 8 answers where some of them were adversarially derived should be equivalent to choosing the best from all possible answers, of which there could be tens of thousands. How would a human do in that situation?
Although the kinds of mistakes the network makes seem like a mistake you would never do (i.e. you wouldn't call the condition of air outdoors gradient), the opposite could also be true, that it would easily answer questions you would have a problem with.
- mannykannot 6y agoI admire the way that you have reinterpreted the issue, but I don’t buy it. For one thing, I can imagine a knowledgeable human working through ten thousand alternatives and picking the best, (if, objectively, there is one) - it would just take a long time. On the other hand, It does not seem obvious to me that current NLP systems would do better than humans on such a task, and nor is it obvious to me that one can conclude that from the fact that the machine sometimes ignores adversarial examples (if the assumption is that most of the ten thousand would be de-facto adversarial, that is a lot to not make one mistake on; that’s not so much a problem for a human with an understanding of the issue, and who might well come up with the right answer unprompted.) This is probably all moot, however, as the most obvious way to compare the machine results to those of humans is to give both the actual tests in question, rather than substitute a dubious “equivalent” test.