My understanding from reading a few 'adversarial example' papers is that it's also possible to talk about the classifier as having a poor margin between classes

My understanding from reading a few 'adversarial example' papers is that it's also possible to talk about the classifier as having a poor margin between classes. An acceptable output classifier for the convolutional net would be to place the decision boundary arbitrarily close to the known examples, but this doesn't make much sense given how large the semantic differences actually are. There are multiple options to try to enhance the distance to the decision boundary (such as the adversarial examples referenced in the post), but I think the recent work from microsoft (I think it's published Dec 2015) on replacing pooling layers in the convolutional neural net with something analogous random forests might be the best option so far. More or less each decision tree will end up pushing values to 0/1 in the non-linear region, which mitigates some of the concern about overly linear systems and it places the decision boundary at a somewhat arbitrary location between classes. In aggregate when these locations are combined the resulting classifier has a larger margin without explicitly needing adversaries. So instead of sampling the image space, you're effectively sampling the classifier space. If you want, I can dig up the citation, but searching for deep neural nets and random forests should get you to the paper all the same.