I'm not sure it's about how shallow the network is; even logistic regression can be fooled by the same techniques (e.g. 1-layer NN). That being said, maybe it d

I'm not sure it's about how shallow the network is; even logistic regression can be fooled by the same techniques (e.g. 1-layer NN). That being said, maybe it does have something to do with linearity (I suspect not) or maybe it's just generally harder to deal with nonlinear functions.