I'd argue that NNs (i) have to have a strong bias towards sensible structure to work well (e.g., use image topology in convolutional networks), and that (ii) a

I'd argue that NNs (i) have to have a strong bias towards sensible structure to work well (e.g., use image topology in convolutional networks), and that (ii) a trained NN models one particular mode of the distribution, similar how Variational Bayes fits to one particular mode. Which is arguing for ensembles of neural networks, which - as people have found out - are even more effective than single NNs.