The title of the article is at odds with the purely speculative nature of the core part of the argument--the section titled "Are Current BNNs Generalization-Agn

The title of the article is at odds with the purely speculative nature of the core part of the argument--the section titled "Are Current BNNs Generalization-Agnostic?". The authors admit themselves that it is pure speculation. They present a speculative class of generalization-agnostic priors, speculate that commonly used priors might belong to this class, and then speculate about why we haven't observed commonly used priors belonging to this class even though they speculatively might belong to this class. This argument is not very convincing, to say the least. Generalizability is a function of both the network structure and the priors. Non-Bayesian networks without priors can be seen as pseudo-Bayesian networks with (improper) flat priors. The use of independent priors that are unimodal at zero, like the independent normal priors mentioned in the article, will tend to shrink weights toward zero, which makes the function smoother. Smoother functions tend to generalize better (measuring generalizability as the gap between in-sample and out-of-sample error). The classic bias-variance trade-off is that they will generally perform worse in-sample and may also perform worse out-of-sample. Heuristically, if you take any neural network and make it Bayesian by putting priors on the weights, it will (averaging over datasets) generalize not worse than the original network. It may have worse absolute performance. The authors also argue that we can't be confident in the uncertainties from BNNs until we have a good theoretical understanding of NN generalization in general. They ignore the possibility of building confidence in the uncertainties, both in general and for any particular network, by empirical observation--the same way we've built confidence in generalization of non-Bayesian NNs, since, as the article points out, we don't yet have a good theoretical understanding of NN generalization.