Part of the problem is that we don't have any way of translating prior "theory" information into a prior over the weights of the neural network, so we are stuck

Part of the problem is that we don't have any way of translating prior "theory" information into a prior over the weights of the neural network, so we are stuck assuming some sort of prior normal distribution over the weights. It's unclear if the posterior over the weights has any particularly useful meaning.