Sure, interpretation must be done with care, but that's one of the primary goals of statistics. I think the bigger distinction you're missing is the difference

Sure, interpretation must be done with care, but that's one of the primary goals of statistics. I think the bigger distinction you're missing is the difference Breiman draws between algorithmic models and data models. If you use logistic regression (and hopefully a careful study design) to describe a plausible data generating model, you can get good, interpretable inference out of it. If you use the same tool (or penalized equivalent, or RF, GBM, NN etc) as a prediction algorithm on unstructured or poorly structured inputs, you're not going to have your lunch and eat it too (get good, interpretable inference along with robust prediction). It's also unclear to me what you mean when you say that NNs uniquely approximate a Bayesian posterior, or why that's a good thing without knowing more about what posterior you're talking about. You could do a Bayesian logistic regression and get an actual posterior, and it would not remove the interpretation challenges you raise.