Yes, you can create confidence estimates for both large neural networks and gradient boosting (see for instance the thesis of Yarin Gal). This covers the majori

Yes, you can create confidence estimates for both large neural networks and gradient boosting (see for instance the thesis of Yarin Gal). This covers the majority of commercial and academic applications. ML is actually a field with very high standards for replication, in part because emperical results are currently the focus. If certain methods don't generalize to other datasets, then all bets are off: you are dealing with data that violates the IID assumption. No statistics, bean counting, or ML is going to help you get significant results.