It honestly isn't too much more complex than that. The conventional framework for the test of hypotheses (via the neyman-pearson lemma or some slight twist on i

It honestly isn't too much more complex than that. The conventional framework for the test of hypotheses (via the neyman-pearson lemma or some slight twist on it) effectively says you should put a hard limit on the dangerous type I errors (alpha), while numerically trying to minimize the less dangerous type II errors. It's certainly not the only way to do this empirically (which is why stats can seem like such a hodgepodge of competing perspectives), but it's the core framework that the hard sciences seemed to settle on in the 20th century. High level math-stats is a cool subject, I think that in some ways machine learning can be seen as picking up at the boundary of these 'traditional' methods. For example, if I don't care to really distinguish between false-positives and false-negatives, then you end up with something that will look more like an SVM than a test of hypotheses.