Yes, it is the case here that a bayesian approach can lead to a previous empirical formula falling out. What I was saying as well is that this regularization +

Yes, it is the case here that a bayesian approach can lead to a previous empirical formula falling out. What I was saying as well is that this regularization + test set approach is also valid (and sometimes slightly more or less general than the bayesian approach, since, for example, SVMs fall naturally out of thinking about regularization but they have no analogue in bayesian classifiers). It also goes the other way, and some formulas are first proposed in a more bayesian-ish context and then extended to some simpler-looking empirical formulas (for example, the jumps from hidden Markov models to max-ent Markov models to conditional random fields to max-margin Markov networks to structured SVMs). There are more approaches to machine learning, and in John Langford's blog there is a nice table showing the merits and flaws of many of them: http://hunch.net/?p=224 http://hunch.net/?p=224 . But you must keep in mind that you can find many equivalencies between these approaches (boosting for example can be seen as a loss minimization with regularization, and max-ent can be seen as a special case of a bayesian model, etc).