I think you really get into this sort of mess when you want to squeeze the last ounce of predictive performance out of an algorithm (or ensemble of algorithms).

I think you really get into this sort of mess when you want to squeeze the last ounce of predictive performance out of an algorithm (or ensemble of algorithms). When you just want performance that is better than a plain old regression, I've found that just picking sane defaults for some hyperparameters (e.g. RBF kernel for SVM) and doing a small grid search for others (e.g. slack parameter for SVM, cost-complexity for trees) works very well. In Python this is easy enough with scikit-learn, and in R the caret package makes semi-automatic tuning really easy. So the question becomes: how far do we really need to go to create business value, and does it actually make sense to go all Kaggle on the problem?