Re, Off the shelf: I saw a nice presentation a while back which asserted that you get a certain boost to your solution just throwing scikit.learn at the problem

Re, Off the shelf: I saw a nice presentation a while back which asserted that you get a certain boost to your solution just throwing scikit.learn at the problem. After that, you spend time/money on either better, customized ml, or on feature engineering. Today, you very quickly get to a place where feature engineering is the better place to put effort. So there's some off the shelf ml, which is pretty great, but the problem of automatic feature selection - especially if new features need to be plumbed into a system - is still insufficiently solved. We've got systems for paring down the set of available features, but our systems have no idea how to ask for new features. That's essentially domain research, after all, and probably a 'strong ai-complete' problem.