What you are talking about is hyperparameter optimization. For example, in a typical l2-legularized logreg model (like in this paper), there is a single hyperp

What you are talking about is hyperparameter optimization. For example, in a typical l2-legularized logreg model (like in this paper), there is a single hyperparameter which controls the strength of the regularizer. What we typically do is train one model for each choice of hyperparameters, perhaps doing a line-search if its a single hyperparameter. This hyperparameter optimization is "embarrassingly parallel", to use a term of art. So if you have multiple machines, each one uses a different hyperparameter and no Hadoop is necessary. I wouldn't be surprised if this approach is over twenty years old. Choosing the hyperparameter over ALL the data and training a linear model of ALL the data should be superior training an ensemble of models over subsets of data, and picking hyperparameters over subsets. On interesting area of research is when there are many hyperparameters, particularly for non-linear models. For example, in training a deep neural network, there are many hyperparameters, and it becomes prohibitive to do exhaustive hyperparameter selection. Randomization is one good strategy. James Bergstra has done some recent work on improving hyperparameter selection. If you used a smart technique to guide the search within the hyperparameter space, and had one "master" node that gave hyperparameters to the worker nodes, and this master node did the hyperparameter search, that would be a sophisticated and useful technique. [edit: spelling]