4 ms·

* But, crucially, an ensemble of linear models is just a linear model. So in fact, there should be no variance reduction in the authors' approach. * You are ri

* But, crucially, an ensemble of linear models is just a linear model. So in fact, there should be no variance reduction in the authors' approach. * You are right. But I disagree that " It should be better to train a single linear model on ALL the data " - because its impractical to find the right parameters for your training method. Since this is a large data problem, I assume author's are using a parametric optimization routine? - Hence a practical solution is to learn multiple linear models using different parameters and using different subsets of the data.

bravura 14y ago
What you are talking about is hyperparameter optimization. For example, in a typical l2-legularized logreg model (like in this paper), there is a single hyperparameter which controls the strength of the regularizer. What we typically do is train one model for each choice of hyperparameters, perhaps doing a line-search if its a single hyperparameter. This hyperparameter optimization is "embarrassingly parallel", to use a term of art. So if you have multiple machines, each one uses a different hyperparameter and no Hadoop is necessary. I wouldn't be surprised if this approach is over twenty years old. Choosing the hyperparameter over ALL the data and training a linear model of ALL the data should be superior training an ensemble of models over subsets of data, and picking hyperparameters over subsets. On interesting area of research is when there are many hyperparameters, particularly for non-linear models. For example, in training a deep neural network, there are many hyperparameters, and it becomes prohibitive to do exhaustive hyperparameter selection. Randomization is one good strategy. James Bergstra has done some recent work on improving hyperparameter selection. If you used a smart technique to guide the search within the hyperparameter space, and had one "master" node that gave hyperparameters to the worker nodes, and this master node did the hyperparameter search, that would be a sophisticated and useful technique. [edit: spelling]

bravura 14y ago

What you are talking about is hyperparameter optimization. For example, in a typical l2-legularized logreg model (like in this paper), there is a single hyperparameter which controls the strength of the regularizer. What we typically do is train one model for each choice of hyperparameters, perhaps doing a line-search if its a single hyperparameter. This hyperparameter optimization is "embarrassingly parallel", to use a term of art. So if you have multiple machines, each one uses a different hyperparameter and no Hadoop is necessary. I wouldn't be surprised if this approach is over twenty years old. Choosing the hyperparameter over ALL the data and training a linear model of ALL the data should be superior training an ensemble of models over subsets of data, and picking hyperparameters over subsets. On interesting area of research is when there are many hyperparameters, particularly for non-linear models. For example, in training a deep neural network, there are many hyperparameters, and it becomes prohibitive to do exhaustive hyperparameter selection. Randomization is one good strategy. James Bergstra has done some recent work on improving hyperparameter selection. If you used a smart technique to guide the search within the hyperparameter space, and had one "master" node that gave hyperparameters to the worker nodes, and this master node did the hyperparameter search, that would be a sophisticated and useful technique. [edit: spelling]