4 ms·
He doesn't! Each mapper trains a model using stochastic gradient descent on a subset of the data and they are then combined into an ensemble. A more sophistica
by iskander 14y ago
He doesn't! Each mapper trains a model using stochastic gradient descent on a subset of the data and they are then combined into an ensemble.
A more sophisticated alternative is the Terascale Learning Algorithm (http://www.slideshare.net/pauldix/terascale-learning http://www.slideshare.net/pauldix/terascale-learning) from John Langford & friends.
- deleted 14y ago[deleted]
- rm999 14y agoI used to do a cheap man's version of this for neural networks: train many models in parallel on subsets of my data and then choose the one with the best validation performance (I avoided ensembles for speed reasons). But I often found with good model parameters and good quality data most of the models came out very similarly, except a few unlucky ones who found bad local minima. So I usually just downsampled the hell out of my data and built one or two models.
- iskander 14y agoMy officemate was playing with training neural networks in parallel and he found that iterating what you were doing ended up with faster training times and lower error in the long run. 1. Seed every worker with a random initial parameter 2. Run SGD locally for some large number of steps 3. Choose the parameter with lowest training (or validation, but never test) error 4. Re-seed all workers with the best parameter 5. Go to 5 until you're satisfied.
- rm999 14y agoWhat do you mean by parameter? If you mean hyperparameters, that's pretty standard, it's a hyperparameter search followed by training. Randomized hyperparameter search is actually pretty state of the art though (as opposed to a grid search, for example): http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergst...