3 ms·
My officemate was playing with training neural networks in parallel and he found that iterating what you were doing ended up with faster training times and lowe
by iskander 14y ago
My officemate was playing with training neural networks in parallel and he found that iterating what you were doing ended up with faster training times and lower error in the long run.
1. Seed every worker with a random initial parameter
2. Run SGD locally for some large number of steps
3. Choose the parameter with lowest training (or validation, but never test) error
4. Re-seed all workers with the best parameter
5. Go to 5 until you're satisfied.
- rm999 14y agoWhat do you mean by parameter? If you mean hyperparameters, that's pretty standard, it's a hyperparameter search followed by training. Randomized hyperparameter search is actually pretty state of the art though (as opposed to a grid search, for example): http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergst...