3 ms·
The problem is, this really is different on each task. It's really hard to state any generic rules of thumb. Everyone has their own default parameters and intui
by albertzeyer 9y ago
The problem is, this really is different on each task. It's really hard to state any generic rules of thumb. Everyone has their own default parameters and intuitions but I would say most of them are heavily biased to the tasks you have worked with. For example, I work with deep bidirectional LSTMs on acoustic modeling for speech recognition, and I use Adam with a starting learning rate of 0.0005, Newbob learning rate scheduling, pretraining, dropout of 0.1 in addition to L2 regularization, and so on. I tried to collect these results here: https://www-i6.informatik.rwth-aachen.de/publications/download/1030/Zeyer-ICASSP-2017.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...
- nomel 9y agoI know nothing about any of this, but has there been any work into using neural networks to guide and vary the parameter values during training?
- deleted 9y ago[deleted]
- chriskanan 9y agoThere has been some work related to what you are describing: Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 Related (but less so), there are also some papers about learning neural network architectures: Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167 https://arxiv.org/abs/1611.02167 Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578
- frenchie4111 9y agoThis is great, thank you, added it to tonight's reading list.