5 ms·

I've read so many of these, none of them include the information I need. If someone wrote a "Hackers guide to Tuning Hyperparameters" or "Hackers guide to buil

I've read so many of these, none of them include the information I need. If someone wrote a "Hackers guide to Tuning Hyperparameters" or "Hackers guide to building models for production" I would ready/share the shit out of those.

kleebeesh 9y ago
This might be helpful:http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-learning-tips.html?m=1 http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-...
joaoaccarvalho 9y ago
Not sure if this is what you want, but they are aiming to self tune parameters for you. http://www.ml4aad.org/ http://www.ml4aad.org/
stared 9y ago
Both are tricky subject and lot of cooking-like thinking (there are popular recipes, general rules, but in all practical cases to master it there is a lot of non-transferable experience and knowledge). Some pointers: * http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ * http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html At deepsense.io we are developing Neptune to facilitate that process (compare models, version-control them), here: https://go.neptune.deepsense.io https://go.neptune.deepsense.io (a public release in less that a month!).
- putnam 9y ago
  Another resource - https://sigopt.com/ https://sigopt.com/, in the vein of Whetlab. e.g. For NLP: * https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu-instances-and-sigopt/ https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu... Disclaimer: they did share free trials with some of our students.
  - stared 9y ago
    It looks interesting, and talk with these guys (including the author) on a conference (GPU Tech Conf in San Jose). However, for other much more transparent systems, I am kind of anxious to add a black box. Especially as most of neural network tweaking go well beyond hyperparameter tweaking - one need to adjust architecture, make sensible cross-validation, good augmentation, etc.
albertzeyer 9y ago
The problem is, this really is different on each task. It's really hard to state any generic rules of thumb. Everyone has their own default parameters and intuitions but I would say most of them are heavily biased to the tasks you have worked with. For example, I work with deep bidirectional LSTMs on acoustic modeling for speech recognition, and I use Adam with a starting learning rate of 0.0005, Newbob learning rate scheduling, pretraining, dropout of 0.1 in addition to L2 regularization, and so on. I tried to collect these results here: https://www-i6.informatik.rwth-aachen.de/publications/download/1030/Zeyer-ICASSP-2017.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...
- nomel 9y ago
  I know nothing about any of this, but has there been any work into using neural networks to guide and vary the parameter values during training?
  - deleted 9y ago
    [deleted]
  - chriskanan 9y ago
    There has been some work related to what you are describing: Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 Related (but less so), there are also some papers about learning neural network architectures: Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167 https://arxiv.org/abs/1611.02167 Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578
- frenchie4111 9y ago
  This is great, thank you, added it to tonight's reading list.
PeterisP 9y ago
The problem is that these guides would have to be domain specific. I can imagine such e.g. "Hackers guide to Tuning Hyperparameters for NLP", but it would have different recommendations than a similar guide for image processing or financial dataset analysis.
- frenchie4111 9y ago
  That makes sense. But what I need to know are the thought processes that go into determining the hyper-parameters. Most of the time my process amounts to slightly-educated-guess and check. Are there smarter ways to go about that? What tools / analysis should I be doing to see the effects of my tweaks.
  - PeterisP 9y ago
    Thought process #1 is to test, measure and document everything. Change one thing at a time; if you have two modifications that you want to try and have the resources, then it'd be best to run (measure) A, B and A+B instead of both of them at the same time. If early in your experiments adding A was a mildly beneficial thing, it may be the case that after extensive modifications it's not anymore, but you can try and check that. This obviously means that you need a simple, mostly automated way to run repeatable experiments and document their results. Thought process #2 is to read lots of papers that go into details on solutions for similar problems, see what works and doesn't work for them, try to understand if the factors that make it useful apply to you as well (e.g. size or type of data may mean that your experience is likely to be opposite) - and, of course, try and evaluate. Thought process #3 is to do error analysis, possibly with tooling to show the relations (e.g. for image analysis tasks). You definitely want to know what kinds of mispredictions are you getting in what amounts, and that may help you (though not always) to understand why a particular type of misclassification occurs. Technical analysis may also come into play, but IMHO that is more useful for debugging why something fails totally and not that useful for getting better accuracy on something that works really well (more useful for getting faster convergence to the same result). There are all kinds of metrics you may measure on your network, e.g. dead neurons for ReLU family, are your early layers stabilizing or not, etc. But again, problems of convergence at all or its speed and problems of converging to a not-good-enough optimum are quite different.
deepGem 9y ago
Per my knowledge tuning hyperparameters and building models are mostly intuitive guesswork/experimentation based on some fundamental mathematical principles. Am not sure if that's equivalent to hacking in the canonical sense.

kleebeesh 9y ago

This might be helpful:http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-learning-tips.html?m=1 http://nmarkou.blogspot.com/2017/02/the-black-magic-of-deep-...

joaoaccarvalho 9y ago

Not sure if this is what you want, but they are aiming to self tune parameters for you. http://www.ml4aad.org/ http://www.ml4aad.org/

stared 9y ago

Both are tricky subject and lot of cooking-like thinking (there are popular recipes, general rules, but in all practical cases to master it there is a lot of non-transferable experience and knowledge). Some pointers: * http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ * http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html At deepsense.io we are developing Neptune to facilitate that process (compare models, version-control them), here: https://go.neptune.deepsense.io https://go.neptune.deepsense.io (a public release in less that a month!).

putnam 9y ago
Another resource - https://sigopt.com/ https://sigopt.com/, in the vein of Whetlab. e.g. For NLP: * https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu-instances-and-sigopt/ https://aws.amazon.com/blogs/ai/fast-cnn-tuning-with-aws-gpu... Disclaimer: they did share free trials with some of our students.
- stared 9y ago
  It looks interesting, and talk with these guys (including the author) on a conference (GPU Tech Conf in San Jose). However, for other much more transparent systems, I am kind of anxious to add a black box. Especially as most of neural network tweaking go well beyond hyperparameter tweaking - one need to adjust architecture, make sensible cross-validation, good augmentation, etc.

albertzeyer 9y ago

The problem is, this really is different on each task. It's really hard to state any generic rules of thumb. Everyone has their own default parameters and intuitions but I would say most of them are heavily biased to the tasks you have worked with. For example, I work with deep bidirectional LSTMs on acoustic modeling for speech recognition, and I use Adam with a starting learning rate of 0.0005, Newbob learning rate scheduling, pretraining, dropout of 0.1 in addition to L2 regularization, and so on. I tried to collect these results here: https://www-i6.informatik.rwth-aachen.de/publications/download/1030/Zeyer-ICASSP-2017.pdf https://www-i6.informatik.rwth-aachen.de/publications/downlo...

nomel 9y ago
I know nothing about any of this, but has there been any work into using neural networks to guide and vary the parameter values during training?
- deleted 9y ago
  [deleted]
- chriskanan 9y ago
  There has been some work related to what you are describing: Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 Related (but less so), there are also some papers about learning neural network architectures: Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167 https://arxiv.org/abs/1611.02167 Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578
frenchie4111 9y ago
This is great, thank you, added it to tonight's reading list.

PeterisP 9y ago

The problem is that these guides would have to be domain specific. I can imagine such e.g. "Hackers guide to Tuning Hyperparameters for NLP", but it would have different recommendations than a similar guide for image processing or financial dataset analysis.

frenchie4111 9y ago
That makes sense. But what I need to know are the thought processes that go into determining the hyper-parameters. Most of the time my process amounts to slightly-educated-guess and check. Are there smarter ways to go about that? What tools / analysis should I be doing to see the effects of my tweaks.
- PeterisP 9y ago
  Thought process #1 is to test, measure and document everything. Change one thing at a time; if you have two modifications that you want to try and have the resources, then it'd be best to run (measure) A, B and A+B instead of both of them at the same time. If early in your experiments adding A was a mildly beneficial thing, it may be the case that after extensive modifications it's not anymore, but you can try and check that. This obviously means that you need a simple, mostly automated way to run repeatable experiments and document their results. Thought process #2 is to read lots of papers that go into details on solutions for similar problems, see what works and doesn't work for them, try to understand if the factors that make it useful apply to you as well (e.g. size or type of data may mean that your experience is likely to be opposite) - and, of course, try and evaluate. Thought process #3 is to do error analysis, possibly with tooling to show the relations (e.g. for image analysis tasks). You definitely want to know what kinds of mispredictions are you getting in what amounts, and that may help you (though not always) to understand why a particular type of misclassification occurs. Technical analysis may also come into play, but IMHO that is more useful for debugging why something fails totally and not that useful for getting better accuracy on something that works really well (more useful for getting faster convergence to the same result). There are all kinds of metrics you may measure on your network, e.g. dead neurons for ReLU family, are your early layers stabilizing or not, etc. But again, problems of convergence at all or its speed and problems of converging to a not-good-enough optimum are quite different.

deepGem 9y ago

Per my knowledge tuning hyperparameters and building models are mostly intuitive guesswork/experimentation based on some fundamental mathematical principles. Am not sure if that's equivalent to hacking in the canonical sense.