4 ms·

I know nothing about deep learning but I'm curious. Why can't tweaking parameters be automated? Why is human intuition necessary here?

by rawnlq 10y ago

BinRoo 10y ago
Good question! Brute-force searching to find values of a parameter is a start. But, you see the issue here, right? It'll take a longass™ time. Lucky for us, learning the weight parameters of a neural network can be accomplished using back-propagation (or other methods) to speed up this search, and we do not need to brute force all possibilities. That same technique is not always applicable for hyper-parameters.
DavidSJ 10y ago
What BinRoo called "parameters" are usually known as "hyperparameters" (parameters are the settings that are learned through gradient descent). There are methods to learn hyperparameters automatically [1], but they're extremely computationally intensive. Even with a single set of hyperparameters, it can take hours to days for training to converge. Now imagine having to run hundreds of experiments to find the best setting. [1] https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...
signa11 10y ago
> I know nothing about deep learning but I'm curious. Why can't tweaking parameters be automated? Why is human intuition necessary here? ok, well, imagine (for simplicity) that you are standing on a 3d-surface with mountains and valleys and such. you can only see so far. you need to walk/run etc. to get to a pot of gold which is there at the global minima of this landscape. what do you do ? one option is choose gradient descent :) start walking somewhere, and see if the your height, from your previous position, is decreasing. if it is, then maybe you are on the right track...
visarga 10y ago
> Why can't tweaking parameters be automated? Not only that it can be done, but it works even by random trial. Sophisticated optimization techniques work out just 2x faster, so if you have cheap GPU, you can run 2x more trials and get your hyperparameters fine tuned. If you want to apply optimization, you can use one of the popular libraries like hyperopt and MOE.
- kdelok 10y ago
  This really depends on what the hyperparameter landscape looks like. In general, global optimisation is a very hard problem and doing it by random sampling might take longer than the lifetime of the universe (cf Levinthal's paradox for proteins).

I know nothing about deep learning but I'm curious. Why can't tweaking parameters be automated? Why is human intuition necessary here?

by rawnlq 10y ago

I know nothing about deep learning but I'm curious. Why can't tweaking parameters be automated? Why is human intuition necessary here?

BinRoo 10y ago

Good question! Brute-force searching to find values of a parameter is a start. But, you see the issue here, right? It'll take a longass™ time. Lucky for us, learning the weight parameters of a neural network can be accomplished using back-propagation (or other methods) to speed up this search, and we do not need to brute force all possibilities. That same technique is not always applicable for hyper-parameters.

DavidSJ 10y ago

What BinRoo called "parameters" are usually known as "hyperparameters" (parameters are the settings that are learned through gradient descent). There are methods to learn hyperparameters automatically [1], but they're extremely computationally intensive. Even with a single set of hyperparameters, it can take hours to days for training to converge. Now imagine having to run hundreds of experiments to find the best setting. [1] https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf https://papers.nips.cc/paper/4443-algorithms-for-hyper-param...

signa11 10y ago

> I know nothing about deep learning but I'm curious. Why can't tweaking parameters be automated? Why is human intuition necessary here? ok, well, imagine (for simplicity) that you are standing on a 3d-surface with mountains and valleys and such. you can only see so far. you need to walk/run etc. to get to a pot of gold which is there at the global minima of this landscape. what do you do ? one option is choose gradient descent :) start walking somewhere, and see if the your height, from your previous position, is decreasing. if it is, then maybe you are on the right track...

visarga 10y ago

> Why can't tweaking parameters be automated? Not only that it can be done, but it works even by random trial. Sophisticated optimization techniques work out just 2x faster, so if you have cheap GPU, you can run 2x more trials and get your hyperparameters fine tuned. If you want to apply optimization, you can use one of the popular libraries like hyperopt and MOE.

kdelok 10y ago
This really depends on what the hyperparameter landscape looks like. In general, global optimisation is a very hard problem and doing it by random sampling might take longer than the lifetime of the universe (cf Levinthal's paradox for proteins).