4 ms·

What are you saying goes beyond "101 optimization algorithms"? Are you saying to put another "layer" of deep learning to tune the parameters of the first "layer

by nonbel 8y ago

What are you saying goes beyond "101 optimization algorithms"? Are you saying to put another "layer" of deep learning to tune the parameters of the first "layer"? Doesn't this just add even more parameters to tune?

bitL 8y ago
No, you don't put another layer of Deep Learning (well, you technically could if you had some training data for optimization, but almost certainly you don't). You simply put another layer of (mixed-integer) non-linear optimization on top of Deep Learning hyperparameters, i.e. the hyper-parameters like learning rate, batch size, category weights, even the loss function composition, will be the variables you optimize. The (mixed-integer) non-linear optimization method will have its own set of parameters (they all do), so later you might indeed want to optimize those as well, if you get hold of some hyper-computing device allowing you to compress the first two steps to less than your lifetime. You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is.
- nonbel 8y ago
  Sorry, its still unclear. Do you consider genetic algorithms, etc to be an example of (mixed-integer) non-linear optimization? Or are you talking about some other method? >"You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is." In practice you don't need to find the actual optimum though, just do better than the alternatives. This reminds me of "a single nn layer can approximate any function". Sure, but in practice a reasonable approximation will arise much easier with other architectures. EDIT: By "101" I thought you meant something like "low level" or "first thing you learn/try". Did you mean something else?
- joe_the_user 8y ago
  I seems like you're confusing an algorithm that could attain absolute maximum for hyperparameters with something could attain "good enough" values for hyperparameters. After all, the models that work "well-enough" now, that attain state of the art results, don't have to have absolute best parameters but rather parameters that come from rules of thumb, trial-and-error and sometimes the 101-optimizations mentioned. I mean, we knows things work, there are working results, the challenge is determining exactly how these appear.

by nonbel 8y ago

bitL 8y ago

No, you don't put another layer of Deep Learning (well, you technically could if you had some training data for optimization, but almost certainly you don't). You simply put another layer of (mixed-integer) non-linear optimization on top of Deep Learning hyperparameters, i.e. the hyper-parameters like learning rate, batch size, category weights, even the loss function composition, will be the variables you optimize. The (mixed-integer) non-linear optimization method will have its own set of parameters (they all do), so later you might indeed want to optimize those as well, if you get hold of some hyper-computing device allowing you to compress the first two steps to less than your lifetime. You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is.

nonbel 8y ago
Sorry, its still unclear. Do you consider genetic algorithms, etc to be an example of (mixed-integer) non-linear optimization? Or are you talking about some other method? >"You can probably do this kind of hyper^n optimization as deeply as you like but I doubt it would lead you anywhere, given how terrible non-linear optimization performance on general functions is and how short the existence of Universe is." In practice you don't need to find the actual optimum though, just do better than the alternatives. This reminds me of "a single nn layer can approximate any function". Sure, but in practice a reasonable approximation will arise much easier with other architectures. EDIT: By "101" I thought you meant something like "low level" or "first thing you learn/try". Did you mean something else?
joe_the_user 8y ago
I seems like you're confusing an algorithm that could attain absolute maximum for hyperparameters with something could attain "good enough" values for hyperparameters. After all, the models that work "well-enough" now, that attain state of the art results, don't have to have absolute best parameters but rather parameters that come from rules of thumb, trial-and-error and sometimes the 101-optimizations mentioned. I mean, we knows things work, there are working results, the challenge is determining exactly how these appear.