8 ms·

Why train when you can optimize?

182 points by sicp-enjoyer 4y ago

Buttons840 4y ago
I've always been interested in reinforcement learning, believing it will magically solve anything I throw at it. Unfortunately, I haven't got it to work everywhere yet. I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization, because RL can do almost as good. I don't truly believe this, but I think it is an underlying psychological reason for my love of RL. It's scary to think I might have to learn a lot of new algorithms to solve a new problem, and maybe I won't be able to solve it at all. Any suggestions on the best resources for branching out from RL to more classical control and optimization?
- cweill 4y ago
  The Deepmind lead behind AlphaStar, there StarCraft 2 bot that beat several world champs, once said, "the best way to solve a reinforcement learning problem is with supervised learning". That's because supervised learning is relatively extremely efficient, and just RL with more constraints. Just start learning the basics of supervised learning for classification and regression on common benchmark tasks like CIFAR and UCI. Apply a mix of linear models, neural networks, and trees like random forests and GBDTs. Next try convolutional networks for vision and transformers for NLP. You'll be all set to solve most real world problems.
  - venividivini 4y ago
    I've been working on imitation learning recently and can attest that reducing RL to supervised learning gives remarkable results, even when using naive algorithms (like Behavioral Cloning). That's an excellent quote by the way, do you perhaps have any source? Sounds like a good opener slide :)
- stncls 4y ago
  > I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization I think that you can use optimization without having to learn anything about its algorithms (learning the basics is always advised of course, but that takes nowhere near as much effort). Nowadays, an off-the-shelf black-box solver will perform better than a custom implementation in most cases: what you lose in fine-tuning is more than compensated by the implementation quality and the sheer number of algorithms available. > best resources for branching out from RL to more classical control and optimization It depends on your problem. For general unstructured (nonconvex continuous) optimization, like the OP, you can use NLopt's documentation as a pragmatic starting point [1]. Optimal control is a whole different thing though, and you may have to start with some academic papers/books. I have been recommended this one [2], but I have not read it. [1] https://nlopt.readthedocs.io/en/latest/ https://nlopt.readthedocs.io/en/latest/ [2] https://link.springer.com/book/10.1007/978-1-4471-0967-9 https://link.springer.com/book/10.1007/978-1-4471-0967-9
- AlexanderTheGr8 4y ago
  I feel the exact same way!! Throw RL at the wall for any problem, and it will magically solve it. In the back of my mind, I know that it's not true, but I always want to throw RL at any problem. Right now, I am considering throwing it to make TCP/other-networking-stuff more efficient...
tehsauce 4y ago
Very cool article, but if you want to draw straight lines just use the start and end points :/
- nurettin 4y ago
  Yes but can you trust a human to draw from start to end?
- taylorius 4y ago
  You might accidentally veer off just at the end of the line - better to take all points into account.
steppi 4y ago
Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encode prior understanding into either the structure of the network or choice of activation function. See the paper Neural Networks Fail to Learn Periodic Functions and How to Fix It by Ziwin, Hartweg, and Uweda. Where they propose the activation function f(x) = x + sin(x)^2 that can encode an understanding that the underlying function should be periodic. [1] https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf https://proceedings.neurips.cc/paper/2020/file/1160453108d3e...
- thomasahle 4y ago
  While `x + sin(x)^2` may be monotonic itself, it only takes a simple linear combination of two neurons like `x+sin(x)^2 - (x/2 + sin(x/2)^2)` before you have a completely crazy loss landscape. I have a feeling this is why such activation functions haven't become standard.
- whatshisface 4y ago
  If we're allowed to do that, why can't my activation function be sin(x)?
  - deleted 4y ago
    [deleted]
  - blackcat201 4y ago
    You can use sin as activation function, but that would require careful initialization to avoid gradient explosion as you would ended up with a lot of points where gradient is simply zero. You can refer to Implicit Neural Representations with Periodic Activation Functions for more details.
    - zarzavat 4y ago
taeric 4y ago
Isn't training literally an optimization of a loss function over training data?
- diffeomorphism 4y ago
  Technically yes, but practically no. This is discussed in depth in section 11.
- srean 4y ago
  It is and that is why I find the title a clickbait. The article isnt bad, but the title is certainly being coyly disingenuous. In problems like this there are two aspects, (i) designing or specifying the search space of functions (ii) choosing the best function within the search space. The opposite extremes are a) the search space contains only one function, the right function. In this case the training/optimization is moot. The other extreme is to have a very wide search space, say all smooth functions. In that case searching/training/optimizing is more challenging. The more reasonable example is one uses domain knowledge to design a much more restricted search space (for example, one may encode that the function is periodic with a known period) making the next step easier.
taylorius 4y ago
A small shout-out to Differential Evolution, which is my go-to (derivative free) optimization algorithm. Think of a Genetic Algorithm, but the crossover operator is linear interpolation (DE's natural domain is real-valued vectors). It's simple, and in my experience, it works pretty well.
- fho 4y ago
  Scipy has a good implementation of that. I had some fun with it over the last couple of weeks. Just be aware that it runs a "polishing" optimization at the end which might improve the results, or just take forever :-)
DeathArrow 4y ago
The author claims ML is overused and many problems that could be solved more effectively with optimizations are solved using ML. If wonder what classes of problems fit this? For sure, I can't imagine how can you tackle sentiment analysis or text classifiers using optimization.
- teruakohatu 4y ago
  > tackle sentiment analysis VADER (Valence Aware Dictionary and sEntiment Reasoner) is a large lookup table mapping words to a sentiment score, calculated by surveying people. It is simple to use and involves no ML.
  - PeterisP 4y ago
    I don't consider VADER as suitable for tackling sentiment analysis but perhaps we have a different understanding of 'tackling sentiment analysis'. In my view, 'tackling sentiment analysis' requires doing sentiment analysis well, which no systems currently do. Decent sentiment analysis is a yet unsolved problem and current state of art solutions (mostly based on large pretrained langugage models like BERT, leveraging transfer learning from very large unlabeled corpora) are not sufficiently accurate, and of course neither is VADER. As soon as you go beyond simple binary polarity (blatantly positive/negative sentiment) and something more informative like various five-way sentiment tasks, or aspect-based sentiment or determining sentiment targets, sentiment analysis still has a long, long way to go. Current ML solutions are a possible way to tackle sentiment analysis because while the current models are not good enough, we have evidence showing that with increased model and data size they improve and perhaps might result in sufficiently good sentiment analysis someday. Perhaps not, and we'll need something else; but at least they have some potential. VADER and similar methods are not a way to tackle sentiment analysis because not only the current VADER model is not nearly good enough, it's clear that they can't scale to anything much better than that. Adding extra surveys to increase the lookup table quickly hits diminishing returns (unlike ML systems where we consistently see that ever larger models continue to get improvements), and if they can't even reach the current bar of ML models (which still are not good enough) then they can't bring us to the level of sentiment analysis where we want to be, they are a dead end. Tackling sentiment analysis also requires handling a wide variety of languages, not only English, which is yet another aspect where data-driven models have an advantage over systems that require extensive human labor for each new language. I mean, we all (me included!) want to believe that human knowledge encoded in rules can work, intuitively it's a very appealing concept, however, it does not work out in practice and IMHO in the end we all have to learn to accept the Bitter Lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html).
DeathArrow 4y ago
NN failed to model sine function: https://towardsdatascience.com/can-machine-learn-the-concept-of-sine-4047dced3f11 https://towardsdatascience.com/can-machine-learn-the-concept...
srean 4y ago
The article itself is good but the title is clickbait. The process of training is minimization (in other words, optimization) of generalization error.
- d4rkp4ttern 4y ago
  True, but to be a bit pedantic, I wouldn’t call that generalization error (which you don’t have access to while training). Training minimizes some loss function which is a proxy for generalization error.
  - srean 4y ago
    Spot on.
deterministic 4y ago
Optimisers are perfect for solving problems where you don’t have access to a large training set. I actually recommend starting with an optimisation algorithm first and only then try NN if using an optimisation algorithm isn’t enough to solve the problem.

182 points by sicp-enjoyer 4y ago

I've always been interested in reinforcement learning, believing it will magically solve anything I throw at it. Unfortunately, I haven't got it to work everywhere yet. I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization, because RL can do almost as good. I don't truly believe this, but I think it is an underlying psychological reason for my love of RL. It's scary to think I might have to learn a lot of new algorithms to solve a new problem, and maybe I won't be able to solve it at all. Any suggestions on the best resources for branching out from RL to more classical control and optimization?

cweill 4y ago
The Deepmind lead behind AlphaStar, there StarCraft 2 bot that beat several world champs, once said, "the best way to solve a reinforcement learning problem is with supervised learning". That's because supervised learning is relatively extremely efficient, and just RL with more constraints. Just start learning the basics of supervised learning for classification and regression on common benchmark tasks like CIFAR and UCI. Apply a mix of linear models, neural networks, and trees like random forests and GBDTs. Next try convolutional networks for vision and transformers for NLP. You'll be all set to solve most real world problems.
- venividivini 4y ago
  I've been working on imitation learning recently and can attest that reducing RL to supervised learning gives remarkable results, even when using naive algorithms (like Behavioral Cloning). That's an excellent quote by the way, do you perhaps have any source? Sounds like a good opener slide :)
stncls 4y ago
> I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization I think that you can use optimization without having to learn anything about its algorithms (learning the basics is always advised of course, but that takes nowhere near as much effort). Nowadays, an off-the-shelf black-box solver will perform better than a custom implementation in most cases: what you lose in fine-tuning is more than compensated by the implementation quality and the sheer number of algorithms available. > best resources for branching out from RL to more classical control and optimization It depends on your problem. For general unstructured (nonconvex continuous) optimization, like the OP, you can use NLopt's documentation as a pragmatic starting point [1]. Optimal control is a whole different thing though, and you may have to start with some academic papers/books. I have been recommended this one [2], but I have not read it. [1] https://nlopt.readthedocs.io/en/latest/ https://nlopt.readthedocs.io/en/latest/ [2] https://link.springer.com/book/10.1007/978-1-4471-0967-9 https://link.springer.com/book/10.1007/978-1-4471-0967-9
AlexanderTheGr8 4y ago
I feel the exact same way!! Throw RL at the wall for any problem, and it will magically solve it. In the back of my mind, I know that it's not true, but I always want to throw RL at any problem. Right now, I am considering throwing it to make TCP/other-networking-stuff more efficient...

tehsauce 4y ago

Very cool article, but if you want to draw straight lines just use the start and end points :/

nurettin 4y ago
Yes but can you trust a human to draw from start to end?
taylorius 4y ago
You might accidentally veer off just at the end of the line - better to take all points into account.

steppi 4y ago

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encode prior understanding into either the structure of the network or choice of activation function. See the paper Neural Networks Fail to Learn Periodic Functions and How to Fix It by Ziwin, Hartweg, and Uweda. Where they propose the activation function f(x) = x + sin(x)^2 that can encode an understanding that the underlying function should be periodic. [1] https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf https://proceedings.neurips.cc/paper/2020/file/1160453108d3e...

thomasahle 4y ago
While `x + sin(x)^2` may be monotonic itself, it only takes a simple linear combination of two neurons like `x+sin(x)^2 - (x/2 + sin(x/2)^2)` before you have a completely crazy loss landscape. I have a feeling this is why such activation functions haven't become standard.
whatshisface 4y ago
If we're allowed to do that, why can't my activation function be sin(x)?
- deleted 4y ago
  [deleted]
- blackcat201 4y ago
  You can use sin as activation function, but that would require careful initialization to avoid gradient explosion as you would ended up with a lot of points where gradient is simply zero. You can refer to Implicit Neural Representations with Periodic Activation Functions for more details.
  - zarzavat 4y ago

taeric 4y ago

Isn't training literally an optimization of a loss function over training data?

diffeomorphism 4y ago
Technically yes, but practically no. This is discussed in depth in section 11.
srean 4y ago
It is and that is why I find the title a clickbait. The article isnt bad, but the title is certainly being coyly disingenuous. In problems like this there are two aspects, (i) designing or specifying the search space of functions (ii) choosing the best function within the search space. The opposite extremes are a) the search space contains only one function, the right function. In this case the training/optimization is moot. The other extreme is to have a very wide search space, say all smooth functions. In that case searching/training/optimizing is more challenging. The more reasonable example is one uses domain knowledge to design a much more restricted search space (for example, one may encode that the function is periodic with a known period) making the next step easier.

taylorius 4y ago

A small shout-out to Differential Evolution, which is my go-to (derivative free) optimization algorithm. Think of a Genetic Algorithm, but the crossover operator is linear interpolation (DE's natural domain is real-valued vectors). It's simple, and in my experience, it works pretty well.

fho 4y ago
Scipy has a good implementation of that. I had some fun with it over the last couple of weeks. Just be aware that it runs a "polishing" optimization at the end which might improve the results, or just take forever :-)

DeathArrow 4y ago

The author claims ML is overused and many problems that could be solved more effectively with optimizations are solved using ML. If wonder what classes of problems fit this? For sure, I can't imagine how can you tackle sentiment analysis or text classifiers using optimization.

teruakohatu 4y ago
> tackle sentiment analysis VADER (Valence Aware Dictionary and sEntiment Reasoner) is a large lookup table mapping words to a sentiment score, calculated by surveying people. It is simple to use and involves no ML.
- PeterisP 4y ago
  I don't consider VADER as suitable for tackling sentiment analysis but perhaps we have a different understanding of 'tackling sentiment analysis'. In my view, 'tackling sentiment analysis' requires doing sentiment analysis well, which no systems currently do. Decent sentiment analysis is a yet unsolved problem and current state of art solutions (mostly based on large pretrained langugage models like BERT, leveraging transfer learning from very large unlabeled corpora) are not sufficiently accurate, and of course neither is VADER. As soon as you go beyond simple binary polarity (blatantly positive/negative sentiment) and something more informative like various five-way sentiment tasks, or aspect-based sentiment or determining sentiment targets, sentiment analysis still has a long, long way to go. Current ML solutions are a possible way to tackle sentiment analysis because while the current models are not good enough, we have evidence showing that with increased model and data size they improve and perhaps might result in sufficiently good sentiment analysis someday. Perhaps not, and we'll need something else; but at least they have some potential. VADER and similar methods are not a way to tackle sentiment analysis because not only the current VADER model is not nearly good enough, it's clear that they can't scale to anything much better than that. Adding extra surveys to increase the lookup table quickly hits diminishing returns (unlike ML systems where we consistently see that ever larger models continue to get improvements), and if they can't even reach the current bar of ML models (which still are not good enough) then they can't bring us to the level of sentiment analysis where we want to be, they are a dead end. Tackling sentiment analysis also requires handling a wide variety of languages, not only English, which is yet another aspect where data-driven models have an advantage over systems that require extensive human labor for each new language. I mean, we all (me included!) want to believe that human knowledge encoded in rules can work, intuitively it's a very appealing concept, however, it does not work out in practice and IMHO in the end we all have to learn to accept the Bitter Lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html).

DeathArrow 4y ago

NN failed to model sine function: https://towardsdatascience.com/can-machine-learn-the-concept-of-sine-4047dced3f11 https://towardsdatascience.com/can-machine-learn-the-concept...

srean 4y ago

The article itself is good but the title is clickbait. The process of training is minimization (in other words, optimization) of generalization error.

d4rkp4ttern 4y ago
True, but to be a bit pedantic, I wouldn’t call that generalization error (which you don’t have access to while training). Training minimizes some loss function which is a proxy for generalization error.
- srean 4y ago
  Spot on.

deterministic 4y ago

Optimisers are perfect for solving problems where you don’t have access to a large training set. I actually recommend starting with an optimisation algorithm first and only then try NN if using an optimisation algorithm isn’t enough to solve the problem.