3 ms·

I've always been interested in reinforcement learning, believing it will magically solve anything I throw at it. Unfortunately, I haven't got it to work everywh

I've always been interested in reinforcement learning, believing it will magically solve anything I throw at it. Unfortunately, I haven't got it to work everywhere yet. I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization, because RL can do almost as good. I don't truly believe this, but I think it is an underlying psychological reason for my love of RL. It's scary to think I might have to learn a lot of new algorithms to solve a new problem, and maybe I won't be able to solve it at all. Any suggestions on the best resources for branching out from RL to more classical control and optimization?

cweill 5y ago
The Deepmind lead behind AlphaStar, there StarCraft 2 bot that beat several world champs, once said, "the best way to solve a reinforcement learning problem is with supervised learning". That's because supervised learning is relatively extremely efficient, and just RL with more constraints. Just start learning the basics of supervised learning for classification and regression on common benchmark tasks like CIFAR and UCI. Apply a mix of linear models, neural networks, and trees like random forests and GBDTs. Next try convolutional networks for vision and transformers for NLP. You'll be all set to solve most real world problems.
- venividivini 5y ago
  I've been working on imitation learning recently and can attest that reducing RL to supervised learning gives remarkable results, even when using naive algorithms (like Behavioral Cloning). That's an excellent quote by the way, do you perhaps have any source? Sounds like a good opener slide :)
stncls 5y ago
> I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization I think that you can use optimization without having to learn anything about its algorithms (learning the basics is always advised of course, but that takes nowhere near as much effort). Nowadays, an off-the-shelf black-box solver will perform better than a custom implementation in most cases: what you lose in fine-tuning is more than compensated by the implementation quality and the sheer number of algorithms available. > best resources for branching out from RL to more classical control and optimization It depends on your problem. For general unstructured (nonconvex continuous) optimization, like the OP, you can use NLopt's documentation as a pragmatic starting point [1]. Optimal control is a whole different thing though, and you may have to start with some academic papers/books. I have been recommended this one [2], but I have not read it. [1] https://nlopt.readthedocs.io/en/latest/ https://nlopt.readthedocs.io/en/latest/ [2] https://link.springer.com/book/10.1007/978-1-4471-0967-9 https://link.springer.com/book/10.1007/978-1-4471-0967-9
AlexanderTheGr8 5y ago
I feel the exact same way!! Throw RL at the wall for any problem, and it will magically solve it. In the back of my mind, I know that it's not true, but I always want to throw RL at any problem. Right now, I am considering throwing it to make TCP/other-networking-stuff more efficient...

cweill 5y ago

The Deepmind lead behind AlphaStar, there StarCraft 2 bot that beat several world champs, once said, "the best way to solve a reinforcement learning problem is with supervised learning". That's because supervised learning is relatively extremely efficient, and just RL with more constraints. Just start learning the basics of supervised learning for classification and regression on common benchmark tasks like CIFAR and UCI. Apply a mix of linear models, neural networks, and trees like random forests and GBDTs. Next try convolutional networks for vision and transformers for NLP. You'll be all set to solve most real world problems.

venividivini 5y ago
I've been working on imitation learning recently and can attest that reducing RL to supervised learning gives remarkable results, even when using naive algorithms (like Behavioral Cloning). That's an excellent quote by the way, do you perhaps have any source? Sounds like a good opener slide :)

stncls 5y ago

> I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization I think that you can use optimization without having to learn anything about its algorithms (learning the basics is always advised of course, but that takes nowhere near as much effort). Nowadays, an off-the-shelf black-box solver will perform better than a custom implementation in most cases: what you lose in fine-tuning is more than compensated by the implementation quality and the sheer number of algorithms available. > best resources for branching out from RL to more classical control and optimization It depends on your problem. For general unstructured (nonconvex continuous) optimization, like the OP, you can use NLopt's documentation as a pragmatic starting point [1]. Optimal control is a whole different thing though, and you may have to start with some academic papers/books. I have been recommended this one [2], but I have not read it. [1] https://nlopt.readthedocs.io/en/latest/ https://nlopt.readthedocs.io/en/latest/ [2] https://link.springer.com/book/10.1007/978-1-4471-0967-9 https://link.springer.com/book/10.1007/978-1-4471-0967-9

AlexanderTheGr8 5y ago

I feel the exact same way!! Throw RL at the wall for any problem, and it will magically solve it. In the back of my mind, I know that it's not true, but I always want to throw RL at any problem. Right now, I am considering throwing it to make TCP/other-networking-stuff more efficient...