4 ms·
Trying to dig into Optimal Control Theory a bit, after realizing that - in many ways - OCT and (certain aspects of) Machine Learning are just opposite sides of
by mindcrime 8y ago
Trying to dig into Optimal Control Theory a bit, after realizing that - in many ways - OCT and (certain aspects of) Machine Learning are just opposite sides of the same coin. Reinforcment Learning in particular shares a lot of concepts with OCT.
- gota 8y agoReinforcement learning is direct adaptive optimal control https://ieeexplore.ieee.org/document/126844/?reload=true https://ieeexplore.ieee.org/document/126844/?reload=true
- mindcrime 8y agoNon-paywalled link: http://www.ieeecss.org/CSM/library/1992/april1992/w01-ReinforcementLearning.pdf http://www.ieeecss.org/CSM/library/1992/april1992/w01-Reinfo...
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- rocketflumes 8y agofor more on that subject - check out this recent RL and OCT survey by Ben Recht, also from UC Berkeley: https://arxiv.org/abs/1806.09460 https://arxiv.org/abs/1806.09460
- pjrule 8y agoBen Recht also has an excellent series of blog posts (very related to this survey on arXiv, but broader) on the intersection between reinforcement learning and optimal control. An index is available here: http://www.argmin.net/2018/06/25/outsider-rl/ http://www.argmin.net/2018/06/25/outsider-rl/
- mindcrime 8y agoI was just reading those last night. Definitely good stuff.
- stevesimmons 8y agoBen Recht also gave a 2 hour tutorial on "Optimization Perspectives on Learning to Control" at ICML on 10 July. It was a great talk, loosely based on his blog posts, and very popular, with every seat filled. His slides, references and FB livestreamed video, are here: https://people.eecs.berkeley.edu/~brecht/l2c-icml2018/ https://people.eecs.berkeley.edu/~brecht/l2c-icml2018/
- amelius 8y agoBut isn't OCT more rigorous, with proofs/guarantees and such, and ML more experimental?
- tnecniv 8y agoDepends on what you mean by rigorous and who you are talking to. Both fields are attempting to solve the same problem: choose the optimal action to take at the current time for a given process. Control theorists normally start out with a model, or a family of potential models that describe the behavior of the process and work from there to determine the optimal action. This is very much an area of applied mathematics and academics take rigorous approaches, but, in industry, many engineers just use a PID or LQR controller and call it a day regardless how applicable they are to the actual system theoretically. Meanwhile, the reinforcement learning folk typically work on problems where the models are too complicated to work with computationally or often even write down, so a more tractable approach is to learn a model and control policy from data. There's plenty of people who analyze properties of learning algorithms, etc., within this framework, and others who don't really care beyond whether or not the system works.
- mindcrime 8y agoControl theorists normally start out with a model, or a family of potential models that describe the behavior of the process and work from there to determine the optimal action. This is the main distinction I've been exposed to, between Optimal Control and Reinforcement Learning. I've heard it summarized as "Optimal Control uses models, Reinforcement Learning tries very hard to stay away from using models". That's probably simplifying things a little bit too much, but it seems like a reasonable starting point to see where the two fields diverge.
- tnecniv 8y agoYeah that's the gist of it. There are things like adaptive control where aspects of the model are adjusted on the fly in real-time to improve performance based on data from the system and robust control that tries to account for modeling error.