5 ms·

Reinforcement Learning and Optimal Control

213 points by iron0013 8y ago

samuell 8y ago
Seems like an interesting complement to the (free) book on planning algorithms by Steven Lavalle: http://planning.cs.uiuc.edu/ http://planning.cs.uiuc.edu/
lawrenceyan 8y ago
If you're looking for a great lecture course, I highly recommend CS 294. It's taught by Sergey Levine, who is in my opinion one of the foundational researchers at the frontier of deep reinforcement learning, and provides a comprehensive overview on the current state of the art: http://rail.eecs.berkeley.edu/deeprlcourse/ http://rail.eecs.berkeley.edu/deeprlcourse/
- najarvg 8y ago
  I took the time to look through two of the lectures from the slides available. This is so inspirational. Thanks a lot!
  - lawrenceyan 8y ago
    A link to all the lecture videos for Fall 2018 is available here if you'd like to listen to the actual recordings: https://www.youtube.com/watch?v=opaBjK4TfLc&list=PLkFD6_40KJIxJMR-j5A1mkxK26gh_qg37&index=25 https://www.youtube.com/watch?v=opaBjK4TfLc&list=PLkFD6_40KJ...
- abdelm 8y ago
  Any idea how it compares to the UCL course taught by researchers from DeepMind? http://www.cs.ucl.ac.uk/syllabus/compgi/compgi22_advanced_deep_learning_and_reinforcement_learning/ http://www.cs.ucl.ac.uk/syllabus/compgi/compgi22_advanced_de...
  - lawrenceyan 8y ago
    The Berkeley course is completely open source with all the lectures, homework problem sets, and exams publicly available on the site. Besides getting a grade, everything is available to you. Unfortunately I'm not sure about how the UCL course compares as it seems like the materials are closed to non-UCL students. If you have access though, I'm sure there's lots to learn by taking the course!
clickok 8y ago
Bertsekas' earlier books (Dynamic Programming and Optimal Control + Neurodynamic Programming w/ Tsitsiklis) are great references and collect many insights & results that you'd otherwise have to trawl the literature for. I'm very interested to see what a book focused more narrowly on RL will be like-- Sutton's Introduction to Reinforcement Learning[0] is fantastic, but if you're going to do research on RL, another text such as this one is necessary. 0. http://www.incompleteideas.net/book/the-book-2nd.html http://www.incompleteideas.net/book/the-book-2nd.html
joshuathomas096 8y ago
This inspired me so much. Thank you for sharing!
del_operator 8y ago
I liked the open courseware lectures with John Tsitsiklis and ended up with a few books by Bertsekas: neuro dynamic programming and intro probability. Both Bertsekas and Tsitsiklis recommended the Sutton and Barto intro book for an intuitive overview. I liked it.
dpandya 8y ago
Not meant as a critique to this book, but the fact that RL-based approaches rarely work for optimal control problems (in, for example, robotics) came as a surprise to me, given the hype and focus on RL [1]. It turns out that model-based methods for optimal control (e.g. linear quadratic control) invented quite a long time ago dramatically outperform RL-based approaches in most tasks and require multiple orders of magnitude less computational resources. Maybe there's some hope for RL method if they "course correct" for simpler control methods. At the moment, it seems like RL for robotics and control lies to the side of "research" and not "engineering." [1]: https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html
- Jack000 8y ago
  for linear systems modern control theory is already optimal (?) so given a linear system I'd assume an RL method would just approximate the optimal control method. I think the potential for RL methods is in capturing nonlinearities in actuators like artificial muscle.
  - dpandya 8y ago
    I meant applying a linear approximation.
incompetent 8y ago
awesome. not only is this site for 'tech elite' written by a someone with the programming skills of a child, the mods have recently started leaving alone obvious spam. all content is fine, as ling as you're nice. so the mentality of an insecure high school social reject, who is now a grown angry little adult. hey loser, gurss what. you've shadow banned an IP. A NAT IP. For an internet provider. About 20 people at work can't post. And that's just one of many buildings using the provider, all behind the same IP. Boy you're smart. Surprisingly, not being nice, especially to these incel mods, is still very easy from any of the hundreds of free vpns. So what you've done is killed off a slew of your users, while not accomplishing your goal. PERFECT! Could not want more. The mods have turned this site to shit -a safespace for losers with reddit-level crap discussion. Time to change the mods.

213 points by iron0013 8y ago

Seems like an interesting complement to the (free) book on planning algorithms by Steven Lavalle: http://planning.cs.uiuc.edu/ http://planning.cs.uiuc.edu/

lawrenceyan 8y ago

If you're looking for a great lecture course, I highly recommend CS 294. It's taught by Sergey Levine, who is in my opinion one of the foundational researchers at the frontier of deep reinforcement learning, and provides a comprehensive overview on the current state of the art: http://rail.eecs.berkeley.edu/deeprlcourse/ http://rail.eecs.berkeley.edu/deeprlcourse/

najarvg 8y ago
I took the time to look through two of the lectures from the slides available. This is so inspirational. Thanks a lot!
- lawrenceyan 8y ago
  A link to all the lecture videos for Fall 2018 is available here if you'd like to listen to the actual recordings: https://www.youtube.com/watch?v=opaBjK4TfLc&list=PLkFD6_40KJIxJMR-j5A1mkxK26gh_qg37&index=25 https://www.youtube.com/watch?v=opaBjK4TfLc&list=PLkFD6_40KJ...
abdelm 8y ago
Any idea how it compares to the UCL course taught by researchers from DeepMind? http://www.cs.ucl.ac.uk/syllabus/compgi/compgi22_advanced_deep_learning_and_reinforcement_learning/ http://www.cs.ucl.ac.uk/syllabus/compgi/compgi22_advanced_de...
- lawrenceyan 8y ago
  The Berkeley course is completely open source with all the lectures, homework problem sets, and exams publicly available on the site. Besides getting a grade, everything is available to you. Unfortunately I'm not sure about how the UCL course compares as it seems like the materials are closed to non-UCL students. If you have access though, I'm sure there's lots to learn by taking the course!

clickok 8y ago

Bertsekas' earlier books (Dynamic Programming and Optimal Control + Neurodynamic Programming w/ Tsitsiklis) are great references and collect many insights & results that you'd otherwise have to trawl the literature for. I'm very interested to see what a book focused more narrowly on RL will be like-- Sutton's Introduction to Reinforcement Learning[0] is fantastic, but if you're going to do research on RL, another text such as this one is necessary. 0. http://www.incompleteideas.net/book/the-book-2nd.html http://www.incompleteideas.net/book/the-book-2nd.html

joshuathomas096 8y ago

This inspired me so much. Thank you for sharing!

del_operator 8y ago

I liked the open courseware lectures with John Tsitsiklis and ended up with a few books by Bertsekas: neuro dynamic programming and intro probability. Both Bertsekas and Tsitsiklis recommended the Sutton and Barto intro book for an intuitive overview. I liked it.

dpandya 8y ago

Not meant as a critique to this book, but the fact that RL-based approaches rarely work for optimal control problems (in, for example, robotics) came as a surprise to me, given the hype and focus on RL [1]. It turns out that model-based methods for optimal control (e.g. linear quadratic control) invented quite a long time ago dramatically outperform RL-based approaches in most tasks and require multiple orders of magnitude less computational resources. Maybe there's some hope for RL method if they "course correct" for simpler control methods. At the moment, it seems like RL for robotics and control lies to the side of "research" and not "engineering." [1]: https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html

Jack000 8y ago
for linear systems modern control theory is already optimal (?) so given a linear system I'd assume an RL method would just approximate the optimal control method. I think the potential for RL methods is in capturing nonlinearities in actuators like artificial muscle.
- dpandya 8y ago
  I meant applying a linear approximation.

incompetent 8y ago

awesome. not only is this site for 'tech elite' written by a someone with the programming skills of a child, the mods have recently started leaving alone obvious spam. all content is fine, as ling as you're nice. so the mentality of an insecure high school social reject, who is now a grown angry little adult. hey loser, gurss what. you've shadow banned an IP. A NAT IP. For an internet provider. About 20 people at work can't post. And that's just one of many buildings using the provider, all behind the same IP. Boy you're smart. Surprisingly, not being nice, especially to these incel mods, is still very easy from any of the hundreds of free vpns. So what you've done is killed off a slew of your users, while not accomplishing your goal. PERFECT! Could not want more. The mods have turned this site to shit -a safespace for losers with reddit-level crap discussion. Time to change the mods.