3 ms·

I think "learn" is a bit misleading here but I do have to say it's a nice and intuitive overview of RL. RL is quite hard and math heavy, I don't know if one ca

I think "learn" is a bit misleading here but I do have to say it's a nice and intuitive overview of RL. RL is quite hard and math heavy, I don't know if one can take a short cut in learning RL without solid graduate level math foundation.

nonbel 8y ago
>"RL is quite hard and math heavy" Is it really math heavy? Its not like people are working out these algorithms by hand. I haven't done RL but would assume you just need to download some libraries and tutorials then play around with it to get an understanding. This may go faster if you already read a book about it, worked out the math behind how it works, etc but that shouldn't be necessary.
- yonkshi 8y ago
  For very limited applications you can probably plug and play something like DDQN or A3C, but since we don't have a good algorithm that applies to many RL problems, any innovation will take quite a bit of theoretical background. A while ago we tried to use DDQN to self-play a multi-agent pacman environment that we built, it was able to converge but the convergence was very suboptimal. We had to modify the sampling process in order to get slightly better convergence.
Buttons840 8y ago
I disagree. At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases. I think it might be perceived as being math heavy because the best resources on the topic [0][1] use a lot of math notation to express their ideas. The ideas are subtle and easy to mess up though. I think these books are some of the first that made me appreciate math notation, it can look scary, but it does convey the ideas more accurately than words can. [0] http://incompleteideas.net/ http://incompleteideas.net/ (his "Reinforcement Learning: An Introduction" is a great book) [1] https://www.manning.com/books/grokking-deep-reinforcement-learning https://www.manning.com/books/grokking-deep-reinforcement-le... looks promising, I've read the few chapters available and was impressed. It's still an early work with grammatical mistakes, but the layout of the ideas was clear and organized.
- yonkshi 8y ago
  Yes, bellman equation fundamental idea for all RL algorithms (i.e. updating table of values), but RL is also much more fragile than supervised learning methods, thus to ensure stability, modern algorithms have complex mathematical tools to fix that. I wouldn't say it's the math notations that's scary, rather the concept behind modern algorithms require higher mathematical concepts. For example, Max Entropy algorithm for inverse RL, it's essential to know the concepts of Shannon information entropy to understand why it works.
  - Buttons840 8y ago
    Granted, I am not an expert in RL. It sounds like you have more experience than me so maybe I will change my mind as I learn more.
  - moultano 8y ago
    Information theory does not require graduate level math. Even if it's entirely new to you, you can pick it up with a few days on wikipedia.
    - yonkshi 8y ago
      Ah yeah I agree it's not a terribly difficult concept to learn, but perhaps we have a different definition of graduate level math. Things like information theory are, at best, skimmed over in upper div math classes during undergrad. Having a solid understanding things like KL divergence, information entropy is just an indicator of one's overall math level, and if you are already there, you can consider yourself at graduate level. Though I guess math used in ML is probably kids play comparing to a graduate level mathematician or physicist.
- vinn124 8y ago
  > At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases. this seems to be a common assertion about ml. other refrains include "ml is just matrix multiplication" and "ml is just affine transformations followed by nonlinearity". while technically true, it is an unhelpful comment as it doesnt shed any light on the salient questions, such as: * how do you reduce bias/variance, since youre sampling a minuscule slice of the state/action space? * how do you construct a value function (or something like it) in a sparse reward environment, with possibly thousands of time steps? * how do you know your policy network is exploring new states? it is the equivalent of learning how to draw an owl: draw a few circles, then draw the fucking owl.
MarkMMullin 8y ago
It depends on whether you are implementing a library to support RL or using one that does - in the former case, yah, you better have a grip, in the latter, you need to understand enough to know how to utilize the library and not feed it nasty things.... this isn't to say you can be mathematically illiterate, however there are levels to required knowledge - the same as you don't need to know how to make a compiler to use one
- yonkshi 8y ago
  I agree to an extend. The current state of RL algorithms are more fragmented and have limited application comparing to something like a CNN. So unless one only wants to learn how to play Atari, they won't be able to "fix" an algorithm if it breaks down in an untested environment. E.g. a non deterministic sparse reward game. If one day there is a generalized algorithm that can solve large set of RL problems, then I think a high level intuition is probably enough to use RL, but for now, I'd say RL is definitely not for the faint of heart.
  - MarkMMullin 8y ago
    True - I often make the heretical argument that SL vs RL is just a question of where the labeling comes from :-) You are correct that the tooling is weaker in this space, but it is growing - my point is only that there's a difference between knowing how to use the tool and knowing how to make the tool - making the tool can liquify your brain -- using someone else's tool (assuming its a good tool) will simply give you headaches from time to time :-D

nonbel 8y ago

>"RL is quite hard and math heavy" Is it really math heavy? Its not like people are working out these algorithms by hand. I haven't done RL but would assume you just need to download some libraries and tutorials then play around with it to get an understanding. This may go faster if you already read a book about it, worked out the math behind how it works, etc but that shouldn't be necessary.

yonkshi 8y ago
For very limited applications you can probably plug and play something like DDQN or A3C, but since we don't have a good algorithm that applies to many RL problems, any innovation will take quite a bit of theoretical background. A while ago we tried to use DDQN to self-play a multi-agent pacman environment that we built, it was able to converge but the convergence was very suboptimal. We had to modify the sampling process in order to get slightly better convergence.

Buttons840 8y ago

I disagree. At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases. I think it might be perceived as being math heavy because the best resources on the topic [0][1] use a lot of math notation to express their ideas. The ideas are subtle and easy to mess up though. I think these books are some of the first that made me appreciate math notation, it can look scary, but it does convey the ideas more accurately than words can. [0] http://incompleteideas.net/ http://incompleteideas.net/ (his "Reinforcement Learning: An Introduction" is a great book) [1] https://www.manning.com/books/grokking-deep-reinforcement-learning https://www.manning.com/books/grokking-deep-reinforcement-le... looks promising, I've read the few chapters available and was impressed. It's still an early work with grammatical mistakes, but the layout of the ideas was clear and organized.

yonkshi 8y ago
Yes, bellman equation fundamental idea for all RL algorithms (i.e. updating table of values), but RL is also much more fragile than supervised learning methods, thus to ensure stability, modern algorithms have complex mathematical tools to fix that. I wouldn't say it's the math notations that's scary, rather the concept behind modern algorithms require higher mathematical concepts. For example, Max Entropy algorithm for inverse RL, it's essential to know the concepts of Shannon information entropy to understand why it works.
- Buttons840 8y ago
  Granted, I am not an expert in RL. It sounds like you have more experience than me so maybe I will change my mind as I learn more.
- moultano 8y ago
  Information theory does not require graduate level math. Even if it's entirely new to you, you can pick it up with a few days on wikipedia.
  - yonkshi 8y ago
    Ah yeah I agree it's not a terribly difficult concept to learn, but perhaps we have a different definition of graduate level math. Things like information theory are, at best, skimmed over in upper div math classes during undergrad. Having a solid understanding things like KL divergence, information entropy is just an indicator of one's overall math level, and if you are already there, you can consider yourself at graduate level. Though I guess math used in ML is probably kids play comparing to a graduate level mathematician or physicist.
vinn124 8y ago
> At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases. this seems to be a common assertion about ml. other refrains include "ml is just matrix multiplication" and "ml is just affine transformations followed by nonlinearity". while technically true, it is an unhelpful comment as it doesnt shed any light on the salient questions, such as: * how do you reduce bias/variance, since youre sampling a minuscule slice of the state/action space? * how do you construct a value function (or something like it) in a sparse reward environment, with possibly thousands of time steps? * how do you know your policy network is exploring new states? it is the equivalent of learning how to draw an owl: draw a few circles, then draw the fucking owl.

MarkMMullin 8y ago

It depends on whether you are implementing a library to support RL or using one that does - in the former case, yah, you better have a grip, in the latter, you need to understand enough to know how to utilize the library and not feed it nasty things.... this isn't to say you can be mathematically illiterate, however there are levels to required knowledge - the same as you don't need to know how to make a compiler to use one

yonkshi 8y ago
I agree to an extend. The current state of RL algorithms are more fragmented and have limited application comparing to something like a CNN. So unless one only wants to learn how to play Atari, they won't be able to "fix" an algorithm if it breaks down in an untested environment. E.g. a non deterministic sparse reward game. If one day there is a generalized algorithm that can solve large set of RL problems, then I think a high level intuition is probably enough to use RL, but for now, I'd say RL is definitely not for the faint of heart.
- MarkMMullin 8y ago
  True - I often make the heretical argument that SL vs RL is just a question of where the labeling comes from :-) You are correct that the tooling is weaker in this space, but it is growing - my point is only that there's a difference between knowing how to use the tool and knowing how to make the tool - making the tool can liquify your brain -- using someone else's tool (assuming its a good tool) will simply give you headaches from time to time :-D