5 ms·

>Whenever someone asks me if reinforcement learning can solve their problem, I tell them it can’t. I think this is right at least 70% of the time. This seems t

>Whenever someone asks me if reinforcement learning can solve their problem, I tell them it can’t. I think this is right at least 70% of the time. This seems to be a really strange calibration for "doesn't work". If you replace "reinforcement learning" with other well known technologies, and ask "of the instances where someone asks if X is a good solution, what percentage of the time is it actually a good solution?" I feel like 30% would be on the high end of the scale. The rest of the article seems has a lot of interesting discussion about RL's limitations, but it seems weird to make the article's thesis that RL doesn't work, rather than just "RL still has a lot of limitations".

deepGem 9y ago
I think the point he makes is that RL doesn't work as the world believes it to. RL is seen as the holy grail for AGI. He is saying that isn't the case, at least as of yet since RL algorithms don't generalize. For all other tasks, where RL can be used, other existing alternatives fare much better.
- stochastic_monk 9y ago
  That's a little unfair to RL. Reinforcement Learning is great even outside of autonomous control. [It's become quite important in NLP, for example.] It should be seen as a valuable set of approaches in a toolbox, not a silver bullet.
  - Eridrus 9y ago
    > [It's become quite important in NLP, for example.] [citation needed] Maybe it's the specific NLP tasks I've been paying attention to (goal oriented dialog), but most of the RL for NLP work I've seen has not been super impressive.
    - stochastic_monk 9y ago
      Williams, 1992. http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92s... This is even within dialogue generation. I’ve found plenty more recent citations, but it’s been a very important part of that subfield for decades. I can’t speak for goal-oriented dialogue generation, however. Perhaps your area of work has benefited less or given it less attention.
      - Eridrus 9y ago
        There's plenty of work applying RL techniques, it's just not actually useful for building real systems. It's basically in the same state as the rest of this blog post where it's so horribly sample inefficient you're usually better off with supervised learning if you're paying annotators. Or you're doing REINFORCE for some proxy metric or simulator which you're probably over-fitting and not actually improving your system with. The Alexa prize is basically the only situation where you're getting anywhere near enough RL rewards to be meaningful, because everywhere else the feedback is rare enough to not be helpful due to the sample inefficiency. Which is why I disagree with the characterization that it's important. There's a lot of it, and it can squeeze a little bit of performance on whatever dataset you're looking at, but it's had nowhere near the impact of, say, word vectors or (Bi)LSTMs.
        stochastic_monk 9y ago
        Thanks for the feedback! I'm more familiar with NLP through my colleagues and coursework, so I appreciate the real world experience.
hateduser2 9y ago
You don’t understand that quote. It’s saying 70% of the time when people think reinforcement learning is the solution it’s not. Not saying that 30% of all problems are solvable by reinforcement learning. Sometimes I can’t tell if people here you included are intentionally misreading things to further their agenda.
- neom 9y ago
  I don't give a shit and I read it as 30% of the time it works, so I guess i'm just a dumbass? :(
- mistercow 9y ago
  Did you misread my comment? I'm really confused how you could think that I was saying "of all problems", when I specifically said "of the instances where someone asks if X is a good solution".
- TeMPOraL 9y ago
  > Sometimes I can’t tell if people here you included are intentionally misreading things to further their agenda. More likely, sometimes people are just tired, distracted, or after consumption of alcohol/drugs. Personally, every other week I find myself writing a comment - sometimes long - and then deleting it a minute later, after re-reading the comment I was replying to and realizing I completely misunderstood it, and/or was arguing against a strawman of my own creation. And every other month I have a comment wrt. which I realize my mistake only few hours later.
GolDDranks 9y ago
If you read the article, I think that is a reference to the later passage where he notes that even RL models that are considered "working" still fail 30% percent of the time because the models are so unstable that their success depends on the random seed. If that's the level required, then he is allowed to say with confidence that they don't work, because even if it sometimes would, that would be way under 30% of the cases.
carlmr 9y ago
I find that any technology has limitations. We have the problem in industry that whenever there is a new buzzword there are "Thought leaders" in companies telling everybody to use it, because it's the new thing. Like when washing machines needed to have Fuzzy Logic in the 90s, just because it got kinda hyped. They didn't do anything better because of it, but marketing could print it on the front. Now you have AI, Industry 4.0/IoT, Blockchain, Big Data and the Cloud. Most of these technologies are useless to most problems, while being useful in a select few. However this won't stop marketing and sales departments from selling it to clients because it sounds good.
alexirpan 9y ago
Author here. My intent was to say, "When someone I know asks me if RL is a good idea, the answer is no >= 70% of the time." The important missing piece is that it's conditioned on that person asking me in particular. I don't have the largest profile, so the people who ask me about RL are generally well-informed about RL and have some practical ML experience too. Within that set, 70% felt like the right lower bound.

deepGem 9y ago

I think the point he makes is that RL doesn't work as the world believes it to. RL is seen as the holy grail for AGI. He is saying that isn't the case, at least as of yet since RL algorithms don't generalize. For all other tasks, where RL can be used, other existing alternatives fare much better.

stochastic_monk 9y ago
That's a little unfair to RL. Reinforcement Learning is great even outside of autonomous control. [It's become quite important in NLP, for example.] It should be seen as a valuable set of approaches in a toolbox, not a silver bullet.
- Eridrus 9y ago
  > [It's become quite important in NLP, for example.] [citation needed] Maybe it's the specific NLP tasks I've been paying attention to (goal oriented dialog), but most of the RL for NLP work I've seen has not been super impressive.
  - stochastic_monk 9y ago
    Williams, 1992. http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92s... This is even within dialogue generation. I’ve found plenty more recent citations, but it’s been a very important part of that subfield for decades. I can’t speak for goal-oriented dialogue generation, however. Perhaps your area of work has benefited less or given it less attention.
    - Eridrus 9y ago
      There's plenty of work applying RL techniques, it's just not actually useful for building real systems. It's basically in the same state as the rest of this blog post where it's so horribly sample inefficient you're usually better off with supervised learning if you're paying annotators. Or you're doing REINFORCE for some proxy metric or simulator which you're probably over-fitting and not actually improving your system with. The Alexa prize is basically the only situation where you're getting anywhere near enough RL rewards to be meaningful, because everywhere else the feedback is rare enough to not be helpful due to the sample inefficiency. Which is why I disagree with the characterization that it's important. There's a lot of it, and it can squeeze a little bit of performance on whatever dataset you're looking at, but it's had nowhere near the impact of, say, word vectors or (Bi)LSTMs.
      - stochastic_monk 9y ago
        Thanks for the feedback! I'm more familiar with NLP through my colleagues and coursework, so I appreciate the real world experience.

hateduser2 9y ago

You don’t understand that quote. It’s saying 70% of the time when people think reinforcement learning is the solution it’s not. Not saying that 30% of all problems are solvable by reinforcement learning. Sometimes I can’t tell if people here you included are intentionally misreading things to further their agenda.

neom 9y ago
I don't give a shit and I read it as 30% of the time it works, so I guess i'm just a dumbass? :(
mistercow 9y ago
Did you misread my comment? I'm really confused how you could think that I was saying "of all problems", when I specifically said "of the instances where someone asks if X is a good solution".
TeMPOraL 9y ago
> Sometimes I can’t tell if people here you included are intentionally misreading things to further their agenda. More likely, sometimes people are just tired, distracted, or after consumption of alcohol/drugs. Personally, every other week I find myself writing a comment - sometimes long - and then deleting it a minute later, after re-reading the comment I was replying to and realizing I completely misunderstood it, and/or was arguing against a strawman of my own creation. And every other month I have a comment wrt. which I realize my mistake only few hours later.

GolDDranks 9y ago

If you read the article, I think that is a reference to the later passage where he notes that even RL models that are considered "working" still fail 30% percent of the time because the models are so unstable that their success depends on the random seed. If that's the level required, then he is allowed to say with confidence that they don't work, because even if it sometimes would, that would be way under 30% of the cases.

carlmr 9y ago

I find that any technology has limitations. We have the problem in industry that whenever there is a new buzzword there are "Thought leaders" in companies telling everybody to use it, because it's the new thing. Like when washing machines needed to have Fuzzy Logic in the 90s, just because it got kinda hyped. They didn't do anything better because of it, but marketing could print it on the front. Now you have AI, Industry 4.0/IoT, Blockchain, Big Data and the Cloud. Most of these technologies are useless to most problems, while being useful in a select few. However this won't stop marketing and sales departments from selling it to clients because it sounds good.

alexirpan 9y ago

Author here. My intent was to say, "When someone I know asks me if RL is a good idea, the answer is no >= 70% of the time." The important missing piece is that it's conditioned on that person asking me in particular. I don't have the largest profile, so the people who ask me about RL are generally well-informed about RL and have some practical ML experience too. Within that set, 70% felt like the right lower bound.