4 ms·

I have to disagree with a good chunk of this article. The article completely misses the main advantage (to me) of reinforcement learning: Reinforcement learni

I have to disagree with a good chunk of this article. The article completely misses the main advantage (to me) of reinforcement learning: Reinforcement learning allows you to optimise on non-differentiable outcomes. I can't differentiate real life, but I want to optimise a process within real life. This feels tantalisingly close to AGI. If I can figure out a reward function, I can use reinforcement learning. Yes, this requires a reward function to be defined. Yes, this is a challenge to AGI. But to say that the big labs are not aware that this is a challenge to AGI is unfair. DeepMind is actively investigating open ended learning: https://deepmind.com/research/publications/open-ended-learning-symmetric-zero-sum-games https://deepmind.com/research/publications/open-ended-learni.... Just because the labs haven't tackled all the questions doesn't mean that they're not busy tackling difficult questions.

soVeryTired 5y ago
Can you clarify what you mean by 'non-differentiable outcomes'?
- ArnoVW 5y ago
  My guess: a function that you can evaluate but that you can not differentiate. Differentiate being "know the slope". Differentiable functions are great because you can run gradient descent on them in a very optimized way. Example: if your objective is to have a very high value, search in the direction that has a mounting slope. Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions. It just requires more evaluations.
  - LeegleechN 5y ago
    Neural network models have millions of parameters, which means you are trying to optimize in spaces with millions of dimensions. If you can't use differentiation you get hit really hard with the curse of dimensionality.
  - jeeeb 5y ago
    > Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions Isn't that exactly the point though? If you don't have an analytical solution for the gradient of the loss (reward) wrt the parameters - yes - you could brute force a numerical solution but as the number of parameters grows that quickly becomes infeasible. Approaches such as RL and GA provide a more intelligent way to search the parameter space.
toisanji 5y ago
DeepMind just published a paper "Reward is enough": Furthermore, we suggest that agents that learn through trial and error experience to maximise reward could learn behaviour that exhibits most if not all of these abilities, and therefore that powerful reinforcement learning agents could constitute a solution to artificial general intelligence.
- phonebucket 5y ago
  > DeepMind just published a paper "Reward is enough" That paper does not mean that DeepMind believes that reward is enough. It is one of their many papers that is seeking to espouse a point a view. It does not mean it is their exclusive point of view. For example, DeepMind are actively hiring for researchers in open-ended learning (check their website). This is not a new thing for them, e.g. they were quickly on the ball with apprenticeship learning too (https://arxiv.org/abs/1706.06617 https://arxiv.org/abs/1706.06617).

soVeryTired 5y ago

Can you clarify what you mean by 'non-differentiable outcomes'?

ArnoVW 5y ago
My guess: a function that you can evaluate but that you can not differentiate. Differentiate being "know the slope". Differentiable functions are great because you can run gradient descent on them in a very optimized way. Example: if your objective is to have a very high value, search in the direction that has a mounting slope. Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions. It just requires more evaluations.
- LeegleechN 5y ago
  Neural network models have millions of parameters, which means you are trying to optimize in spaces with millions of dimensions. If you can't use differentiation you get hit really hard with the curse of dimensionality.
- jeeeb 5y ago
  > Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions Isn't that exactly the point though? If you don't have an analytical solution for the gradient of the loss (reward) wrt the parameters - yes - you could brute force a numerical solution but as the number of parameters grows that quickly becomes infeasible. Approaches such as RL and GA provide a more intelligent way to search the parameter space.

toisanji 5y ago

DeepMind just published a paper "Reward is enough": Furthermore, we suggest that agents that learn through trial and error experience to maximise reward could learn behaviour that exhibits most if not all of these abilities, and therefore that powerful reinforcement learning agents could constitute a solution to artificial general intelligence.

phonebucket 5y ago
> DeepMind just published a paper "Reward is enough" That paper does not mean that DeepMind believes that reward is enough. It is one of their many papers that is seeking to espouse a point a view. It does not mean it is their exclusive point of view. For example, DeepMind are actively hiring for researchers in open-ended learning (check their website). This is not a new thing for them, e.g. they were quickly on the ball with apprenticeship learning too (https://arxiv.org/abs/1706.06617 https://arxiv.org/abs/1706.06617).