4 ms·

It's a shame how data hungry DRL is (even when compared to DL), but the DRL framework encompasses all the standard classification/regression tasks and also incl

It's a shame how data hungry DRL is (even when compared to DL), but the DRL framework encompasses all the standard classification/regression tasks and also includes decision making, planning and pretty much anything else you can think of. Model generality and data efficiency are in an inverse relationship and a lot of research has been in moving up or down this hyperbola. On one extreme, tailoring models to specific use cases/datasets/environments, on the other end transferring learning across domains. DRL is stuck pretty high on that generality end. Some breakthroughs seemed to have moved progress to a higher level curve, optimizers (Adam, DQN, TRPO) have gotten better which helps everything in general, core structures like CNNs or memory cells, which seem to be somewhat universal (or our best guess yet), but there's still something fundamental that seems to be missing. Or maybe this is all there is and we just need a computer with a richer/higher-resolution sense field and the flops to process them.

deepGem 9y ago
but the DRL framework encompasses all the standard classification/regression tasks and also includes decision making Except labelling data, correct ? which is non trivial effort.
- gwern 9y ago
  > Except labelling data, correct ? which is non trivial effort. Choosing choice of labeling is a RL problem too. If you're choosing environment actions to learn how the environment 'labels' them, that's the classic topic of 'how do we make DRL models explore' well (and arguably is the Achilles heel of the model-free DRL approaches OP is criticizing: the NNs can easily learn to optimize their actions, even tiny NNs are more than enough, but they just don't get fed the 'right' data ie. exploration is bad). Relevant papers: https://www.reddit.com/r/reinforcementlearning/search?q=flair%3AExp&sort=new&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... If you're being very narrow and considering a classification problem, well, that's a RL problem too: you can optimize which datapoints you get labels for based on how informative a datapoint is (most datapoints are simply redundant) or how expensive it is to label. That's called 'active learning': https://www.reddit.com/r/reinforcementlearning/search?q=flair%3AActive&sort=new&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... It's particularly natural if you are doing large-scale image classification and have a service like Amazon Turk plugged in to get (or correct) labels.
- PeterisP 9y ago
  In many cases people greatly overestimate the effort of labelling data. For the price of just a single engineer man-month you can generally label a lot of data, sufficient for many tasks; especially if you can do bootstrapping (the humans don't label all data, but correct mistakes of a previous insufficiently good automated solution) or some transfer learning from a somewhat similar task or an unsupervised one. Like, if you need a solution for a some niche of image classification, even a single afternoon of labeling data might be sufficient to adapt an ImageNet classifier to your particular labels and get reasonable accuracy.
tanilama 9y ago
DRL is too fragile to be realistically useful while supervised learning is not.
eli_gottlieb 9y ago
>It's a shame how data hungry DRL is (even when compared to DL), but the DRL framework encompasses all the standard classification/regression tasks and also includes decision making, planning and pretty much anything else you can think of. Sure, and a Turing machine can express any program you care to write. RL seems like the Turing tar pit of machine learning: theoretically able to express everything, practically convenient mostly just for trivial examples. >Model generality and data efficiency are in an inverse relationship and a lot of research has been in moving up or down this hyperbola. Sure, basic Curse of Dimensionality, but the whole success of hierarchical modeling in the brain, hierarchical Bayesian methods, and deep neural networks has been that hierarchical modeling seems to ameliorate or even defeat the Curse of Dimensionality. The question is: well, why can't it do that in reinforcement learning? In interesting terms: why does the teaching signal have more information in supervised learning than reinforcement learning, relative to the inherent uncertainty of the task?

deepGem 9y ago

but the DRL framework encompasses all the standard classification/regression tasks and also includes decision making Except labelling data, correct ? which is non trivial effort.

gwern 9y ago
> Except labelling data, correct ? which is non trivial effort. Choosing choice of labeling is a RL problem too. If you're choosing environment actions to learn how the environment 'labels' them, that's the classic topic of 'how do we make DRL models explore' well (and arguably is the Achilles heel of the model-free DRL approaches OP is criticizing: the NNs can easily learn to optimize their actions, even tiny NNs are more than enough, but they just don't get fed the 'right' data ie. exploration is bad). Relevant papers: https://www.reddit.com/r/reinforcementlearning/search?q=flair%3AExp&sort=new&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... If you're being very narrow and considering a classification problem, well, that's a RL problem too: you can optimize which datapoints you get labels for based on how informative a datapoint is (most datapoints are simply redundant) or how expensive it is to label. That's called 'active learning': https://www.reddit.com/r/reinforcementlearning/search?q=flair%3AActive&sort=new&restrict_sr=on https://www.reddit.com/r/reinforcementlearning/search?q=flai... It's particularly natural if you are doing large-scale image classification and have a service like Amazon Turk plugged in to get (or correct) labels.
PeterisP 9y ago
In many cases people greatly overestimate the effort of labelling data. For the price of just a single engineer man-month you can generally label a lot of data, sufficient for many tasks; especially if you can do bootstrapping (the humans don't label all data, but correct mistakes of a previous insufficiently good automated solution) or some transfer learning from a somewhat similar task or an unsupervised one. Like, if you need a solution for a some niche of image classification, even a single afternoon of labeling data might be sufficient to adapt an ImageNet classifier to your particular labels and get reasonable accuracy.

tanilama 9y ago

DRL is too fragile to be realistically useful while supervised learning is not.

eli_gottlieb 9y ago

>It's a shame how data hungry DRL is (even when compared to DL), but the DRL framework encompasses all the standard classification/regression tasks and also includes decision making, planning and pretty much anything else you can think of. Sure, and a Turing machine can express any program you care to write. RL seems like the Turing tar pit of machine learning: theoretically able to express everything, practically convenient mostly just for trivial examples. >Model generality and data efficiency are in an inverse relationship and a lot of research has been in moving up or down this hyperbola. Sure, basic Curse of Dimensionality, but the whole success of hierarchical modeling in the brain, hierarchical Bayesian methods, and deep neural networks has been that hierarchical modeling seems to ameliorate or even defeat the Curse of Dimensionality. The question is: well, why can't it do that in reinforcement learning? In interesting terms: why does the teaching signal have more information in supervised learning than reinforcement learning, relative to the inherent uncertainty of the task?