3 ms·
I kept expecting the phrase "reinforcement learning" to show up, but even after mentioning a Bellman-style time-decayed sum over future rewards, it doesn't appe
by abeppu 5y ago
I kept expecting the phrase "reinforcement learning" to show up, but even after mentioning a Bellman-style time-decayed sum over future rewards, it doesn't appear, which seems odd to me. And roughly, I think "runs" == "exploitation", "tumbles" == "exploration" in terms other people use.
A thing that the author glosses past is the relationship between having a model of the world and maximizing rewards. It's taken as given that a model-based approach leads to better predictions about rewards ... but when we try to get systems to actually solve problems, this often isn't the case. Learning policies which yield high rewards is often easier than learning a model which actually describes the environment. I think of value iteration as learning "I feel this is a good state", policy iteration as "I feel this is a good action", and a real model as implicitly providing "I think this is a good action because I expect that ..."
If artificial systems have the greatest successes with model-free approaches following algorithms very similar to the one the author describes, but humans seem to have a rich model _and_ do better than those artificial systems (at least accounting for data efficiency), doesn't that at least suggest that we're thinking differently?
- Enginerrrd 5y agoI think it really just suggests the models are being trained improperly. We just don't train alogorithms on data as complex as the real world. So, it's not surprising that simpler (less correct) models win out over more accurate nuanced ones.
- abeppu 5y agoMy take is more that the framing of maximizing an externally provided reward describing a single task only helps an agent who can rely on the rules of that task remaining fixed. Humans don't just try to do one task well; we switch between tasks. We don't have episodes that reset really; we have one long episode, and our actions decades ago can have bearing on our well-being today. What is a good state or a good policy varies substantially over time. The concept of the value of a state or (state, action) is ambiguous, because who knows what you'll want tomorrow? If the definition of value shifts, but the rules of the world don't, then a modeling becomes more "valuable".
- iroh2727 5y agoYep, this is very similar to my take as well. Also, now that I think of it, although we often talk about how much more data-efficient humans are compared to ML models, we could say that we're actually much more data-inefficient than most (or all) other animals. Our gestation period is long and then we're totally useless as a baby. We require a lot of learning and development early on in order to reap the benefits later (which consist of generality and flexibility like you mentioned and also, for example, being able to generate our own synthetic data via imagination and counterfactual reasoning).
- skybrian 5y agoModel-free systems are easier to build, but researchers are working on model-based approaches to reinforcement learning and I remember seeing blog posts last year describing some promising results. It sounded like it was still reinforcement learning, though.