7 ms·
Prefrontal cortex as a meta-reinforcement learning system
- spitfire 8y agoPaper is behind a paywall. Sci-Hub doesn't have it. Does anyone have a link to the paper PDF?
- dcre 8y agohttps://www.biorxiv.org/content/early/2018/04/13/295964 https://www.biorxiv.org/content/early/2018/04/13/295964 (I searched it on Google Scholar.)
- ColanR 8y agohttp://sci-hub.tw/10.1038/s41593-018-0147-8 http://sci-hub.tw/10.1038/s41593-018-0147-8
- gumby 8y agoScihub link: https://sci-hub.tw/10.1038/s41593-018-0147-8 https://sci-hub.tw/10.1038/s41593-018-0147-8
- mindcrime 8y agoSo here's something interesting to note. The "buy this paper" price for this article is almost reasonable in and of itself. $22.00. In fact, for somebody who works in the field or even a tangentially connected field, they could probably justify spending the $22.00 for it. But... hilariously, a one year subscription to the journal, is only $59.00 (for individuals, online only). https://www.nature.com/neuro/subscribe https://www.nature.com/neuro/subscribe Something seems a bit out of whack when the price of one article is about 1/3rd of the price of the journal for an entire yet. I'm almost tempted to subscribe. The problem with this model though, is that it doesn't scale. Even at "only" $59.00, how many journals can one afford to subscribe to before the aggregate cost breaks the bank? sigh
- keithwhor 8y agoIt scales because nobody subscribes individually. If you can't access a journal, you get your department head to forward things up the chain so your institution can just add another line item to the millions of dollars they spend annually on journal subscriptions for every researcher and student. AFAIK almost all of that money goes directly to Elsevier [1] who, well, they pretty much run a monopoly. [1] https://en.wikipedia.org/wiki/Elsevier https://en.wikipedia.org/wiki/Elsevier
- mindcrime 8y agoWhat I mean is, for those people (like me) who don't have a "department head" (or a university at all) to take care of journal subscriptions, and who might be willing to subscri be individually - it doesn't scale beyond a journal to two. At most, in years past, I subscribed to maybe a total of 10 journals, but that's when I was maintaining an IEEE membership and a lot of their journals are actually not very expensive, in terms of the marginal cost once you are already a dues paying member. These days though, I find it hard to justify. I find myself wondering if the publishers would do a better job of providing a model that works for individuals, if they might not actually make more money in the end.
- davidjnelson 8y agoThat is so darn insightful! So check it out, if there were a way to pay $20/mo for access to ~5 publications/mo from a list of hundreds/thousands, that would be fantastic! Netflix for research. I'd pay for it. It wouldn't be terribly challenging to build. The tricks are in making the deals with the content owners, and getting the word out to users. The total market is 137 Billion/year ( https://www.ibisworld.com/industry-trends/market-research-reports/professional-scientific-technical-services/professional-scientific-technical-services/scientific-research-development.html https://www.ibisworld.com/industry-trends/market-research-re... ) Not bad. Looks like half the market is the government. Might be enough there to build a business on. Hard to say. Requires more research.
- 8y ago
- dcre 8y agoFull preprint available here: https://www.biorxiv.org/content/early/2018/04/13/295964 https://www.biorxiv.org/content/early/2018/04/13/295964
- cs702 8y agoVery interesting. Oversimplifying and ignoring a lot of important details, the key idea proposed by the authors is that the brain's phasic dopamine system is a model-free reinforcement-learning system that learns to train the prefrontal cortex as a more efficient model-based reinforcement-learning sytem -- a form of meta-learning which the authors accurately refer to as meta-reinforcement learning. The "Results" section provides compelling evidence that the authors might be on to something. The authors show and discuss the outcomes of six different kinds of computer experiments in which a (relatively simple) meta-reinforcement learning software system is shown to learn and behave in qualitatively similar ways as, for example, monkeys and rats in equivalent lab experiments. I'm still digesting the implications. Highly recommended reading.
- epicureanideal 8y agoAlthough the behavior of the monkeys and rats was simply selecting the correct image to receive a reward, so not an especially complicated task.
- red75prime 8y agoWell, it took 50+ years to solve this simple problem somewhat reliably.
- dlwdlw 8y agoTo simplify even further, would it be right to say that dopamine is not just a carrot being used to adjust weights but that the entire dopamine system actively learns how/when/why to use the carrot to effectively train? if so, some interesting real world parallels i can see: 1/ this is why good teachers early on have such a profound effect. Teachers act as a dopamine system so good teachers teach meta learning via whatever they teach 2/ teaching the meta directly is difficult as the meta of the meta is too abstract. Which is why grammar/math may tend to feel out of touch. They are already the meta. 3/ drugs can mess up the dopamine system and throw the feedback loop out of whack. Even if you have the best resources, the “teacher” is now inept
- epberry 8y ago
- mr_overalls 8y agoA more accessible discussion by the paper's authors is here at DeepMind: https://deepmind.com/blog/prefrontal-cortex-meta-reinforcement-learning-system/ https://deepmind.com/blog/prefrontal-cortex-meta-reinforceme...