4 ms·
Neural networks are merely generalized methods used to translate data. These translations occur by multiplying / adding / dividing / etc weights to the input th
by lettergram 4y ago
Neural networks are merely generalized methods used to translate data. These translations occur by multiplying / adding / dividing / etc weights to the input then passing the output to the next node(s).
RNNs keep a memory of prior values such that you can pass on a “memory”.
At the end what this is doing is replacing components of the graph with mini-RNNs and pruning based on another network overseeing the first.
Having done quite a bit of work in this space I have a couple of thoughts.
What was the major advance in games? It was networks playing themselves.
Here we may want to do the same thing. Meta-learners need lots of “experience” (data) just like anything living.
This paper doesn’t dive too in-depth on the idea that the meta learner can be made general, but they can. There’s only so many problem types (classification, regression, generation, etc) and only so many data formats. Further, networks are fairly well defined, much more so than language.
This is the field of AutoML of anyones interested.
- melony 4y agoWhat's the reasoning behind choosing LSTMs here? Won't an attention model be even more effective?
- woadwarrior01 4y agoDid you see who the co-author of the paper is? :)
- Skyy93 4y agoAs you should know: Schmidhuber even invented the Transformer-Models, to be read in: https://arxiv.org/abs/2102.11174 https://arxiv.org/abs/2102.11174 ;)
- YorkshireSeason 4y agoThe claim of formal equivalence is restricted to "linear transformers". The tone of the comment seems to suggest that S.'s claim is slightly ridiculous. Can you point out a concrete technical shortcoming of the paper?
- Skyy93 4y agoNaaa, I just wanted to pointing out his attitude towards other research.
- YorkshireSeason 4y agoIt is valuable to point out connections with existing work, if only to avoid reinventing the wheel, and properly stand on the shoulders of gigants: אֵין כָּל חָדָשׁ תַּחַת הַשָּׁמֶשׁ (there is nothing new under the sun).
- Skyy93 4y agoYep, but you can do this without acting like an ass.
- melony 4y agoI bet OP was his student
- YorkshireSeason 4y agoHow much are you willing to bet?
- mlajtos 4y agoMy guess is that tiny LSTM is less computationally expensive than a tiny transformer. Since paper proposes replacing every weight with this tiny shared net, you want to be as tiny & performant as possible.
- lettergram 4y agoTransformers are more complicated than RNNs and require more fine tuning. I’m guessing the RNNs were used to simplify the problem. I’m not even sure transformers would work here given we’re theyre dropping them into a process
- laichzeit0 4y agoIf you want it tiny and performant and choose LSTM over transformers why not just skip LSTM and use GRUs?
- eru 4y ago> What was the major advance in games? It was networks playing themselves. I'm not sure. We have been using self-play since forever. That's how Alpha-Beta search works..
- yobbo 4y agoSelf-play probably refers to different agents playing each other, but that has also been done since forever. It just didn't seem that interesting when the models were small with games like 4x4 tac-tic-toe.
- _0ffh 4y agoTesauro used self-play to train a master level backgammon player nigh 30 years ago. https://www.semanticscholar.org/paper/TD-Gammon%3A-A-Self-Teaching-Backgammon-Program-Tesauro/1ea7603876a796f6007e4b6419cb1c9a9cb6bf8e https://www.semanticscholar.org/paper/TD-Gammon%3A-A-Self-Te...
- YorkshireSeason 4y ago> Neural networks are merely generalized methods used to translate data. That's true of all computation! To see this, consider lambda-calculus, which is a calculus of functions (i.e. things that convert inputs into outputs). The computational equivalence of lambda-calculus with Turing machines was one of the early computability theory.
- chaxor 4y agoYeah that cringy start to the OP's point removed quite a bit of credibility, despite stating 'they work in the area'. Probing towards a meta learner that is useful in many contexts is an intriguing idea though. I also wonder about the dream for meta learners and if they can approach a limit that is different than 'Adam-and-crew' when getting away from domain-specific parameter landscapes. I imagine much of the work will tend towards context switching to deal with effectively multiple domain specific optimizers, but I may be very far off from what is observed here.