7 ms·
A modern self-referential weight matrix that learns to modify itself
- goodmattg 5y agoNeed time to digest this paper, but you can assume if it's from Schmidhuber's group it will have some impact, even if only intellectual.
- TekMol 5y agoI have been playing with alternative ways to do machine learning on and off for a few years now. Some experiments went very well. I am never sure if it is a waste of time or has some value. If you guys had some unique ML technology that is different to what all the others do, what would you do with it?
- swagasaurus-rex 5y agoCreate a demo of it doing -something-. Literally anything. Then show it off and see where it goes.
- nynx 5y agoI’d make a blog and post about my experiments.
- andai 5y agoAnd a video too, please :)
- javajosh 5y agoHost it on a $5 VPS with full internet access and "see what happens".
- ggerganov 5y agoI would make a "Show HN" post
- deleted 5y ago[deleted]
- daveguy 5y agoDemo speaks louder than words. If you don't want to go into the details of how it works, it would still be interesting to just see where it over and under performs compared to existing systems.
- mark_l_watson 5y agoAbsolutely! Also, if possible, a Colab (or plain Jupiter notebook) and data would be good.
- Scene_Cast2 5y agoIf you do end up posting any sort of musings on this topic, I'd be really interested in taking a look.
- drewm1980 5y agoStart with the assumption that someone has already done it... Do a thorough literature survey... Ask experts working on the most similar thing. Don't be disheartened if you weren't the first; ideas don't have to be original to have value; some ideas need reviving from time to time, or were ahead of their time when first discovered.
- Szpadel 5y agoML is still fairly new topic and if you have some idea there is high chance that nobody actually tried it yet
- hwers 5y agoWrite a paper about it. Post it on arxiv.org. Contact some open minded researchers on twitter or here (show HN) for critique.
- nitrogen 5y agoYou have to be affiliated with an institution to submit to arxiv.
- rsfern 5y agoYou don’t strictly need an institutional affiliation, but it makes things easier for sure. Arxiv has an endorsement process you can go through to post without an affiliation https://arxiv.org/help/endorsement https://arxiv.org/help/endorsement
- jah242 5y agoSounds like we are in very similar positions and have a very similar question :). My only real plan so far is to try and beat or match SOTA on a recent benchmark from a large corporate / research lab, give them an email and hope they are willing to talk to you.
- Eliezer 5y agoDon't burn the capabilities commons. You probably don't have anything, in which case, why bother people? If you do have something, that advances AI capabilities and shortens the time before AGI; and while nobody actually has anything resembling a viable plan for surviving that, the fake plans tend to rely on having more time rather than less time.
- voldacar 5y agoA bit LARPy, don't you think?
- ur-whale 5y ago> what would you do with it? Use the "proof is in the pudding" method: Do something with it - preferably useful - that no one else can.
- nh23423fefe 5y agoIt's only a matter of time until the technological singularity > The WM of a self-referential NN, however, can keep rapidly modifying all of itself during runtime. In principle, such NNs can meta-learn to learn, and metameta-learn to meta-learn to learn, and so on, in the sense of recursive self-improvement. Everyone who doubts is hanging everything on "in principle" being too hard. Seems ridiculous to me, a failure of imagination.
- _Microft 5y ago"It suddenly stopped self-improving." "What happened?" "It ... it looks like it found a way to autogenerate content that is optimized for its reward function and now binges on it 24/7..." ><
- cwmoore 5y ago"It seems to have optimized for reruns of Breaking Bad, and exhibits markedly elevated energy consumption during the car chase scenes."
- synquid 5y agoSchmidhuber has written about recursive self-improvement since his diploma thesis in the 80s: "Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook". Your quote sounds like it could just as well have been from that thesis.
- deleted 5y ago[deleted]
- erdos4d 5y agoEven if the algorithms were here to do that job, where will the hardware come from? I'm staring at almost a decade of commercial processor (i7) lineage right now and the jump has been from 4 to 6 cores with no change in clock speed (maybe the newer one is even slower actually). There definitely won't be any singularity jumping off unless today's hardware gets another few decades of Moore's law, and that is not happening.
- jdeaton 5y agoI'm having a hard time reading this paper without hearing you-again's voice in my head.
- ricardobayes 5y agoIt's a super weird feeling to click on a hacker news top post and find out I know one of the authors. The world is a super small place.
- watersb 5y agoFirst, congratulations! It's a paper worth HN attention. Very cool. Second: do Hacker News posts form a small-world network? I don't know. I don't even know if my question is well posed (it might be a meaningless question). Does the set of Hacker News articles change over time in ways that resemble annealing or self-training matrices? (likewise, I question this question, but I wonder.) https://en.m.wikipedia.org/wiki/Small_world_network https://en.m.wikipedia.org/wiki/Small_world_network
- mark_l_watson 5y agoI haven't really absorbed this paper yet, but first thoughts were Hopfield Networks we used in the 1980s. For unsupervised learning algorithms like masked models (BERT and some other Transformers), it makes sense to train in parallel with prediction. Why not? My imagination can't wrap around using this for supervised (labeled data) learning.
- codelord 5y agoI haven't read the paper yet, no comment on the content. But it's amusing that more than 30% of references are self-citation.
- lol1lol 5y agoHinton et al. self cite. Schmidhuber et al. self cite. One got Turing, the other got angry.
- deleted 5y ago[deleted]
- heyitsguay 5y agoI know Schmidhuber is famously miffed for missing out on the AI revolution limelight, and despite that he runs a pretty famous and well-resourced group. So with a paper like this demonstrating a new fundamental technique, you'd think they would eat the labor and compute costs of getting this up and running on a full gauntlet of high-profile benchmarks, in comparison with existing SOTA methods, vs the sort of half-hearted benchmarking that happens in this paper. It's a hassle, but all it would take for something like this to catch the community's attention would be a clear demonstration of viability in line with what groups at any of the other large research institutions do. The failure to put something like that front and center makes me wonder how strong the method is, because you have to assume that someone on the team has tried more benchmarks. Still, the idea of learning a better update rule than gradient descent is intriguing, so maybe something cool will come from this :)
- P-NP 5y ago"miffed for missing out on the AI revolution limelight?" Despite all those TV docs and newspaper articles about him? :-)
- nullc 5y agoOr they hurried the publication to avoid getting scooped and will follow up with interesting benchmarks later.
- taneq 5y agoIf it’s really that new and different, maybe it’d be a little premature and even misleading to present the sort of full sweep you suggest. People are much better at pooh-poohing new ideas than at accurately assessing their potential.
- savant_penguin 5y agoJust skimmed the paper but the benchmarks are super weird