3 ms·
>Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source. Heh, wha
by kgeist 5mo ago
>Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source.
Heh, what a coincidence, just today one of my students presented research results which also confirmed this. He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge.
- mlperson 5mo agoSounds very interesting!
- mahmoudimus 5mo agocan knowledge then be queried via tool? :)
- andai 5mo agogrep knowledge I'm thinking more like some kind of local wiki with an inverted index. Has anyone tried that? I know RAG isn't cool anymore and now we just do markdown files, but has anyone converted the useful parts of common crawl into .md ?
- andai 5mo agoHow does that work? Don't you need knowledge to understand the meaning of the inputs? Or is it the difference between, recognizing something vs recalling it being much more difficult? (Classification vs generation?)
- cheekygeeky 5mo ago> He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge. But not deterministic?
- HenryNdubuaku 5mo agoBullseye!
- lostmsu 5mo agoThat sounds giant! Any unformatted unfiltered preliminary records of said findings?