5 ms·
The innovation is the amount of resources people are willing to spend right now. From looking at the research code it's clear that the whole field is basically
by delusional 2y ago
The innovation is the amount of resources people are willing to spend right now. From looking at the research code it's clear that the whole field is basically doing a (somewhat) guided search in the entire space of possible layer permutations.
There seems to be no rhyme or reason, no scientific insight, no analysis. They just try a million different permutations, and whatever scores the highest on the benchmarks gets published.
- moffkalast 2y agoWell it took evolution 4 billion years of testing out random permutations that resulted in a pretty good local maximum, so there is hope for us yet.
- greenchair 2y ago[flagged]
- stefs 2y agoOr maybe a watch made by a blind guy instead of a bridge?
- moffkalast 2y agoWould that make it more of a hear? Since obviously the guy can't watch.
- ben_w 2y agoI can't see the flagged comment, but I'm fairly sure @stefs is making a reference to this: https://en.wikipedia.org/wiki/The_Blind_Watchmaker https://en.wikipedia.org/wiki/The_Blind_Watchmaker
- stefs 2y agoyes. the flagged comment wanted to sell moffkalast a bridge, w/o further explanation. i interpreted it as him saying that the human brain isn't governed by evolution and calling moffkalast naive.
- WanderPanda 2y ago„I‘m a pretty good local maximum“ that is what any local maximum would tell you if asked how it likes itself
- moffkalast 2y ago"The brain is the most important part of the body", the brain said.
- psychoslave 2y agoNote that not all brains are so severely damaged with this illusion. Most of them actually get pretty clearly that they are next to useless without its organic, social and environmental companions.
- deleted 2y ago[deleted]
- TibbityFlanders 2y ago[dead]
- killerstorm 2y agoThere's definitely scientific insight and analysis. E.g. "In-context Learning and Induction Heads" is an excellent paper. Another paper ("ROME") https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 formulates hypothesis over how these models store information, and provide experimental evidence. The thing is, a 3-layer MLP is basically an associative memory + a bit of compute. People understand that if you stack enough of them you can compute or memorize pretty much anything. Attention provides information routing. Again, that is pretty well-understood. The rest is basically finding an optimal trade-off. These trade-off are based on insights based on experimental data. So this architecture is not so much accidental as it is general. Specific representations used by MLPs are poorly understood, but there's definitely a progress on understanding them from first principles by building specialized models.
- inciampati 2y agoOne 3-layer (1 hidden layer) neural network can already approximate anything. You don't even need to stack them.