2 ms·
> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. My take is that p
by HarHarVeryFunny 3y ago
> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify.
My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience.
The ultimate intelligent architecture, for now, is our own cortex, which can be architecturally analyzed as a prediction machine - utilizing masses of perceptual feedback to correct/update predictions of how the perceptual scene, and results of our own actions, will evolve.
With prediction as the basis of intelligence, any model capable of predicting - to varying degrees of success - will be perceived to have a commensurate degree of intelligence. Transformer-based LLMs of course aren't the only possible way to predict, but they do seem significantly better at it than competing approaches such as Mamba or the RNN (LSTM etc) seq2seq approaches that were the direct precursor to the transformer.
I think the reason the transformer architecture is so much better than the alternatives, even if there are alternatives, is down to this specific way it does it - able to create these attention "keys" to query the context, and the ways that multiple attention heads learn to coordinate such as "induction heads" copying data from the context to achieve in-context learning.
- visarga 3y agoIf you invented the transformer but didn't have trillions of tokens to train it with, no chatGPT. But if you had Mamba/RWKV/SSSM and trillions of tokens you would have almost the same thing with chatGPT. The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intelligence is a social process, even though we like to think about keys and queries, or synapses and neurotransmitters, in fact it is the work of many people that made it possible. And language is that central medium between all of us, an evolutionary system of ideas, evolving at a much faster rate than biology. Now AI have become language replicators like humans, a new era in the history of language has begun. The same language trains humans and LLMs to achieve similar sets of abilities.
- HarHarVeryFunny 3y agoI agree about language - which might be though of as "thought macros". Human experience has taught us what things (objects, actions, etc) are worth labelling, what thought patterns are useful to reason about them, etc. Being able to reason about things in the realm of, and using the patterns of, human language is tremendously powerful. Are there any Mamba benchmarks that show it matching transformer (GPT, say) benchmark performance for similiar size models and training sets?
- visarga 3y agoI don't think there are Mamba LLMs larger than 2.8B at the moment. But here a crop of papers building on it, mostly vision applications: https://trendingpapers.com/search?q=mamba https://trendingpapers.com/search?q=mamba