4 ms·
eh, transformers are universal differentiable layered hash tables. that's incredibly powerful. most logic is just pulling symbols and matching structures with
by throwawaymaths 1y ago
eh, transformers are universal differentiable layered hash tables. that's incredibly powerful. most logic is just pulling symbols and matching structures with "hash"es.
if intelligence is just reasonable manipulation of logic it's unsurprising that an LLM could be intelligent, what maybe is surprising is that we have ~intelligence without going up a few more orders of magnitude in size, what's possibly more surprising is that training it on the internet got it doing the things it's doing
- almostgotcaught 1y ago[flagged]
- throwawaymaths 1y ago> I love these word-salad exaltations of LLMs - people don't even know what they're writing half the time but it sure is provocative! Well FWIW I have done an implementation of Llama LLM (3 8-70b to be specific) in non-python so I do sort of know what I'm talking about. I'm not the originator of the hash table analogy. I got it from here: https://www.youtube.com/watch?v=iDulhoQ2pro https://www.youtube.com/watch?v=iDulhoQ2pro Hell, the vectors that are generated are called K/V, so. Yeah it's a hash table. And the idea that first order logic and other facets of intelligence can just be cleverly arranged lookup tables comes from here: https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_Analogies https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_An...
- gpjt 1y agoAs the author of the original post above, let me say that if that's word salad, it's a Michelin star salad. Just the right mix of lettuce and tomato, and the dressing is spot on :-) Seriously, though, differentiable hash tables is an awesome way to look at them, I wish I'd heard it before.