3 ms·
Looking at Transformer model architecture and their training schemes to understand how they think misses the point. You wouldn't gain insight into how Doom game
by keskival 3y ago
Looking at Transformer model architecture and their training schemes to understand how they think misses the point. You wouldn't gain insight into how Doom game works by just looking at schematics of an x86 processor.
These systems are Turing complete and they can execute any computation. To see what they actually execute requires looking at what happens in memory while Doom runs, in the activations when the Transformer model executes.
At this time we don't have good tools to extract the actual algorithms and submodels these LLMs execute, but there are reasons to believe there are better algorithms learned in there for example for reinforcement learning than what we have ever been able to engineer.
See for example my research: https://github.com/keskival/king-algorithm-manifesto#readme https://github.com/keskival/king-algorithm-manifesto#readme
- skybrian 3y agoMechanistic interpretability is just getting started, but there's been some interesting research on induction heads. Here's an explanation: https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-heads-illustrated https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-...
- rcme 3y ago> You wouldn't gain insight into how Doom game works by just looking at schematics of an x86 processor. How is this a relevant analogy?
- arc619 3y agoThat was fascinating, thanks. Please do consider posting this here as its own article, it would be interesting to read others comments. My reading of this - and please correct me if I'm wrong, I'm still learning - is that you're extracting hyper-parameter planes from the data flow in the model's embedding space? Its really exciting to think of the hidden knowledge and relationships we could extract from our own linguistic interactions.