3 ms·
you have described an RNN I think, don't attention heads add something that you could compare to rough &ready understanding?
by polotics 2y ago
you have described an RNN I think, don't attention heads add something that you could compare to rough &ready understanding?
- magicalhippo 2y agoAuto-regressive LLMs do this as I understand it, though it can vary if they feed the combined input and output[1] through the whole net like GPT-2 and friends, or just the decoder[2]. I described the former, and I should have clarified that. In either case you can "prime it" like it was suggested. A regular RNN has more feedback[3], like each layer feeding back to itself, as I understand it. Happy to be corrected though. [1]: https://jalammar.github.io/illustrated-gpt2/#one-difference-from-bert https://jalammar.github.io/illustrated-gpt2/#one-difference-... [2]: https://medium.com/@ikim1994914/understanding-the-modern-llm-part-2-understanding-auto-regression-and-auto-regression-training-5b255535f099 https://medium.com/@ikim1994914/understanding-the-modern-llm... [3]: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/