3 ms·
That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.
by forgotTheLast 1mo ago
That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.
- clhodapp 1mo agoYup. You basically just need something for probability to push off of