4 ms·
Not training full attention might score nicely in benchmarks but humans will instantly notice the whole spectrum is not represented. What you are proposing is b
by treprinum 3y ago
Not training full attention might score nicely in benchmarks but humans will instantly notice the whole spectrum is not represented. What you are proposing is basically get rid of infrequent combinations but those happen in the real world and will be missing from whatever your LLM will produce.
- cypress66 3y agoYour typical LLM benchmarks simply do not test or use large context sizes. We need benchmarks for tasks that requiere large context sizes (like recalling facts and understanding them in long stories). I'm sure OpenAI have internal benchmarks for these tasks.
- jumpCastle 3y agohttps://github.com/google-research/long-range-arena https://github.com/google-research/long-range-arena
- DavidSJ 3y agoThe attention is sparse in a single attention head in a single layer, but the model still has a way to route information where it needs it given multiple attention heads and multiple layers. Similarly, your computer is not physically connected to mine but I'm still able to read your comment.