3 ms·
A massive problem with current generation llms is that they have a single globally ordered context and that the model is only allowed to append to the context.
by imtringued 1y ago
A massive problem with current generation llms is that they have a single globally ordered context and that the model is only allowed to append to the context.
This is like having a single tape Turing machine. They can simulate a multi tape machine, but at O(n^2) complexity.
The computation budget of an LLM is finite, so this has a massive practical impact.
- cubefox 1y agoThe article explains that this is not a problem but an advantage.
- Sabinus 1y agoApart from execution speedup due to caching, how does the article explain this is an advantage?
- mattnewton 1y agoI wouldn't agree it's an advantage per se, just that the caching workaround works better than you would expect! It's still quite slow to generate each token at a time versus diffusing batches, it's just hard to cache all the work you do in each encoder step.