3 ms·
LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated
by tome 3y ago
LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.
- NorwegianDude 3y agoTechnically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token at the same time? Though it probably has it's own unforeseen challenges. Everything is predictable with enough guesses.
- _diq5 3y agoPeople are pretty cagey about what they use in production, but yes, speculative sampling can offer massive speedups in inference
- Aeolun 3y agoThey’re using several hundred cards here. Clearly there is ‘something’ that can be done in parallel.