6 ms·
Technically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token
by NorwegianDude 3y ago
Technically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token at the same time? Though it probably has it's own unforeseen challenges.
Everything is predictable with enough guesses.
- _diq5 3y agoPeople are pretty cagey about what they use in production, but yes, speculative sampling can offer massive speedups in inference