3 ms·
Speculative: PoC for speeding-up inference via speculative sampling by ggerganov
- brucethemoose2 3y agoInteresting, its almost like speculative execution on a CPU. A smaller model guesses the output of the larger model, and the larger model can effectively check and skip the correctly guessed part of the output.