Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This gua

Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This guarantees exactly the same quality with a big speedup, as you don't need to predict with the large model each individual token. Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318