3 ms·

What is "speculative decoding"?

• regularfry 2y ago
  Speculative decoding is using a small model to quickly generate a sequence that every so often you pass through a larger model to check and correct. It can be much faster than just using the larger model, with tolerably close accuracy.
  • qeternity 2y ago
    > with tolerably close accuracy. No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.
    • kgc 2y ago
      Is there a reference for this? I was wondering the same thing.
      • qeternity 2y ago
        Read the original whitepaper or go look at how any framework implements it. You will see that tokens not predicted by greedy sampling of the target model are rejected. Ergo, they are mathematically identical.