4 ms·
So, the way speculative decoding works, the model begins predicting at the first wrong token, so you still get 'is' for free.
by cwyers 1y ago
So, the way speculative decoding works, the model begins predicting at the first wrong token, so you still get 'is' for free.