4 ms·
Everyone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.
by londons_explore 3y ago
Everyone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.