4 ms·
They mention previous work on speculative decoding using similar techniques, but "ANPD dynamically generates draft outputs via an adaptive N-gram module using r
by johntb86 2y ago
They mention previous work on speculative decoding using similar techniques, but "ANPD dynamically generates draft outputs
via an adaptive N-gram module using real-time
statistics, after which the drafts are verified by the
LLM. This characteristic is exactly the difference
between ANPD and the previous speculative decoding methods."