5 ms·

Good idea. Generate 8 tokens with a small model, then give them to a large model as one batch (much faster) and if tokens 1..3 are in agreement but 4..8 nut, ta

Good idea. Generate 8 tokens with a small model, then give them to a large model as one batch (much faster) and if tokens 1..3 are in agreement but 4..8 nut, take only 1..4 and start again. Most tokens are easy to guess so you'll get x3.5 gains. I do have a feeling of dejavú, like I've seen this before on hn.

sdenton4 3y ago
Very reminiscent of parallel wavenet, which speed things up by generating multiple audio samples at a time.
milkey_mouse 3y ago
> I do have a feeling of dejavú, like I've seen this before on hn. You're either thinking of speculative encoding more generally, or Medusa: https://arxiv.org/abs/2401.10774 https://arxiv.org/abs/2401.10774
outofpaper 3y ago
I think a number of us are using the method and these folks decided to do a paper and release their findings.
- jimmySixDOF 3y ago
  Lots of speedups already on Llama3 like Groq (hardware arch) and Modal [1] getting 200 t/s From the below Modal link they use: >continuous batching, so multiple generations can take place at the same time on a single container >PagedAttention, which applies memory paging to the attention mechanism’s key-value cache, increasing throughput [1]https://modal.com/docs/examples/text_generation_inference https://modal.com/docs/examples/text_generation_inference
johntb86 3y ago
They mention previous work on speculative decoding using similar techniques, but "ANPD dynamically generates draft outputs via an adaptive N-gram module using real-time statistics, after which the drafts are verified by the LLM. This characteristic is exactly the difference between ANPD and the previous speculative decoding methods."
contravariant 3y ago
Just so I understand, this is essentially faster because reading all the model weights takes more time than calculating the model output?
- SonOfLilit 3y ago
  GPUs are great at doing the same math on every item of a large multidimensional array. Therefore, unsurprisingly, the cost per item of inference on a batch of items is significantly lower when the batch is e.g. 8 than 1 (in the case of Transformers there are further gains to be made because roughly half of the attention calculations in token k+1 are identical to the calculations of token k and can be easily reused by writing the formulas a certain way, the keyword to look for is causal attention mask). In any reasonable GPU inference setup the weights would be preloaded.
  - contravariant 3y ago
    Indeed GPUs are great at doing the same calculation in parallel. But if it was just that there should be enough opportunity to parallelise even without doing the exact same calculation multiple times. The main reason I can come up with why doing the same calculation 8 times in parallel instead of 8 times sequentially is that you benefit from better locality of reference.
    - SonOfLilit 3y ago
      As I said, the attention step is O(n^2) per token sequentially and O(n^2) for the entire sequence when calculating the entire sequence in parallel, where n is the length of the sequence.

sdenton4 3y ago

Very reminiscent of parallel wavenet, which speed things up by generating multiple audio samples at a time.

milkey_mouse 3y ago

> I do have a feeling of dejavú, like I've seen this before on hn. You're either thinking of speculative encoding more generally, or Medusa: https://arxiv.org/abs/2401.10774 https://arxiv.org/abs/2401.10774

outofpaper 3y ago

I think a number of us are using the method and these folks decided to do a paper and release their findings.

jimmySixDOF 3y ago
Lots of speedups already on Llama3 like Groq (hardware arch) and Modal [1] getting 200 t/s From the below Modal link they use: >continuous batching, so multiple generations can take place at the same time on a single container >PagedAttention, which applies memory paging to the attention mechanism’s key-value cache, increasing throughput [1]https://modal.com/docs/examples/text_generation_inference https://modal.com/docs/examples/text_generation_inference

johntb86 3y ago

They mention previous work on speculative decoding using similar techniques, but "ANPD dynamically generates draft outputs via an adaptive N-gram module using real-time statistics, after which the drafts are verified by the LLM. This characteristic is exactly the difference between ANPD and the previous speculative decoding methods."

contravariant 3y ago

Just so I understand, this is essentially faster because reading all the model weights takes more time than calculating the model output?

SonOfLilit 3y ago
GPUs are great at doing the same math on every item of a large multidimensional array. Therefore, unsurprisingly, the cost per item of inference on a batch of items is significantly lower when the batch is e.g. 8 than 1 (in the case of Transformers there are further gains to be made because roughly half of the attention calculations in token k+1 are identical to the calculations of token k and can be easily reused by writing the formulas a certain way, the keyword to look for is causal attention mask). In any reasonable GPU inference setup the weights would be preloaded.
- contravariant 3y ago
  Indeed GPUs are great at doing the same calculation in parallel. But if it was just that there should be enough opportunity to parallelise even without doing the exact same calculation multiple times. The main reason I can come up with why doing the same calculation 8 times in parallel instead of 8 times sequentially is that you benefit from better locality of reference.
  - SonOfLilit 3y ago
    As I said, the attention step is O(n^2) per token sequentially and O(n^2) for the entire sequence when calculating the entire sequence in parallel, where n is the length of the sequence.