4 ms·

For all the 'I know every number' certainty of this post, there's some weird stuff: >(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $2

by RC_ITR 3y ago

For all the 'I know every number' certainty of this post, there's some weird stuff: >(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour.) Why flex both system size and training time to arbitrary numbers? >For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. This means parts may sit dormant when other parts are being used. When serving users, this really hurts utilization rates. Utilization of what? Memory? If you're that worried about inference utilization, then why not just fire up a non-MOE model? Here's what the post said about MQA: >Because of that only 1 head is needed and memory capacity can be significantly reduced for the KV cache This is close but wrong. You only need one Key and Value (KV) head, but you still have the same amount of query heads. My guess is that this is all a relatively knowledgeable person, using formulas laid out by the 2020 scaling paper and making a fantasy system (with the correct math), based on that. Put differently, I could probably fake my way through a similar post and be an equal level of close but definitely wrong because I'm way out of my league. That vibe makes me very suspicious.

deleted 3y ago
[deleted]
moconnor 3y ago
No, the post is correct about MQA. A KV-cache only caches the key and value heads. The point of MQA is that your KV-cache is 1/heads smaller than usual because of this sharing. Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.
- RC_ITR 3y ago
  >Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses multiple query heads but single key and value heads. Emphasis mine, source here [0] [0] https://arxiv.org/pdf/2305.13245.pdf https://arxiv.org/pdf/2305.13245.pdf FWIW the original MQA paper is called One Write head is all you need. Here's the quote from that referencing multiple heads [1] >We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline. [1]https://arxiv.org/pdf/1911.02150.pdf https://arxiv.org/pdf/1911.02150.pdf

by RC_ITR 3y ago

deleted 3y ago

[deleted]

moconnor 3y ago

No, the post is correct about MQA. A KV-cache only caches the key and value heads. The point of MQA is that your KV-cache is 1/heads smaller than usual because of this sharing. Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.

RC_ITR 3y ago
>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses multiple query heads but single key and value heads. Emphasis mine, source here [0] [0] https://arxiv.org/pdf/2305.13245.pdf https://arxiv.org/pdf/2305.13245.pdf FWIW the original MQA paper is called One Write head is all you need. Here's the quote from that referencing multiple heads [1] >We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline. [1]https://arxiv.org/pdf/1911.02150.pdf https://arxiv.org/pdf/1911.02150.pdf