4 ms·

Can someone knowledgeable please put these numbers in context with a comparison to an H100? For example they say 1053 tokens per second throughput with batch si

Can someone knowledgeable please put these numbers in context with a comparison to an H100? For example they say 1053 tokens per second throughput with batch size 4 on llama2 70b. Is that good?

qeternity 2y ago
Where do you see 1053 t/s for L2 70b? From their benchmark table, they achieve 1.8s response time at bs=4 for 256 output tokens, which would be 569 t/s throughput. At bs=1 you are memory bandwidth bound. So as a simple ceiling we divide the memory bandwidth by the model size (70b is roughly 140gb at fp16). That would yield roughly 24 t/s for H100 and 39 t/s for MI300x. I'm not really sure how GEMM optimizations alone would yield these sort of improvements at fp16. With quantization, where dequant overheads start to become compute bound, or at very high batch sizes, it makes sense and there has been a lot of work with various kernels in this area. But I'm a bit confused at fp16 and bs=1. Maybe someone smarter can chime in? EDIT: Ok I watched their video, which shows 39.75 t/s in the unoptimized (which checks out given the above rough maths) and 313.95 t/s for the optimized. I'm not really sure what's going on here.
- fancyfredbot 2y ago
  I found 1053 t/s in cell G18 in the throughput tab of the full results sheet they linked to: https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOzEI_ZZN1LpdY_bBSbvDOVPdfYslM/htmlview# https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOz...
  - qeternity 2y ago
    Hmm the post now links to this doc where I got my figures from: https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_ZZN1LpdY_bBSbvDOVPdfYslM/edit?gid=1853523197#gid=1853523197 https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_Z... This sheet only lists the response latency, but it does look more recent than the one you listed. In any event, the sheet you link to shows generation speed and throughput (generation speed is cell E18 which shows 527 t/s). G18 is perhaps prefill + gen but even that doesn't add up.
    - fancyfredbot 2y ago
      I noticed column G is always 2x column E. So I think they are counting both input and output tokens in the throughput for G but not for E. There's only one Google doc and you are linking to the same one I mentioned and the same one in the article... Not sure why you only see latency - there's definitely a throughput tab too.
      - qeternity 2y ago
        It's not though, it's close, but it's not. E18 is 527.75. 2 * 527.75 = 1055.5. G18 is 1053.16. So yeah, I have no idea.
- Lindon4290 2y ago
  Right. On that track, I want to confirm something. Maybe I am doing my math wrong or don't understand how transformers work. There is a video about the bs=1 case, i.e. a single prompt with input 256 tokens and output 256 tokens with a Llama-2 70B model, on a single MI300X (no tensor parallelism). The optimized result is giving 314 t/s and completes the entire request in 1.63s. Now, the Llama-2 70B is a an autoregressive decoded-only model. So, that means the entire model weights are processed for every generated token. The model weights are 128.48 GB (also shown in the video, and can be confirmed from HuggingFace). The card has 192GB of HBM, and the model entirely fits on the card. The HBM memory bandwidth for this card is 5.3 TB/s. Even assuming the prompt 256 tokens took no time, to generate 256 tokens, we'd need roughly 6s even at ideal memory bandwidth. (256 * 128.48e9) / (5.3e12). What's going on here?
latchkey 2y ago
https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/ https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3... Take that article and increase it by the gemm gains. At the end of the day, MI300x is outperforming an H100. More importantly is that you need 2xH100 to do 70B and only 1xMI300x.
- Lindon4290 2y ago
  Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq system is also a bs=1 apparatus and gets you around 300 tokens/s. Wild! [1] https://wow.groq.com/groq-sets-new-large-language-model-performance-record-of-300-tokens-per-second-per-user-on-meta-ai-foundational-llm-llama-2-70b/ https://wow.groq.com/groq-sets-new-large-language-model-perf...
  - latchkey 2y ago
    Someone else on reddit [0] noticed this as well. Groq does not talk about how many cards they need to get those results. Someone replied to me with this comment [1] a while ago... [0] https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/laq9xtm/ https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/... [1] https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
    - Lindon4290 2y ago
      Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https://twitter.com/swyx/status/1759759125314146699 https://twitter.com/swyx/status/1759759125314146699
      - latchkey 2y ago
        The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.

Can someone knowledgeable please put these numbers in context with a comparison to an H100? For example they say 1053 tokens per second throughput with batch size 4 on llama2 70b. Is that good?

qeternity 2y ago

Where do you see 1053 t/s for L2 70b? From their benchmark table, they achieve 1.8s response time at bs=4 for 256 output tokens, which would be 569 t/s throughput. At bs=1 you are memory bandwidth bound. So as a simple ceiling we divide the memory bandwidth by the model size (70b is roughly 140gb at fp16). That would yield roughly 24 t/s for H100 and 39 t/s for MI300x. I'm not really sure how GEMM optimizations alone would yield these sort of improvements at fp16. With quantization, where dequant overheads start to become compute bound, or at very high batch sizes, it makes sense and there has been a lot of work with various kernels in this area. But I'm a bit confused at fp16 and bs=1. Maybe someone smarter can chime in? EDIT: Ok I watched their video, which shows 39.75 t/s in the unoptimized (which checks out given the above rough maths) and 313.95 t/s for the optimized. I'm not really sure what's going on here.

fancyfredbot 2y ago
I found 1053 t/s in cell G18 in the throughput tab of the full results sheet they linked to: https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOzEI_ZZN1LpdY_bBSbvDOVPdfYslM/htmlview# https://docs.google.com/spreadsheets/u/0/d/1hHd9riWh-hQJnNOz...
- qeternity 2y ago
  Hmm the post now links to this doc where I got my figures from: https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_ZZN1LpdY_bBSbvDOVPdfYslM/edit?gid=1853523197#gid=1853523197 https://docs.google.com/spreadsheets/d/1hHd9riWh-hQJnNOzEI_Z... This sheet only lists the response latency, but it does look more recent than the one you listed. In any event, the sheet you link to shows generation speed and throughput (generation speed is cell E18 which shows 527 t/s). G18 is perhaps prefill + gen but even that doesn't add up.
  - fancyfredbot 2y ago
    I noticed column G is always 2x column E. So I think they are counting both input and output tokens in the throughput for G but not for E. There's only one Google doc and you are linking to the same one I mentioned and the same one in the article... Not sure why you only see latency - there's definitely a throughput tab too.
    - qeternity 2y ago
      It's not though, it's close, but it's not. E18 is 527.75. 2 * 527.75 = 1055.5. G18 is 1053.16. So yeah, I have no idea.
Lindon4290 2y ago
Right. On that track, I want to confirm something. Maybe I am doing my math wrong or don't understand how transformers work. There is a video about the bs=1 case, i.e. a single prompt with input 256 tokens and output 256 tokens with a Llama-2 70B model, on a single MI300X (no tensor parallelism). The optimized result is giving 314 t/s and completes the entire request in 1.63s. Now, the Llama-2 70B is a an autoregressive decoded-only model. So, that means the entire model weights are processed for every generated token. The model weights are 128.48 GB (also shown in the video, and can be confirmed from HuggingFace). The card has 192GB of HBM, and the model entirely fits on the card. The HBM memory bandwidth for this card is 5.3 TB/s. Even assuming the prompt 256 tokens took no time, to generate 256 tokens, we'd need roughly 6s even at ideal memory bandwidth. (256 * 128.48e9) / (5.3e12). What's going on here?

latchkey 2y ago

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/ https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3... Take that article and increase it by the gemm gains. At the end of the day, MI300x is outperforming an H100. More importantly is that you need 2xH100 to do 70B and only 1xMI300x.

Lindon4290 2y ago
Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq system is also a bs=1 apparatus and gets you around 300 tokens/s. Wild! [1] https://wow.groq.com/groq-sets-new-large-language-model-performance-record-of-300-tokens-per-second-per-user-on-meta-ai-foundational-llm-llama-2-70b/ https://wow.groq.com/groq-sets-new-large-language-model-perf...
- latchkey 2y ago
  Someone else on reddit [0] noticed this as well. Groq does not talk about how many cards they need to get those results. Someone replied to me with this comment [1] a while ago... [0] https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/laq9xtm/ https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/... [1] https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
  - Lindon4290 2y ago
    Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https://twitter.com/swyx/status/1759759125314146699 https://twitter.com/swyx/status/1759759125314146699
    - latchkey 2y ago
      The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.