3 ms·

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/ https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3... Take that article and increase

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/ https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3... Take that article and increase it by the gemm gains. At the end of the day, MI300x is outperforming an H100. More importantly is that you need 2xH100 to do 70B and only 1xMI300x.

Lindon4290 2y ago
Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq system is also a bs=1 apparatus and gets you around 300 tokens/s. Wild! [1] https://wow.groq.com/groq-sets-new-large-language-model-performance-record-of-300-tokens-per-second-per-user-on-meta-ai-foundational-llm-llama-2-70b/ https://wow.groq.com/groq-sets-new-large-language-model-perf...
- latchkey 2y ago
  Someone else on reddit [0] noticed this as well. Groq does not talk about how many cards they need to get those results. Someone replied to me with this comment [1] a while ago... [0] https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/laq9xtm/ https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/... [1] https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
  - Lindon4290 2y ago
    Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https://twitter.com/swyx/status/1759759125314146699 https://twitter.com/swyx/status/1759759125314146699
    - latchkey 2y ago
      The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.
      - Lindon4290 2y ago
        You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
        latchkey 2y ago
        This is why I sponsored doing the chipsandcheese tests on my hardware. That instigated Elio to up the game even further. All open source by the way.
        fancyfredbot 2y ago
        Thank you for sponsoring this. There's so little buzz about this hardware despite the fact it's clearly amazing for AI use cases. I don't understand why not. Maybe this is why Nvidia is the most valuable company in the world - nobody can be bothered to try a competitor.
fancyfredbot 2y ago
I can't square the numbers in the two articles at all. Other sources seem closer to their optimised results than their unoptimised results: llama3-70b throughput: Chips and Cheese: 4800 tok/sec (?!) This article: 849 tok/sec (fastest optimized) Mixtral 7B latency: Chips and cheese: 1.17s This article: 2.04s (fastest unoptimised) / 0.81 (fastest optimised) And then finally comparing mixtral 8x7b to the Reddit article using batch size 1... Reddit: 157 tok/sec This article: 165 tok/sec (optimized) or 141 tok/sec (unoptimised)
- fancyfredbot 2y ago
  For mixtral 7B latency, I think the explanation is just that chips and cheese used 128 token requests, and in the article it is 256 token requests I still can't explain throughput differences though

Lindon4290 2y ago

Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256 generation. The Groq system is also a bs=1 apparatus and gets you around 300 tokens/s. Wild! [1] https://wow.groq.com/groq-sets-new-large-language-model-performance-record-of-300-tokens-per-second-per-user-on-meta-ai-foundational-llm-llama-2-70b/ https://wow.groq.com/groq-sets-new-large-language-model-perf...

latchkey 2y ago
Someone else on reddit [0] noticed this as well. Groq does not talk about how many cards they need to get those results. Someone replied to me with this comment [1] a while ago... [0] https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/laq9xtm/ https://www.reddit.com/r/AMD_MI300/comments/1dqhrbn/comment/... [1] https://news.ycombinator.com/item?id=39966620 https://news.ycombinator.com/item?id=39966620
- Lindon4290 2y ago
  Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https://twitter.com/swyx/status/1759759125314146699 https://twitter.com/swyx/status/1759759125314146699
  - latchkey 2y ago
    The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.
    - Lindon4290 2y ago
      You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
      - latchkey 2y ago
        This is why I sponsored doing the chipsandcheese tests on my hardware. That instigated Elio to up the game even further. All open source by the way.
        fancyfredbot 2y ago
        Thank you for sponsoring this. There's so little buzz about this hardware despite the fact it's clearly amazing for AI use cases. I don't understand why not. Maybe this is why Nvidia is the most valuable company in the world - nobody can be bothered to try a competitor.

fancyfredbot 2y ago

I can't square the numbers in the two articles at all. Other sources seem closer to their optimised results than their unoptimised results: llama3-70b throughput: Chips and Cheese: 4800 tok/sec (?!) This article: 849 tok/sec (fastest optimized) Mixtral 7B latency: Chips and cheese: 1.17s This article: 2.04s (fastest unoptimised) / 0.81 (fastest optimised) And then finally comparing mixtral 8x7b to the Reddit article using batch size 1... Reddit: 157 tok/sec This article: 165 tok/sec (optimized) or 141 tok/sec (unoptimised)

fancyfredbot 2y ago
For mixtral 7B latency, I think the explanation is just that chips and cheese used 128 token requests, and in the article it is 256 token requests I still can't explain throughput differences though