7 ms·

Someone else on reddit [0] noticed this as well. Groq does not talk about how many cards they need to get those results. Someone replied to me with this commen

• Lindon4290 2y ago
  Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1] https://twitter.com/swyx/status/1759759125314146699 https://twitter.com/swyx/status/1759759125314146699
  • latchkey 2y ago
    The exact cost of a mi300x is closely guarded by amd. I buy them and do not know how much they are. That said a whole chassis of 8x is far far far less than 10m.
    • Lindon4290 2y ago
      You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
      • latchkey 2y ago
        This is why I sponsored doing the chipsandcheese tests on my hardware. That instigated Elio to up the game even further. All open source by the way.
        • fancyfredbot 2y ago
          Thank you for sponsoring this. There's so little buzz about this hardware despite the fact it's clearly amazing for AI use cases. I don't understand why not. Maybe this is why Nvidia is the most valuable company in the world - nobody can be bothered to try a competitor.