4 ms·
Why don't we have LLM-Accelerator cards (yet)? Just seems like graphics cards are a good choice, but not a purpose-built choice.
by exabrial 6mo ago
Why don't we have LLM-Accelerator cards (yet)? Just seems like graphics cards are a good choice, but not a purpose-built choice.
- cjbprime 6mo agoInference (not training) is bottlenecked by memory access speed, not compute. Having special hardware wouldn't make it faster unless you somehow found a faster memory controller than the GPU has.