2 ms·
> saying it is 30x faster than the Hopper generation when inferencing a 1.8 trillion parameter mixture-of-experts model. Is that the size of gpt4? The upcomin
by kristianp 3y ago
> saying it is 30x faster than the Hopper generation when inferencing a 1.8 trillion parameter mixture-of-experts model.
Is that the size of gpt4? The upcoming systems seem designed to fit gpt4 on one system with 4bit quantisation, making gpt4 cheaper to deliver.
- sigmoid10 3y agoIt's the alleged size of GPT4. There's no official info anymore about architectures from OpenAI since they turned for-profit. But if the number is true, the new generation of cards still won't be able to run GPT4 on a single instance, because even with 4bit, it would need 900GB of VRAM just for the parameters. You'd need at least four B200s for that. And we're not even talking about activations yet.