2 ms·
> For instance, the 7B model requires approximately 10 seconds for inference on 300 Chinese characters on an Nvidia A100 (40GB). Is that right? That sounds lik
by gwern 3y ago
> For instance, the 7B model requires approximately 10 seconds for inference on 300 Chinese characters on an Nvidia A100 (40GB).
Is that right? That sounds like, 2 OOMs higher than I would've expected. Is he doing something wrong like loading the model from a cold start?
- mergejoin 3y agoi was _very_ surprised by that number as well. 10s for inference on 300 chars? using an entire A100 GB GPU? that does not make any sense. OpenAI and other companies would NEVER be able to scale to 100s of millions of users if that was the real number, right?