4 ms·
"Running" and "acceptable inference speed and quality" are two different constraints, particularly at scale/production.
by minimaxir 3y ago
"Running" and "acceptable inference speed and quality" are two different constraints, particularly at scale/production.
- moonchrome 3y agoI don't understand what you're trying to say ? From what I've read 4090 should blow A100 away if you can fit within 22GB VRAM, which a 7B model should comfortably. And the latency (along with variability and availability) on OpenAI API is terrible because of the load they are getting.