3 ms·
> If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency. Good advice. Does that mean that I can install li
by habibur 4y ago
> If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency.
Good advice. Does that mean that I can install like 64 gb ram on a PC and run those models in comparable time?
- fxtentacle 4y agoThat's how cloud speech recognition is usually deployed. OpenAI whisper is faster than realtime on regular desktop CPUs, which I guess is good enough. And for a datacenter, a few $100 AMD CPUs will beat a single $20k NVIDIA A100 at throughput per dollar.