3 ms·
This is very wrong. Smaller models would generate response instantly on 4090. I run 3090's and easily get 30-50 tokens/seconds with small models. Folks with
by segmondy 2y ago
This is very wrong. Smaller models would generate response instantly on 4090. I run 3090's and easily get 30-50 tokens/seconds with small models. Folks with 4090 will see easily 80 tokens/sec for 7-8b model in Q8 and probably 120-160 for 3B models. Faster than most public APIs
- hedgehog 2y ago8B models are pretty fast even on something like a 3060 depending on deployment method (for example Q4 on Ollama).
- kolbe 2y agoThey might be talking about 70B
- nobodyandproud 2y agoWait, is 70b considered on the smaller side these days?
- kolbe 2y agoRelative to deepseek's latest, yeah
- imiric 2y agoSure, except those smaller models are only useful for some novelty and creative tasks. Give them a programming or logic problem and they fall flat on their face. As I mentioned in a comment upthread, I find ~30B models to be the minimum for getting somewhat reliable output. Though even ~70B models pale in comparison with popular cloud LLMs. Local LLMs just can't compete with the quality of cloud services, so they're not worth using for most professional tasks.