3 ms·
I just tested llama3:70b with ollama on my old AMD ThreadRipper Pro 3965WX workstation (16-core Zen4 with 8 DDR4 mem channels), with a single RTX 4090. Got 3.5
by tanelpoder 3y ago
I just tested llama3:70b with ollama on my old AMD ThreadRipper Pro 3965WX workstation (16-core Zen4 with 8 DDR4 mem channels), with a single RTX 4090.
Got 3.5-4 tokens/s, GPU compute was <20% busy (~90W) and the 16 CPU cores / 32 threads were about 50% busy.
- llm_trw 3y agoJesus that's the old one?
- reasonabl_human 3y agoAnd that’s not quantized at all, correct? If so, then the parent comment’s sentiment holds true…. Exciting stuff.