4 ms·
> ...but you can't run it locally. Not unless you're sitting on some monster metal. I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRA
by mythz 2y ago
> ...but you can't run it locally. Not unless you're sitting on some monster metal.
I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRAM).
Both GPUs cost less than USD $1,400 on eBay.
qwen2.5-coder:32b is 19GB on ollama [1]
[1] https://ollama.com/library/qwen2.5-coder:32b https://ollama.com/library/qwen2.5-coder:32b
- menaerus 2y agoI believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.
- mistercheph 2y agoor probably both? Seriously doubt 3.5 Sonnet is in the same order of magnitude as 32B
- menaerus 2y agoRight, it's almost certainly a magnitude larger than 32B.
- FloatArtifact 2y agoI'm curious if the M4 Max will be good enough.
- simonw 2y agoThat will run Qwen 2.5 Coder 32B just fine. I'm using a M2 Max with 64GB of RAM.
- FloatArtifact 2y agoHow many tokens per second?
- simonw 2y agoWith MLX I got: Prompt: 49 tokens, 95.691 tokens-per-sec Generation: 723 tokens, 10.016 tokens-per-sec