3 ms·
...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things.
by LeoPanthera 2y ago
...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things. They're completely different.
- guerrilla 2y agoYeah, over 65GB VRAM... that'd be expensive but not impossible. I think three RTX 4090's could do it, with their 24GB each.
- csomar 2y agoOnly 21.3GB is required.
- guerrilla 2y agoIs this wrong? It says 65.8GB. If it's wrong, what source should I be using instead? https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Instruct,6nvrT0uDyEPCowu5gDhQAA https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Inst...
- exe34 2y agothe ollama one is probably quantised.
- mistercheph 2y agoOllama's default quantization is q4_0 which is quite bad, but you can go to ollama's model page to see all the quantizations they have available, you can do e.g. "ollama run qwen2.5-coder:32b-instruct-q8_0" which will need ~35G + space for context
- tmikaeld 2y ago... with 4-bit quant and 4000 token context.
- mistercheph 2y agoYou really don't need models to fit into GPU's anymore for inference, e.g. llama.cpp is pretty good at partial GPU offload and I've gotten pretty fast results with only being able to fit ~30% of a model in VRAM and the rest in DDR5.
- mythz 2y ago> ...but you can't run it locally. Not unless you're sitting on some monster metal. I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRAM). Both GPUs cost less than USD $1,400 on eBay. qwen2.5-coder:32b is 19GB on ollama [1] [1] https://ollama.com/library/qwen2.5-coder:32b https://ollama.com/library/qwen2.5-coder:32b
- menaerus 2y agoI believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.
- mistercheph 2y agoor probably both? Seriously doubt 3.5 Sonnet is in the same order of magnitude as 32B
- menaerus 2y agoRight, it's almost certainly a magnitude larger than 32B.
- FloatArtifact 2y agoI'm curious if the M4 Max will be good enough.
- simonw 2y agoThat will run Qwen 2.5 Coder 32B just fine. I'm using a M2 Max with 64GB of RAM.
- FloatArtifact 2y agoHow many tokens per second?
- deleted 2y ago[deleted]