6 ms·
Qwen2.5 Coder 32B is great for an OSS model, but in my testing (ollama) Sonnet 3.5 yields noticeably better results, a lot more than what the provided benchmark
by mythz 2y ago
Qwen2.5 Coder 32B is great for an OSS model, but in my testing (ollama) Sonnet 3.5 yields noticeably better results, a lot more than what the provided benchmarks suggest.
Best thing about it is that it's an OSS model that can be hosted by anyone, resulting in an open competitive market bringing hosting costs down, currently sitting at $0.18/$0.18 M tok/s [1] making it 50x cheaper than Sonnet 3.5 and ~17x cheaper than Haiku 3.5.
[1] https://openrouter.ai/qwen/qwen-2.5-coder-32b-instruct https://openrouter.ai/qwen/qwen-2.5-coder-32b-instruct
- Tiberium 2y agoYeah, benchmarks are one thing but when you actually interact with the model it becomes clear very fast how "intelligent" the model actually is, by doing or noting small things that other models won't. 3.5 Sonnet v1 was great, v2 is already incredible.
- LeoPanthera 2y ago...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things. They're completely different.
- guerrilla 2y agoYeah, over 65GB VRAM... that'd be expensive but not impossible. I think three RTX 4090's could do it, with their 24GB each.
- csomar 2y agoOnly 21.3GB is required.
- guerrilla 2y agoIs this wrong? It says 65.8GB. If it's wrong, what source should I be using instead? https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Instruct,6nvrT0uDyEPCowu5gDhQAA https://llm.extractum.io/model/Qwen%2FQwen2.5-Coder-32B-Inst...
- exe34 2y agothe ollama one is probably quantised.
- mistercheph 2y agoOllama's default quantization is q4_0 which is quite bad, but you can go to ollama's model page to see all the quantizations they have available, you can do e.g. "ollama run qwen2.5-coder:32b-instruct-q8_0" which will need ~35G + space for context
- tmikaeld 2y ago... with 4-bit quant and 4000 token context.
- mistercheph 2y agoYou really don't need models to fit into GPU's anymore for inference, e.g. llama.cpp is pretty good at partial GPU offload and I've gotten pretty fast results with only being able to fit ~30% of a model in VRAM and the rest in DDR5.
- mythz 2y ago> ...but you can't run it locally. Not unless you're sitting on some monster metal. I'm getting a very usable ~18 tok/s running it on 2x NVIDIA A4000 (32GB VRAM). Both GPUs cost less than USD $1,400 on eBay. qwen2.5-coder:32b is 19GB on ollama [1] [1] https://ollama.com/library/qwen2.5-coder:32b https://ollama.com/library/qwen2.5-coder:32b
- menaerus 2y agoI believe that parent comment was referring to the point that Sonnet 3.5 cannot be run locally which it obviously cannot but not because of locally available compute but because it's not OSS.
- mistercheph 2y agoor probably both? Seriously doubt 3.5 Sonnet is in the same order of magnitude as 32B
- menaerus 2y agoRight, it's almost certainly a magnitude larger than 32B.
- FloatArtifact 2y agoI'm curious if the M4 Max will be good enough.
- simonw 2y agoThat will run Qwen 2.5 Coder 32B just fine. I'm using a M2 Max with 64GB of RAM.
- FloatArtifact 2y agoHow many tokens per second?
- deleted 2y ago[deleted]
- faangguyindia 2y agoToo bad zed editor doesn't have code completion via custom LLM So far I am using zed editor and can't switch to somethints else untill zed editor gets the update which support FIM directive via custom LLMs
- aargh_aargh 2y agoThis question keeps popping up but I don't get it. Everyone and their dog has an OpenAI-compatible API. Why not just serve a local LLM and put api.openai.com 127.0.0.1 in your hosts file?
- tbocek 2y agoThere is a difference between chat and code completion. While with chat, you can use localhost with llama.cpp, but code completion you cannot do that: https://github.com/zed-industries/zed/issues/12519 https://github.com/zed-industries/zed/issues/12519. The config for chat, you can do: "language_models": { "openai": { "version": "1", "api_url": "http://localhost:8080", "low_speed_timeout_in_seconds": 120, "available_models": [ { "provider": "openai", "name": "Qwen2.5-Coder-7B-Instruct-Q8_0.gguf", "display_name": "llama.cpp", "max_tokens": 131072 } ] } }, While for code completion, you have two choices atm: supermaven and copilot: https://zed.dev/docs/completions https://zed.dev/docs/completions.
- aargh_aargh 2y agoThank you, makes sense. I haven't used code completion yet.
- satvikpendem 2y agoYou could...switch editors? Why not do that until Zed gets that support?
- faangguyindia 2y agoZed is the best ai code editor with its editable context and super fast renders.
- Vetch 2y agoClaude Sonnet 3.5s are bars too high to clear. No other model comes close, with the occasional exception of o1-preview. But o1-preview is always a gamble, your rolls are limited and it will either be the best answer possible from an LLM or it returns after a wild goose chase, having talked itself into a tangled mess of confusion. I'd personally rank the Qwen2.5 32B model only a little behind GPT4o at worst, and preferable to gemini 1.5 pro 002 (at code only, Gemini is a model that's surprisingly bad at code considering its top class STEM reasoning). This makes Qwen2.5-coder-32B astounding all considered. It's really quite capable and is finally an accessible model that's useful for real work. I tested it on some linear algebra, discussed pros and cons of a belief propagation based approach to SAT solving, had it implement a fast simple approximate nearest neighbor based on the near orthogonality of random vectors in high dimensions (in OCaml, not perfect with but close enough to useful/easily correctable), simulate execution of a very simple recursive program (also Ocaml) and write a basic post processing shader for Unity. It did really well on each of those tasks.
- dragonsh 2y agoNot really tried the Claude 3.5, later tried o1-preview on github models and recently Qwen2.5 32B for a prompt to generate a litestar[0] app to manage a wysiwyg content using grapesjs[1] and use pelican[2] to generate static site. It generated very bad code and invented many libraries in import which didn't exist. Cluade was one of the worst code generator, later tried sieve of atkin to generate primes to N and then use miller-rabin test to test each generated prime both using all the cpu core available. Claude completely failed and could never get a correct code without some or the other errors especially using multiprocess, o1-preview got it right in first attempt, Qwen 2.5 32B got it right in 3'rd error fix. In general for some very simple code Claude is correct but when using something new it completely fails, o1-preview performs much better. Give a try to generate some manim community edition visualization using Claude, it generates something not working correct or with errors, o1-preview does much better job. In most of my test o1-preview performed way better than Claude and Qwen was not that bad either. [0] https://github.com/litestar-org/litestar https://github.com/litestar-org/litestar [1] https://grapesjs.com/ https://grapesjs.com/ [3] https://getpelican.com/ https://getpelican.com/
- mistercheph 2y agoIsn't it a bit unreasonable to compare something free that you can run today on totally standard consumer hardware with *the* state of the art LLM that is probably >= 1T parameters?
- mjwweb44 2y agoYes