5 ms·
Qwen2.5-Coder-32B is an LLM that can code well that runs on my Mac
- manamorphic 2y agoI heard conflicting things about it. Some claim it was trained so it can do well on benchmarks and in real world scenarios it's lacking. Can somebody deny/confirm ?
- csomar 2y agoI tried the Qwen2.5 32B a couple weeks ago. It was amazing for a model that can run on my laptop but far from Claude/GPT-4o. I am downloading the coder tuned version now.
- anonzzzies 2y agoIt's small... And for that size, it does very well. Been using it a few days and it's quite good for it's size and the fact you can run it locally. So not sure if it's true what you say; for us it works really well.
- dodslaser 2y agoWhat else should you train for? If the benchmark dosn't represent real world scenarios, isn't that a problem with the benchmark rather than the model?
- exitb 2y agoOverfitting is a concern.
- isoprophlex 2y agoIf your benchmark covers all possible programming tasks then you dont need an llm, you need search over your benchmark. Hypothetically let's say the benchmark contains "test divisibility of this integer by n" for all n of the form 3x+1. An extremely overfit llm won't be able to code divisibility for all n not of the form 3x+1, and your benchmark will never tell.
- YetAnotherNick 2y agoNo, because solving a well defined problem with well defined right or wrong is generally not what people use llm for. Most of the times my query to llm is underspecified, and lot of time I figure out the problem when chatting with LLM. And benchmark by definition only measures just right/wrong answer.
- LeoPanthera 2y agoThis is called Goodhart's law, who said: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." But in modern usage it is often rephrased to: "When a measure becomes a target, it ceases to be a good measure" https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- ithkuil 2y agoIt's more subtle than this: https://en.m.wikipedia.org/wiki/Training,_validation,_and_test_data_sets https://en.m.wikipedia.org/wiki/Training,_validation,_and_te...
- tyler33 2y agoi tried qwen and it is surprinsingly good, maybe not as good as claude but could replaced it
- mythz 2y agoQwen2.5 Coder 32B is great for an OSS model, but in my testing (ollama) Sonnet 3.5 yields noticeably better results, a lot more than what the provided benchmarks suggest. Best thing about it is that it's an OSS model that can be hosted by anyone, resulting in an open competitive market bringing hosting costs down, currently sitting at $0.18/$0.18 M tok/s [1] making it 50x cheaper than Sonnet 3.5 and ~17x cheaper than Haiku 3.5. [1] https://openrouter.ai/qwen/qwen-2.5-coder-32b-instruct https://openrouter.ai/qwen/qwen-2.5-coder-32b-instruct
- Tiberium 2y agoYeah, benchmarks are one thing but when you actually interact with the model it becomes clear very fast how "intelligent" the model actually is, by doing or noting small things that other models won't. 3.5 Sonnet v1 was great, v2 is already incredible.
- LeoPanthera 2y ago...but you can't run it locally. Not unless you're sitting on some monster metal. It's tiresome when people compare enormous cloud models to tiny little things. They're completely different.
- guerrilla 2y agoYeah, over 65GB VRAM... that'd be expensive but not impossible. I think three RTX 4090's could do it, with their 24GB each.
- tucnak 2y agoThe issue with some recent models is that they're basically overfitting on public evals, and it's not clear who's the biggest offender—OpenAI, or the Chinese? And regardless, "Mandelbrot in plaintext" is a bad choice for evaluation's sake. The public datasets are full of stuff like that. You really want to be testing stuff that isn't overfit to death, beginning with tasks that notoriously don't generalise all too well, all the while being most indicative of capability: like, translating a program that is unlikely to have been included in the training set verbatim, from a lesser-known language—to a well-known language, and back. I'd be shocked if this model held up in the comprehensive private evals.
- isoprophlex 2y agoAll the big guys are hiring domain experts - serious brains, phd level, in some cases - to build bespoke train and test data for their models. As long as Jensen Huang keeps shitting out nvidia cards, progress is just a function of cash to burn on paying humans to dump their knowledge into train data... and hoping this silly transformer architecture keeps holding up
- Wheatman 2y agoInterestingly enough, the new "Orion" model by OpenAI doesnt outperform, and even sometimes underperforms in programing tasks, when compared to GPT-4. There is an interesing discussion about it here:https://news.ycombinator.com/item?id=42104964 https://news.ycombinator.com/item?id=42104964.
- tucnak 2y ago> All the big guys are hiring domain experts - serious brains, phd level, in some cases I don't know where this myth had originated, and perhaps it was true at least at some point, but you just have to consider that all the recent major advances in datasets had to do with _unsupervised_ reward models, synthetic, generational datasets, and new advanced alignment methods. The big labs _are_ hiring serious PhD level researchers, and most of these are physicists, Bayesians of many kind and breed, not "domain experts." However, perception matters a lot these days; some labs, I won't point, but OpenAI is probably the biggest offender, simply cannot control themselves. The fact of the matter is they LOVE including the public evals in their finetuning, as it makes them appear stronger in the "benchmarks."
- adricg23 2y ago[dead]
- doomlaser 2y ago[flagged]
- joseferben 2y ago@simonw what is the token/s like on your 64gb m2 mbp?
- simonw 2y agoWith MLX: Prompt: 49 tokens, 95.691 tokens-per-sec Generation: 723 tokens, 10.016 tokens-per-sec Peak memory: 32.685 GB
- joseferben 2y agoso quite usable, thanks!
- kwlranb 2y agoXerox is a photocopier that can write a Shakespeare drama if I put the correct book on it. What is the end goal here? Reduction of the workforce by 90% while the remaining 10% click buttons to produce buggy, insecure and bloated code?
- smusamashah 2y agoWe (engineers specially) have been automating things ever since. Why/when should it ever stop?
- simonw 2y agoRight. Have your heard of open source? Biggest concerted effort to save each other time by avoiding repeating work I've ever seen.
- nenadst 2y agotry it with something more "obscure" like e.g. creating an OPC/Ua Server from scratch. It fails spectacularly - the code wont even compile and there are many things missing to get a working solution.
- simonw 2y agoCan you share a transcript?
- deleted 2y ago[deleted]
- kundi 2y agoIt seems fine-tuned for benchmarking more than the actual tasks
- fareesh 2y agoI like the idea of offline LLMs but in practice there's no way I'm wasting battery life on running a Language Model. On a desktop too, I wonder if it's worth the additional stress and heat on my GPU as opposed to one somewhere in a datacenter which will cost me a few dollars per month, or a few cents per hour if I spin up the infra myself on demand. Super useful for confidential / secret work though
- dizhn 2y agoWhat provider/service do you use for this?
- InsideOutSanta 2y agoIn my experience, a lot of companies still have rules against using tools like Copilot due to security and copyright concerns, even though many software engineers just to ignore them. This could be a way to satisfy both sides, although it only solves the issue of sending internal data to companies like OpenAI, it doesn't solve the "we might accidentally end up with somebody else's copyrighted code in our code base" issue.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]