6 ms·
I really hope this doesn't hinder development too much. As Simon says, Qwen3.5 is very impressive. I've been testing Qwen3.5-35B-A3B over the past couple of da
by sosodev 7mo ago
I really hope this doesn't hinder development too much. As Simon says, Qwen3.5 is very impressive.
I've been testing Qwen3.5-35B-A3B over the past couple of days and it's a very impressive model. It's the most capable agentic coding model I've tested at that size by far. I've had it writing Rust and Elixir via the Pi harness and found that it's very capable of handling well defined tasks with minimal steering from me. I tell it to write tests and it writes sane ones ensuring they pass without cheating. It handles the loop of responding to test and compiler errors while pushing towards its goal very well.
- paoliniluis 7mo agowhat's your take between Qwen3.5-35B-A3B and Qwen3-Coder-Next?
- sosodev 7mo agoIn my experience Qwen3.5 is better even at smaller distillations. From what I understand the Qwen3-next series of models was just a test/preview of the architectural changes underpinning Qwen3.5. So Qwen3.5 is a more complete and well trained version of those models.
- kamranjon 7mo agoIn my experience qwen 3 coder next is better. I ran quite a few tests yesterday and it was much better at utilizing tool calls properly and understanding complex code. For its size though 3.5 35B was very impressive. coder next is an 80b model so i think its just a size thing - also for whatever reason coder next is faster on my machine. Only model that is competitive in speed is GLM 4.7 flash
- xrd 7mo agoWhat do you use as the orchestrator? By this I mean opencode, or the like. Is that the right term?
- simonw 7mo agoI use the term "harness" for those - or just "coding agent". I think orchestrator is more appropriate for systems that try to coordinate multiple agents running at the same time. This terminology is still very much undefined though, so my version may not be the winning definition.
- kamranjon 7mo agoI'm basically using the agentic features of the Zed editor: https://zed.dev/agentic https://zed.dev/agentic It's really easy to setup with any OpenAI compatible API and I self host Qwen Coder 3 Next on my personal MBP using LM Studio and just dial in from my work laptop with Zed and tailscale so i can connect from wherever i might be. It's able to do all sorts of things like run linting checks and tests and look for issues and refactor code and create files and things like this. I'm definitely still learning, but it's a pretty exciting jump from just talking to a chat bot and copying and pasting things manually.
- nvader 7mo agoAnother vote in favour of "harness". I'm aligning on Agent for the combination of harness + model + context history (so after you fork an agent you now have two distinct agents) And orchestrator means the system to run multiple agents together.
- Zetaphor 7mo agoThis has also been my understanding of all of these terms so far
- karmakaze 7mo agoWe don't have a Qwen3.5-Coder to compare with, but there is a chart comparing Qwen3.5 to Qwen3 including Qwen3-Next[0]. [0] https://www.reddit.com/r/LocalLLaMA/comments/1rivckt/visualizing_all_qwen_35_vs_qwen_3_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1rivckt/visuali...
- nekitamo 7mo agoIn my tests, Qwen3.5-35B-A3B is better, there is no comparison. Better tool calling and reasoning than Qwen3-Coder-Next for Html/Js coding tasks of medium size. Beware the quants and llama.cpp settings, they matter a lot and you have to try out a bunch of different quants to find one with acceptable settings, depending on your hardware.
- Twirrim 7mo agoI've been testing the same with some rust, and it's has spent a fair bit of time going through an infinite seeming loop before finally unjamming itself. It seems a little more likely to jam up than some other models I've experimented with. It's also driving itself crazy with deadpool & deadpool-r2d2 that it chose during planning phase. That said, it does seem to be doing a very good job in general, the code it has created is mostly sane other than this fuss over the database layer, which I suspect I'll have to intervene on. It's certainly doing a better job than other models I'm able to self-host so far.
- sosodev 7mo agoSome of the early quants had issues with tool calling and looping. So you might want to check that you're running the latest version / recommended settings.
- Aurornis 7mo ago> it's has spent a fair bit of time going through an infinite seeming loop before finally unjamming itself. I think this is part of the model’s success. It’s cheap enough that we’re all willing to let it run for extremely long times. It takes advantage of that by being tenacious. In my experience it will just keep trying things relentlessly until eventually something works. The downside is that it’s more likely to arrive at a solution that solves the problem I asked but does it in a terribly hacky way. It reminds me of some of the junior devs I’ve worked with who trial and error their way into tests passing. I frequently have to reset it and start it over with extra guidance. It’s not going to be touching any of my serious projects for these reasons but it’s fun to play with on the side.
- misnome 7mo ago> and it's has spent a fair bit of time going through an infinite seeming loop before finally unjamming itself I can live with this on my own hardware. Where Opus4.6 has developed this tendency to where it will happily chew through the entire 5-hour allowance on the first instruction going in endless circles. I’ve stopped using it for anything except the extreme planning now.
- cbm-vic-20 7mo ago
- a3b_unknown 7mo agoWhat is the meaning of 'A3B'?
- simonw 7mo agoIt's the number of active parameters for a Mixture of Experts (misleading name IMO) model. Qwen3.5-35B-A3B means that the model itself consists of 35 billion floating point numbers - very roughly 35GB of data - which are all loaded into memory at once. But... on any given pass through the model weights only 3 billion of those parameters are "active" aka have matrix arithmetic applied against them. This speeds up inference considerably because the computer has to do less operations for each token that is processed. It still needs the full amount of memory though as the 3B active it uses are likely different on every iteration.
- zozbot234 7mo agoIt will benefit from a full amount of memory for sure, but AIUI if you use system memory and mmap for your experts you can execute the model with only enough memory for the active parameters, it's just unbearably slow since it has to swap in new experts for every token. So the more memory you have in excess to that, the more inactive but often-used experts can be kept in RAM for better performance.
- EnPissant 7mo agoThe ability to stream weights from disk has nothing to do with MoE or not. You can always do this. It will be unusable either way.
- zozbot234 7mo agoAgreed but for a dense model you'd have to stream the whole model for every token, whereas with MoE there's at least the possibility that some experts may be "cold" for any given request and not be streamed in or cached. This will probably become more likely as models get even sparser. (The "it's unusable" judgmemt is correct if you're considering close-to-minimum reauirements, but for just getting a model to fit, caching "almost all of it" in RAM may be an excellent choice.)
- abhikul0 7mo agoAre you running it locally with llama.cpp? If so, is it working without any tweaking of the chat template? The tool calls fail for me when using the default chat template, however it seems to work a whole lot better with this: https://huggingface.co/Qwen/Qwen3.5-35B-A3B/discussions/9#699f8e8d16fcfc9bb99aa77f https://huggingface.co/Qwen/Qwen3.5-35B-A3B/discussions/9#69...
- arcanemachiner 7mo agoHave you tried the '--jinja' flag in llama-server?
- abhikul0 7mo agoYes, it fails too. I’m using the unsloth q4_km quant. Similarly fails with devstral2 small too, fixed that by using a similar template i found for it. Maybe it’s the quants that are broken, need to redownload I guess.
- sosodev 7mo agoI’ve been running it via llama-server with no issues. Running the latest Bartowski 6-bit quant
- brightball 7mo agoBartowski? Like Chuck Bartowski from the TV show?
- BoredomIsFun 7mo agoDifferent one. Bartowski is a minor celebrity in the local LLM world, together with Unsloth.
- Balinares 7mo agoWhat's the selling point of these quants vs the Unsloth ones?
- nu11ptr 7mo agoWhat hardware do you have it running on? Do you feel you could replace the frontier models with it for everyday coding? Would/will you?
- politelemon 7mo ago60 to 70 on a 5080, but only tinkering for now. The smaller models seem exceptionally good for what they are, and some can even do OCR reliably.
- cpburns2009 7mo agoWhat quantization are you running on the 5080? I'm waiting to receive mine.
- bigyabai 7mo agoI'm getting ~30 tok/s on the A3B model with my 3070 Ti and 32k context. > Do you feel you could replace the frontier models with it for everyday coding? Would/will you? Probably not yet, but it's really good at composing shell commands. For scripting or one-liner generation, the A3B is really good. The web development skills are markedly better than Qwen's prior models in this parameter range, too.
- deleted 7mo ago[deleted]
- jasonjmcghee 7mo agoThat seems oddly low / slower by a fair amount than i get on my m4. (I believe it was ~45 tok/s?) What quant are you using? How much ram does it have?
- sosodev 7mo agoAround 20ish tokens a second with 6-bit quant at very long context lengths on my AMD AI Max 395+ I’m trying to use local models whenever possible. Still need to lean on the frontier models sometimes.
- misnome 7mo agoI've been playing with 3.5:122b on a GH200 the past few days for rust/react/ts, and while it's clearly sub-Sonnet, with tight descriptions it can get small-medium tasks done OK - as well as Sonnet if the scope is small. The main quirk I've found is that it has a tendency to decide halfway through following my detailed instructions that it would be "simpler" to just... not do what I asked, and I find it has stripped all the preliminary support infrastructure for the new feature out of the code.
- reactordev 7mo agoTurn down the temperature and you’ll see less “simpler” short cuts.
- smokel 7mo agoFor the uninitiated: Interestingly, it is not advisable to take this to the extreme and set temperature to 0. That would seem logical, as the results are then completely deterministic, but it turns out that a suboptimal token may result in a better answer in the long run. Also, allowing for a little bit of noise gives the model room to talk itself out of a suboptimal path.
- LoganDark 7mo agoI like to think of this like tempering the output space. With a temperature of zero, there is only one possible output and it may be completely wrong. With even a low temperature, you drastically increase the chances that the output space contains a correct answer, through containing multiple responses rather than only one. I wonder if determinism will be less harmful to diffusion models because they perform multiple iterations over the response rather than having only a single shot at each position that lacks lookahead. I'm looking forward to finding out and have been playing with a diffusion model locally for a few days.
- reactordev 7mo agoYup. I think of it as how off the rails do you want to explore? For creative things or exploratory reasoning, a temperature of 0.8 lends us to all sorts of excursions down the rabbit hole. However, when coding and needing something precise, a temperature of 0.2 is what I use. If I don’t like the output, I’ll rephrase or add context.
- anana_ 7mo agoI've had even better results using the dense 27B model -- less looping and churning on problems
- androiddrew 7mo agoWhich dense model are you referring to? The dense model isn’t finetuned for code instruction according to the model card.
- whalesalad 7mo agoWhat hardware are you running this on?
- Zetaphor 7mo agoI'm running this exact same setup on a Framework Desktop and I'm seeing ~30 tokens/second