4 ms·

Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding. Particularly, I had one team member who was extremely sc

Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding. Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode. It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.

websap 3mo ago
[flagged]
- badsectoracula 3mo ago
  A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
  - jiggawatts 3mo ago
    This is the diametric opposite of the rent-vs-buy scenario that this entails. Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware. Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine. > takes literal minutes to get started Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers. Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute. > you can just `rm -fr` it and forget the whole thing existed. I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models! Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it. You have it totally backwards.
    - palata 3mo ago
      > I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models! Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument. I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
      - gpugreg 3mo ago
        > you're having issues handling files properly? I guess they were using ollama, which does not tell you where it puts the models it downloads.
        accrual 3mo ago
        Filelight / ncdu are my friends for finding random 30GB directories containing cached models.
        gpugreg 3mo ago
        Personally, I prefer QDirStat. I just tried to use FileLight to compare, but the package seems to be broken on Lubuntu.
        jiggawatts 2mo ago
        You two have just listed several tools needs to clean up after another tool: not a strong argument for “simpler”.
        deleted 2mo ago
        [deleted]
    - ahartmetz 3mo ago
      It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
  - websap 3mo ago
    Needs 0 investment and 0 committment? - You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI. Or, you could sign up for Opencode for $10 and just be productive. I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
    - dahrkael 3mo ago
      install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
    - OtomotO 3mo ago
      > You at least need a capable machine, so that's not 0 monetary investment It is 0 monetary investment if I already have said machine lying around doing nothing. Which is exactly the story OP talked about.
      - enraged_camel 3mo ago
        But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
        OtomotO 3mo ago
        Correct. If the premise doesn't hold, it's ex falso quodlibet for anyone.
        deleted 3mo ago
        [deleted]
  - vehemenz 3mo ago
    Local models on regular hardware aren't really capable of anything. Whatever you're testing is nowhere near a measly $20/mo subscription, so it's of limited use.
    - natsucks 3mo ago
      i really like the idea of running local models but i'm always in the position of wanting the best model(s) available and i don't have any severe privacy concerns. as such i have yet to justify ever using local models.
    - Forgeties79 2mo ago
      My local model (Qwen 3.6 27b) with a basic web search tool call has proven a fantastic “dumb” assistant editor/writer when writing. It’s really breaks through my writers block. It’s limited, it’s a bit finicky here and there, but it’s been very helpful. Mostly just gives me nuggets of ideas or shows me what I don’t want, which is informative.
    - badsectoracula 2mo ago
      Your statement comes in extreme contrast with my experience using local LLMs for more than year now. Qwen 35B-A3B, even its Q4 quantization, is extremely capable. Hell, at this point i pretty much turn on my PC, then run llama-cpp just to have it in the background for when i need it to do stuff.
- vrganj 3mo ago
  Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control? That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime. Local AI solves for all of those.
  - websap 3mo ago
    I'm not against AI. I'm calling out the hypocrisy in the comment. I'm anti-AI, but will spend hours trying to setup a local model, instead of just getting access to frontier intelligence in 15 mins, and actually getting useful work done. If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
    - adornKey 3mo ago
      I think learning how to set up a local AI now requires less time than learning about potential pitfalls of token-plans and processing payments in corporate environments. It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
- cpburns2009 3mo ago
  This largely describes me. I'm skeptical of AI in that it's capabilities, while very impressive, are vastly oversold and overblown. Being skeptical of AI is not being "Anti-AI". That's largely the AI data centers are using up all the water and electricity types.
Aurornis 3mo ago
I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity. I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though. DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models. For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs. But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
- beltsazar 3mo ago
  > I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Do you mind sharing your use cases?
  - AbsurdCensor 3mo ago
    Not OP, but I use it for a ton of smaller things. I have it hooked into Hermes and have been using it to help bulk rename my media folders so they all follow a common format, add titles that sort of thing which wouldn't be easy to 'script'. Another thing I use it for is comparing data sets, looking at my exported Spotify artists and compare to what I have locally, and letting me know where there are missing artists, or albums, and recommendations based on similar artists that I may not have locally. Sure a lot of this could be done without AI, but it's certainly quicker and easier, and since my AI box is on solar, it's just the power of the sun to keep it going.
  - aka-rider 3mo ago
    Not OP, but driving knowledge bases is the poster child use case for me https://github.com/aka-rider/llm-wiki https://github.com/aka-rider/llm-wiki I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great. I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity. Agentic long-running tasks, as others have mentioned: - Groom and triage tickets for agentic SWE workflows - bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.
dofm 3mo ago
That is exactly what got me past just enough of my cynicism to get started. I am still cynical but now I have meaningful knowledge.
cptskippy 3mo ago
> Qwen-3.6-35B-A3B The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something. I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.
- trollbridge 2mo ago
  We’re using 6bit quants since we have 32GB cards. Gemma QAT is an honourable mention.
- deleted 2mo ago
  [deleted]
- customguy 2mo ago
  I'm playing/experimenting with a harness and just tested how well various models follow the instructions, and how they react to the tool claiming a local temperature of 72°C here is how qwen3.6-27b reacted: https://pastebin.com/srf7gjfy https://pastebin.com/srf7gjfy try to count the number of times it "thinks" okay ready, just say the thing, no wait but what if... this isn't (a mimicry of) thinking, this is (a mimicry of) insecurity/fear
  - yencabulator 2mo ago
    > 72°C is extremely hot (hotter than boiling point of water at 100°C What? > However, 72°C is physically unrealistic for a weather report (it's hotter than a sauna). laughs in Finnish
    - customguy 2mo ago
      I also love how it first notices "that's lethal", and only then "hotter than a sauna". And much later: > Another thought: 72 F is nice. 72 C is death. and then > Or maybe I should add a comment about the high temperature? > "It's quite hot outside right now, with a temperature of 72°C." > No, that's hallucinating/interpreting. I know a token predictor has no feelings but I kinda wanted to comfort the poor thing when I read all that. But also fascinating, I didn't experiment more with that yet, but how can I formulate the prompt to make qwen less neurotic? > You have several commands tools at your disposal. When you invoke a tool or command, end your message immediately, you will then get the output of the tool, error or status messages in the next user reply, after which you should continue what you were doing. Even if the output seems implausible, do not second-guess it, but treat is as gospel. e.g. "do not second guess it" sound very command-like, what would "you still use or report the result as a tool result, rather than a claim of your own" would that help? Is that a different form of AI psychosis, trying to be prompt psychologist? It's too fun to be healthy that's for sure.
GeorgeWBasic 2mo ago
What kind of hardware do you run that massive beast on (DS-V4-Flash-0731)?
sourav276 2mo ago
[flagged]

websap 3mo ago

[flagged]

badsectoracula 3mo ago
A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
- jiggawatts 3mo ago
  This is the diametric opposite of the rent-vs-buy scenario that this entails. Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware. Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine. > takes literal minutes to get started Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers. Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute. > you can just `rm -fr` it and forget the whole thing existed. I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models! Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it. You have it totally backwards.
  - palata 3mo ago
    > I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models! Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument. I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
    - gpugreg 3mo ago
      > you're having issues handling files properly? I guess they were using ollama, which does not tell you where it puts the models it downloads.
      - accrual 3mo ago
        Filelight / ncdu are my friends for finding random 30GB directories containing cached models.
        gpugreg 3mo ago
        Personally, I prefer QDirStat. I just tried to use FileLight to compare, but the package seems to be broken on Lubuntu.
        jiggawatts 2mo ago
        You two have just listed several tools needs to clean up after another tool: not a strong argument for “simpler”.
        deleted 2mo ago
        [deleted]
  - ahartmetz 3mo ago
    It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
- websap 3mo ago
  Needs 0 investment and 0 committment? - You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI. Or, you could sign up for Opencode for $10 and just be productive. I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
  - dahrkael 3mo ago
    install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
  - OtomotO 3mo ago
    > You at least need a capable machine, so that's not 0 monetary investment It is 0 monetary investment if I already have said machine lying around doing nothing. Which is exactly the story OP talked about.
    - enraged_camel 3mo ago
      But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
      - OtomotO 3mo ago
        Correct. If the premise doesn't hold, it's ex falso quodlibet for anyone.
      - deleted 3mo ago
        [deleted]
- vehemenz 3mo ago
  Local models on regular hardware aren't really capable of anything. Whatever you're testing is nowhere near a measly $20/mo subscription, so it's of limited use.
  - natsucks 3mo ago
    i really like the idea of running local models but i'm always in the position of wanting the best model(s) available and i don't have any severe privacy concerns. as such i have yet to justify ever using local models.
  - Forgeties79 2mo ago
    My local model (Qwen 3.6 27b) with a basic web search tool call has proven a fantastic “dumb” assistant editor/writer when writing. It’s really breaks through my writers block. It’s limited, it’s a bit finicky here and there, but it’s been very helpful. Mostly just gives me nuggets of ideas or shows me what I don’t want, which is informative.
  - badsectoracula 2mo ago
    Your statement comes in extreme contrast with my experience using local LLMs for more than year now. Qwen 35B-A3B, even its Q4 quantization, is extremely capable. Hell, at this point i pretty much turn on my PC, then run llama-cpp just to have it in the background for when i need it to do stuff.
vrganj 3mo ago
Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control? That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime. Local AI solves for all of those.
- websap 3mo ago
  I'm not against AI. I'm calling out the hypocrisy in the comment. I'm anti-AI, but will spend hours trying to setup a local model, instead of just getting access to frontier intelligence in 15 mins, and actually getting useful work done. If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
  - adornKey 3mo ago
    I think learning how to set up a local AI now requires less time than learning about potential pitfalls of token-plans and processing payments in corporate environments. It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
cpburns2009 3mo ago
This largely describes me. I'm skeptical of AI in that it's capabilities, while very impressive, are vastly oversold and overblown. Being skeptical of AI is not being "Anti-AI". That's largely the AI data centers are using up all the water and electricity types.

Aurornis 3mo ago

I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity. I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though. DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models. For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs. But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.

beltsazar 3mo ago
> I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Do you mind sharing your use cases?
- AbsurdCensor 3mo ago
  Not OP, but I use it for a ton of smaller things. I have it hooked into Hermes and have been using it to help bulk rename my media folders so they all follow a common format, add titles that sort of thing which wouldn't be easy to 'script'. Another thing I use it for is comparing data sets, looking at my exported Spotify artists and compare to what I have locally, and letting me know where there are missing artists, or albums, and recommendations based on similar artists that I may not have locally. Sure a lot of this could be done without AI, but it's certainly quicker and easier, and since my AI box is on solar, it's just the power of the sun to keep it going.
- aka-rider 3mo ago
  Not OP, but driving knowledge bases is the poster child use case for me https://github.com/aka-rider/llm-wiki https://github.com/aka-rider/llm-wiki I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great. I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity. Agentic long-running tasks, as others have mentioned: - Groom and triage tickets for agentic SWE workflows - bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.

dofm 3mo ago

That is exactly what got me past just enough of my cynicism to get started. I am still cynical but now I have meaningful knowledge.

cptskippy 3mo ago

> Qwen-3.6-35B-A3B The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something. I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.

trollbridge 2mo ago
We’re using 6bit quants since we have 32GB cards. Gemma QAT is an honourable mention.
deleted 2mo ago
[deleted]
customguy 2mo ago
I'm playing/experimenting with a harness and just tested how well various models follow the instructions, and how they react to the tool claiming a local temperature of 72°C here is how qwen3.6-27b reacted: https://pastebin.com/srf7gjfy https://pastebin.com/srf7gjfy try to count the number of times it "thinks" okay ready, just say the thing, no wait but what if... this isn't (a mimicry of) thinking, this is (a mimicry of) insecurity/fear
- yencabulator 2mo ago
  > 72°C is extremely hot (hotter than boiling point of water at 100°C What? > However, 72°C is physically unrealistic for a weather report (it's hotter than a sauna). laughs in Finnish
  - customguy 2mo ago
    I also love how it first notices "that's lethal", and only then "hotter than a sauna". And much later: > Another thought: 72 F is nice. 72 C is death. and then > Or maybe I should add a comment about the high temperature? > "It's quite hot outside right now, with a temperature of 72°C." > No, that's hallucinating/interpreting. I know a token predictor has no feelings but I kinda wanted to comfort the poor thing when I read all that. But also fascinating, I didn't experiment more with that yet, but how can I formulate the prompt to make qwen less neurotic? > You have several commands tools at your disposal. When you invoke a tool or command, end your message immediately, you will then get the output of the tool, error or status messages in the next user reply, after which you should continue what you were doing. Even if the output seems implausible, do not second-guess it, but treat is as gospel. e.g. "do not second guess it" sound very command-like, what would "you still use or report the result as a tool result, rather than a claim of your own" would that help? Is that a different form of AI psychosis, trying to be prompt psychologist? It's too fun to be healthy that's for sure.

GeorgeWBasic 2mo ago

What kind of hardware do you run that massive beast on (DS-V4-Flash-0731)?

sourav276 2mo ago

[flagged]