5 ms·

qwen3.5/3.6 (30B) works well,locally, with opencode

zozbot234 6mo ago
Mind you, a 30B model (3B active) is not going to be comparable to Opus. There are open models that are near-SOTA but they are ~750B-1T total params. That's going to require substantial infrastructure if you want to use them agentically, scaled up even further if you expect quick real-time response for at least some fraction of that work. (Your only hope of getting reasonable utilization out of local hardware in single-user or few-users scenarios is to always have something useful cranking in the background during downtime.)
- pitched 6mo ago
  For a business with ten or more engineers/people-using-ai, it might still make sense to set this up. For an individual though, I can’t imagine you’d make it through to positive ROI before the hardware ages out.
  - zozbot234 6mo ago
    It's hard to tell for sure because the local inference engines/frameworks we have today are not really that capable. We have barely started exploring the implications of SSD offload, saving KV-caches to storage for reuse, setting up distributed inference in multi-GPU setups or over the network, making use of specialty hardware such as NPUs etc. All of these can reuse fairly ordinary, run-of-the-mill hardware.
  - DeathArrow 6mo ago
    Since you need at least a few of H100 class hardware, I guess you need at least few tens of coders to justify the costs.
    - pitched 6mo ago
      I see the 512GB Mac Studios aren’t for sale anymore but that was a much cheaper path
- wuschel 6mo ago
  What near SOTA open models are you referring to?
- cyberax 6mo ago
  I'm backing up a big dataset onto tapes, so I wanted to automate it. I have an idle 64Gb VRAM setup in my basement, so I decided to experiment and tasked it with writing an LTFS implementation. LTFS is an open standard for filesystems for tapes, and there's an implementation in C that can be used as the baseline. So far, Qwen 3.6 created a functionally equivalent Golang implementation that works against the flat file backend within the last 2 days. I'm extremely impressed.
  - Gareth321 6mo ago
    It is surprisingly competent. It's not Opus 4.6 but it works well for well structured tasks.
pitched 6mo ago
I want to bump this more than just a +1 by recommending everyone try out OpenCode. It can still run on a Codex subscription so you aren’t in fully unfamiliar territory but unlocks a lot of options.
- zozbot234 6mo ago
  The Codex TUI harness is also open source and you can use open models with it, so you can stay in even more familiar territory.
- pwython 6mo ago
  pi-coding-agent (pi.dev) is also great. I've been using it with Gemma 4 and Qwen 3.6.
cpursley 6mo ago
How are you running it with opencode, any tips/pointers on the setup?
jherdman 6mo ago
Is this sort of setup tenable on a consumer MBP or similar?
- pitched 6mo ago
  For a 30B model, you want at least 20GB of VRAM and a 24GB MBP can’t quite allocate that much of it to VRAM. So you’d want at least a 32GB MBP.
  - zozbot234 6mo ago
    It's a MoE model so I'd assume a cheaper MBP would simply result in some experts staying on CPU? And those would still have a sizeable fraction of the unified memory bandwidth available.
    - pitched 6mo ago
      I haven’t tried this myself yet but you would still need enough non-vram ram available to the cpu to offload to cpu, right? This is a fully novice question, I have not ever tried it.
      - tredre3 6mo ago
        You're correct. If you don't have enough RAM for the model, it can still run but most of it will run on the CPU and be continuously reloaded from the SSD (through mmap). A medium MoE like 35B can still achieve usable speeds in that setup, mind you, depending on what you're doing.
  - _blk 6mo ago
    Is there any model that practically compares to Sonnet 4.6 in code and vision and runs on home-grade (12G-24G) cards?
    - macwhisperer 6mo ago
      im currently running a custom Gemma4 26b MoE model on my 24gb m2... super fast and it beat deepseek, chatgpt, and gemini in 3 different puzzles/code challenges I tested it on. the issue now is the low context... I can only do 2048 tokens with my vram... the gap is slowly closing on the frontier models
equasar 6mo ago
The thing I dislike about OpenCode is the lack of capabilities of their editor, also, resource intensive, for some reason on a VM it chuckles each 30 mins, that I need to discard all sessions, commits, etc. I don't know if it is bun related, but in task manager, is the thing that is almost at the top always on CPU usage, turns out for me, bun is not production ready at all. Wish Zed editor had something like BigPickle which is free to use without limits.
- Jarred 6mo ago
  > turns out for me, bun is not production ready What issue did you run into?

Mind you, a 30B model (3B active) is not going to be comparable to Opus. There are open models that are near-SOTA but they are ~750B-1T total params. That's going to require substantial infrastructure if you want to use them agentically, scaled up even further if you expect quick real-time response for at least some fraction of that work. (Your only hope of getting reasonable utilization out of local hardware in single-user or few-users scenarios is to always have something useful cranking in the background during downtime.)

pitched 6mo ago
For a business with ten or more engineers/people-using-ai, it might still make sense to set this up. For an individual though, I can’t imagine you’d make it through to positive ROI before the hardware ages out.
- zozbot234 6mo ago
  It's hard to tell for sure because the local inference engines/frameworks we have today are not really that capable. We have barely started exploring the implications of SSD offload, saving KV-caches to storage for reuse, setting up distributed inference in multi-GPU setups or over the network, making use of specialty hardware such as NPUs etc. All of these can reuse fairly ordinary, run-of-the-mill hardware.
- DeathArrow 6mo ago
  Since you need at least a few of H100 class hardware, I guess you need at least few tens of coders to justify the costs.
  - pitched 6mo ago
    I see the 512GB Mac Studios aren’t for sale anymore but that was a much cheaper path
wuschel 6mo ago
What near SOTA open models are you referring to?
cyberax 6mo ago
I'm backing up a big dataset onto tapes, so I wanted to automate it. I have an idle 64Gb VRAM setup in my basement, so I decided to experiment and tasked it with writing an LTFS implementation. LTFS is an open standard for filesystems for tapes, and there's an implementation in C that can be used as the baseline. So far, Qwen 3.6 created a functionally equivalent Golang implementation that works against the flat file backend within the last 2 days. I'm extremely impressed.
- Gareth321 6mo ago
  It is surprisingly competent. It's not Opus 4.6 but it works well for well structured tasks.

pitched 6mo ago

I want to bump this more than just a +1 by recommending everyone try out OpenCode. It can still run on a Codex subscription so you aren’t in fully unfamiliar territory but unlocks a lot of options.

zozbot234 6mo ago
The Codex TUI harness is also open source and you can use open models with it, so you can stay in even more familiar territory.
pwython 6mo ago
pi-coding-agent (pi.dev) is also great. I've been using it with Gemma 4 and Qwen 3.6.

cpursley 6mo ago

How are you running it with opencode, any tips/pointers on the setup?

jherdman 6mo ago

Is this sort of setup tenable on a consumer MBP or similar?

pitched 6mo ago
For a 30B model, you want at least 20GB of VRAM and a 24GB MBP can’t quite allocate that much of it to VRAM. So you’d want at least a 32GB MBP.
- zozbot234 6mo ago
  It's a MoE model so I'd assume a cheaper MBP would simply result in some experts staying on CPU? And those would still have a sizeable fraction of the unified memory bandwidth available.
  - pitched 6mo ago
    I haven’t tried this myself yet but you would still need enough non-vram ram available to the cpu to offload to cpu, right? This is a fully novice question, I have not ever tried it.
    - tredre3 6mo ago
      You're correct. If you don't have enough RAM for the model, it can still run but most of it will run on the CPU and be continuously reloaded from the SSD (through mmap). A medium MoE like 35B can still achieve usable speeds in that setup, mind you, depending on what you're doing.
- _blk 6mo ago
  Is there any model that practically compares to Sonnet 4.6 in code and vision and runs on home-grade (12G-24G) cards?
  - macwhisperer 6mo ago
    im currently running a custom Gemma4 26b MoE model on my 24gb m2... super fast and it beat deepseek, chatgpt, and gemini in 3 different puzzles/code challenges I tested it on. the issue now is the low context... I can only do 2048 tokens with my vram... the gap is slowly closing on the frontier models

equasar 6mo ago

The thing I dislike about OpenCode is the lack of capabilities of their editor, also, resource intensive, for some reason on a VM it chuckles each 30 mins, that I need to discard all sessions, commits, etc. I don't know if it is bun related, but in task manager, is the thing that is almost at the top always on CPU usage, turns out for me, bun is not production ready at all. Wish Zed editor had something like BigPickle which is free to use without limits.

Jarred 6mo ago
> turns out for me, bun is not production ready What issue did you run into?