4 ms·

For personal use, yes. I replaced a $100/m subscription to claude in favor of running pi harness pointed at unsloth studio, using both qwen (unsloth/Qwen3.6-35

For personal use, yes. I replaced a $100/m subscription to claude in favor of running pi harness pointed at unsloth studio, using both qwen (unsloth/Qwen3.6-35B-A3B-MTP-GGUF) and gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models, depending on my mood. I have a machine I built about 5 years ago with dual RTX3090s in it (I was going to build a new gaming machine anyways, and the llama release had just dropped so I tacked another used 3090 onto the build), and I get ~150tok/s on either of those models (at UD-Q4_K_XL quant) and can use the entire 300k context length without having to exit VRAM. To be very clear - it's not as good as claude. But it's free and not so much worse that it matters significantly. For my personal needs, free beats $100/m. I also have an openclaw instance pointed at the same inference server, and it's great for that (genuinely solid use-case for local models). Some example projects - Replacement launcher for android tvs (with usage monitoring and tracking for kids) - Custom admin portals for my k8s cluster services - Custom home assistant integrations/automations (recently some shelly devices for power monitoring and switching) - Grocery list management and meal planning (mostly via openclaw) - some custom workflows for 3d asset generation in comfyui. --- Long story short, if you're trying to make money via software... I'd probably still recommend using a paid provider. But the local models are very capable of cool stuff.

gonzalohm 4mo ago
Did you double the tokens per second by adding a second GPU or was the increase significantly less?
- mirekrusin 4mo ago
  You’re adding extra gpu for more vram, not speed.
- horsawlarway 4mo ago
  No real change in inference speed. It basically just allows me to slot in more context or a bigger model. A single RTX-3090 will do approximately the same tok/s, but it won't fit the entire 300k context in VRAM. Sometimes that matters, a lot of times it doesn't. On the speed front - MOE models are great. Biggest perf difference in modern models is the move to MOE architectures. I get very similar quality from the both the Gemma-4 31B dense model, and the Gemma-4 26B MOE model (both at Q4 quant) but the MOE version runs at ~3 times the speed (150tok/s vs 46tok/s).
agup792 4mo ago
That sounds amazing. If I had some GPUs sitting around, I would totally do it. Sounds expensive to do it otherwise though.
rootlocus 4mo ago
2x RTX3090 are around $4400. Without any electricity costs or other parts, that's 3.6 years of $100/m claude.
- horsawlarway 4mo ago
  Yes, today is not a great time to purchase hardware. When I bought, I paid $850 a piece. And I needed one anyways for the gaming I was going to do. My guess is the next good time to buy is going to be 24-36 months from now, depending on how the AI bubble goes. --- I'll add to this, I personally don't like Apple hardware (not so much related to the hardware as their company philosophy) but their machines with unified memory (or AMDs latest unified memory offerings) get pretty equivalent speeds to my 3090s, and are probably a much better modern entrypoint to local llms. There's a reason the joke is that Silicon Valley software devs bought up all the Mac minis for OpenClaw. You can get a 48gb unified RAM M4 pro mac mini for ~2k. If you're not going to do much else with the machine, it's what I'd pick as my budget inference device right now. Spend a year of claude now, get ~150tok/s for the next decade (plus) for ~free. If you want more capable and are willing to spend a little more, go with the newer Ryzen AI Max+ 395 machines. You'll spend less on power too. My last suggestion would be to go buy an RTX3090 at this point. You can do a lot better for a lot cheaper.
  - tracker1 4mo ago
    If you're willing to go the AMD route, the AMD Radeon Pro R9700 definitely looks interesting for the price compared to NVidia.
    - felooboolooomba 4mo ago
      Can we also run LLMs on Radeon?
      - lloyd-christmas 4mo ago
        I run qwen 27B:Q4 @ 130k context at 50 t/s on a single R9700, and have a 7900XT that runs mellum 12B:Q8 as its subagent. R9700s do really well at low wattage and underclocking as well. It's designed to run at 300W, mine is throttled at 210W, and only had an 8% slowdown. If I had somewhere else to put my desktop in my house, I'd bump it up to 240W and there would be zero perf degradation.
twothreeone 4mo ago
> unsloth/Qwen3.6-35B-A3B-MTP-GGUF I've actually tried this exact same model locally as well.. albeit on just a single 3090 at 128k context and I got around 40-60tok/s with Q4_K quantization. The thing that bugged me the most was really the quality of the output on moderately complex real-world coding tasks. Having to switch between "prompt/vibe" and "manually implement" is such a big context switch burden, because you really have to ask yourself every few minutes if you're "holding it wrong" or the model is just too stupid. It also doesn't really seem to handle transitions from "low-level implementation detail" to "high-level design" well, e.g., it wouldn't easily render tables and such. With Claude I don't have this issue.. so I think for now my verdict would be that it's not really a viable replacement. I really hope it will be in a few months time. Oh and I used "aider" to replace claude CLI, which maybe that's also sub-optimal.. I'm not sure. The MCP marketplaces are useful of course, though arguably you could just manually replace them over time.
- horsawlarway 4mo ago
  I don't generally switch to implementing myself on the model, although there are definitely times where I stop it and correct it mid-task. It's prone to thinking longer and more repetitively, again - it's definitely not opus 4.7/4.8. I've been using pi.dev as my harness for it, and been pleasantly surprised by how nice it feels (I have used aider, but only very briefly and quite a while back - so I can't realistically compare). I would say it's roughly where I felt claude was a year back - Most of the sessions need to be more "pair programming" and less "I let it run for hours". I'm a big fan of frequent "human in the loop" style workflows even when I'm on something like opus at work, though. I have opinions about lots of things, and re-inforcing that the model should stop and ask frequently seems to get me considerably better output, without having to "re-roll" if you will. I've done a good bit of management, and I think it's roughly producing what a junior dev might produce in a day every 5 minutes. And just like a junior dev, you need to be steering it back on track fairly often. Opus feels more like a mid-level at this point. I can hand it a chunk of work and "leave" but I still get better output if I'm checked-in and watching/steering.
- unethical_ban 4mo ago
  I'm so out of the loop on this stuff, it's the first time in my IT career I feel really behind on things. I've used Claude Opus to quickly and effectively pound out some 100-200 line scripts that integrate with a vendor's API, and it one-shotted them both almost perfectly. I wonder if for a lot of these local models, the scope of the AI assistance should simply be smaller: You architect the tools and the function definitions, and then tell AI to implement one at a time? Does anyone do that rigorously?
kpw94 4mo ago
> gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models Since you're running quantized (at UD-Q4_K_XL) , check out the "qat" models (unsloth/gemma-4-26B-A4B-it-qat-GGUF) ! - https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF (With "Jun 9 Update: Added MTP support.") - https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/ https://blog.google/innovation-and-ai/technology/developers-...
- me_bx 4mo ago
  TIL: > Quantization-Aware Training (QAT) [...] allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model
- SubiculumCode 4mo ago
  How is the the QAT models at coding? I looked for opinions since the release and haven't found much.
anhtqweb 4mo ago
Grocery list management and meal planning sounds interesting. Would you mind sharing a little bit more on your use case please?
l332mn 4mo ago
Mind sharing your setup? I also have dual 3090s, but getting nowhere close to 300k context limits with 4 bit quantized models at that size (using vllm).
deleted 4mo ago
[deleted]

gonzalohm 4mo ago

Did you double the tokens per second by adding a second GPU or was the increase significantly less?

mirekrusin 4mo ago
You’re adding extra gpu for more vram, not speed.
horsawlarway 4mo ago
No real change in inference speed. It basically just allows me to slot in more context or a bigger model. A single RTX-3090 will do approximately the same tok/s, but it won't fit the entire 300k context in VRAM. Sometimes that matters, a lot of times it doesn't. On the speed front - MOE models are great. Biggest perf difference in modern models is the move to MOE architectures. I get very similar quality from the both the Gemma-4 31B dense model, and the Gemma-4 26B MOE model (both at Q4 quant) but the MOE version runs at ~3 times the speed (150tok/s vs 46tok/s).

agup792 4mo ago

That sounds amazing. If I had some GPUs sitting around, I would totally do it. Sounds expensive to do it otherwise though.

rootlocus 4mo ago

2x RTX3090 are around $4400. Without any electricity costs or other parts, that's 3.6 years of $100/m claude.

horsawlarway 4mo ago
Yes, today is not a great time to purchase hardware. When I bought, I paid $850 a piece. And I needed one anyways for the gaming I was going to do. My guess is the next good time to buy is going to be 24-36 months from now, depending on how the AI bubble goes. --- I'll add to this, I personally don't like Apple hardware (not so much related to the hardware as their company philosophy) but their machines with unified memory (or AMDs latest unified memory offerings) get pretty equivalent speeds to my 3090s, and are probably a much better modern entrypoint to local llms. There's a reason the joke is that Silicon Valley software devs bought up all the Mac minis for OpenClaw. You can get a 48gb unified RAM M4 pro mac mini for ~2k. If you're not going to do much else with the machine, it's what I'd pick as my budget inference device right now. Spend a year of claude now, get ~150tok/s for the next decade (plus) for ~free. If you want more capable and are willing to spend a little more, go with the newer Ryzen AI Max+ 395 machines. You'll spend less on power too. My last suggestion would be to go buy an RTX3090 at this point. You can do a lot better for a lot cheaper.
- tracker1 4mo ago
  If you're willing to go the AMD route, the AMD Radeon Pro R9700 definitely looks interesting for the price compared to NVidia.
  - felooboolooomba 4mo ago
    Can we also run LLMs on Radeon?
    - lloyd-christmas 4mo ago
      I run qwen 27B:Q4 @ 130k context at 50 t/s on a single R9700, and have a 7900XT that runs mellum 12B:Q8 as its subagent. R9700s do really well at low wattage and underclocking as well. It's designed to run at 300W, mine is throttled at 210W, and only had an 8% slowdown. If I had somewhere else to put my desktop in my house, I'd bump it up to 240W and there would be zero perf degradation.

twothreeone 4mo ago

> unsloth/Qwen3.6-35B-A3B-MTP-GGUF I've actually tried this exact same model locally as well.. albeit on just a single 3090 at 128k context and I got around 40-60tok/s with Q4_K quantization. The thing that bugged me the most was really the quality of the output on moderately complex real-world coding tasks. Having to switch between "prompt/vibe" and "manually implement" is such a big context switch burden, because you really have to ask yourself every few minutes if you're "holding it wrong" or the model is just too stupid. It also doesn't really seem to handle transitions from "low-level implementation detail" to "high-level design" well, e.g., it wouldn't easily render tables and such. With Claude I don't have this issue.. so I think for now my verdict would be that it's not really a viable replacement. I really hope it will be in a few months time. Oh and I used "aider" to replace claude CLI, which maybe that's also sub-optimal.. I'm not sure. The MCP marketplaces are useful of course, though arguably you could just manually replace them over time.

horsawlarway 4mo ago
I don't generally switch to implementing myself on the model, although there are definitely times where I stop it and correct it mid-task. It's prone to thinking longer and more repetitively, again - it's definitely not opus 4.7/4.8. I've been using pi.dev as my harness for it, and been pleasantly surprised by how nice it feels (I have used aider, but only very briefly and quite a while back - so I can't realistically compare). I would say it's roughly where I felt claude was a year back - Most of the sessions need to be more "pair programming" and less "I let it run for hours". I'm a big fan of frequent "human in the loop" style workflows even when I'm on something like opus at work, though. I have opinions about lots of things, and re-inforcing that the model should stop and ask frequently seems to get me considerably better output, without having to "re-roll" if you will. I've done a good bit of management, and I think it's roughly producing what a junior dev might produce in a day every 5 minutes. And just like a junior dev, you need to be steering it back on track fairly often. Opus feels more like a mid-level at this point. I can hand it a chunk of work and "leave" but I still get better output if I'm checked-in and watching/steering.
unethical_ban 4mo ago
I'm so out of the loop on this stuff, it's the first time in my IT career I feel really behind on things. I've used Claude Opus to quickly and effectively pound out some 100-200 line scripts that integrate with a vendor's API, and it one-shotted them both almost perfectly. I wonder if for a lot of these local models, the scope of the AI assistance should simply be smaller: You architect the tools and the function definitions, and then tell AI to implement one at a time? Does anyone do that rigorously?

kpw94 4mo ago

> gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models Since you're running quantized (at UD-Q4_K_XL) , check out the "qat" models (unsloth/gemma-4-26B-A4B-it-qat-GGUF) ! - https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF (With "Jun 9 Update: Added MTP support.") - https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/ https://blog.google/innovation-and-ai/technology/developers-...

me_bx 4mo ago
TIL: > Quantization-Aware Training (QAT) [...] allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model
SubiculumCode 4mo ago
How is the the QAT models at coding? I looked for opinions since the release and haven't found much.

anhtqweb 4mo ago

Grocery list management and meal planning sounds interesting. Would you mind sharing a little bit more on your use case please?

l332mn 4mo ago

Mind sharing your setup? I also have dual 3090s, but getting nowhere close to 300k context limits with 4 bit quantized models at that size (using vllm).

deleted 4mo ago

[deleted]