7 ms·
Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
- vbtechguy 6mo agoHere is how I set up Gemma 4 26B for local inference on macOS that can be used with Claude Code.
- canyon289 6mo agoThis is a nice writeup!
- trvz 6mo agoollama launch claude --model gemma4:26b
- datadrivenangel 6mo agoIt's amazing how simple this is, and it just works if you have ollama and claude installed!
- pshirshov 6mo agoFor some reason, that doesn't work for me, claude never returns from some ill loop. Nemotron, glm and qwen 3.5 work just fine, gemma - doesn't.
- trvz 6mo agoSince that defaults to the q4 variant, try the q8 one: ollama launch claude --model gemma4:26b-a4b-it-q8_0
- pshirshov 6mo agoEven tried gemma4:31b and gemma4:31b with 128k context (I have 72GiB VRAM). Nothing. I'm cursed I guess. That's ollama-rocm if that matters (I had weird bugs on Vulkan, maybe gemma misbehaves on radeons somehow?..). UPD: tried ollama-vulkan. It works, gemma4:31b-it-q8_0 with 64k context!
- alfiedotwtf 6mo agoThe default context is 128k for the smaller Gemma 4’s and 256k for the bigger ones, so you’re cutting off context and it doesn’t know how to continue. Bump it to native (or -c 0 may work too)
- pshirshov 6mo agoIn that case the model descriptor on ollama.com is incorrect, because it defaults to 16k. So I have to manually change that to 64/128k. I think you are talking about maximum context size.
- trvz 6mo agoNo, the default context in Ollama varies by the memory available: https://docs.ollama.com/context-length https://docs.ollama.com/context-length
- gcampos 6mo agoYou need to increase the context window size or the tool calling feature wont work
- mil22 6mo agoFor those wondering how to do this: OLLAMA_CONTEXT_LENGTH=64000 ollama serve or if you're using the app, open the Ollama app's Settings dialog and adjust there. Codex also works: ollama launch codex --model gemma4:26b
- jonplackett 6mo agoSo wait what is the interaction between Gemma and Claude?
- unsnap_biceps 6mo agolm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn't have the same issue, so I've stuck to using ollama for local development work.
- keerthiko 6mo agoClaude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with the VRAM needed. And if you somehow managed to open up a big enough VRAM playground, the open weights models are not quite as good at wrangling such large context windows (even opus is hardly capable) without basically getting confused about what they were doing before they finish parsing it.
- storus 6mo agoCan't you use Claude caveman mode? https://github.com/JuliusBrussee/caveman https://github.com/JuliusBrussee/caveman
- unsnap_biceps 6mo agoI use CC at work, so I haven't explored other options. Is there a better one to use locally? I presumed they were all going to be pretty similar.
- Someone1234 6mo agoUsing Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
- moomin 6mo agoRight now it suits them down to the ground. You pay for the product and you don’t cost their servers anything.
- phainopepla2 6mo agoYou don't pay anything to use Claude Code as a front end to non-Anthropic models
- quinnjh 6mo agoso no subscription is needed?
- kenmacd 6mo agonot to use the cli tool. You can install it and change the settings to point to pretty much any other model. It's an okay-enough tool, but I don't see a lot of point in using it when open sources tools like Pi and OpenCode exist (or octofriend, or forge, or droid, etc).
- chvid 6mo agoIs it not about the same as using OpenCode? And is running a local model with Claude Code actually usable for any practical work compared to the hosted Anthropic models?
- wyre 6mo agoI think CC is popular because they are catering to the common denominator programmer and are going to continue to do that, not because CC is particularly turn-key.
- martinald 6mo agoJust FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
- IceWreck 6mo agoIt does if you use an inference engine where you can offload some of the experts from VRAM to CPU RAM. That means I can fit a 35 billion param MoE in let's say 12 GB VRAM GPU + 16 gigs of memory.
- Yukonv 6mo agoWith that you are taking a significant performance penalty and become severely I/O bottlenecked. I've been able to stream Qwen3.5-397B-A17B from my M5 Max (12 GB/s SSD Read) using the Flash MoE technique at the brisk pace of 10 tokens per second. As tokens are generated different experts need to be consulted resulting in a lot of I/O churn. So while feasible it's only great for batch jobs not interactive usage.
- zozbot234 6mo ago10 tok/s is quite fine for chatting, though less so for interaction with agentic workloads. So the technique itself is still worthwhile for running a huge model locally.
- IceWreck 6mo ago> So while feasible it's only great for batch jobs not interactive usage. I mean yeah true but depends on how big the model is. The example I gave (Qwen 3.5 35BA3B) was fitting a 35B Q4 K_M (say 20 GB in size) model in 12 GB VRAM. With a 4070Ti + high speed 32 GB DDR5 ram you can easily get 700 token/sec prompt processing and 55-60 token/sec generation which is quite fast. On the other hand if I try to fit a 120B model in 96 GB of DDR5 + the same 12 GB VRAM I get 2-5 token/sec generation.
- 6mo ago
- meidad_g 6mo ago[flagged]
- asymmetric 6mo agoIs a framework desktop with >48GB of RAM a good machine to try this out?
- pshirshov 6mo agoOnly for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).
- nl 6mo agoDoesn't the framework desktop have a Ryzen 395 AI? That's a unified memory architecture like the Macs.
- pshirshov 6mo agoThat's discrete DDR5, it's not as fast as your regular VRAM.
- pshirshov 6mo agoAh, forgot to add, it's not really "unified" you have to explicitly specify your allocations. You may have a reasonably good 48gb chunk assigned to the GPU, but that DDR5 is 5-10 times slower than GDDR/HBM and the GPU itself isn't stellar. So, framework laptops are great for chatting but nearly useless in agentic coding. My Radeon W7900 answers a question ("what is this project") in 2 minutes, it takes my Framework 16 with 5070 addon around 11 minutes without the addon - around 23 (qwen 3.5 27b, claude code)
- ac29 6mo agoThis article is about a MoE model with only 4B active parameters, it shouldn't take 10 minutes to answer a question about a small project. I measured a 4bit quant of this model at 1300t/s prefill and ~60t/s decode on Ryzen 395+.
- aplomb1026 6mo ago[dead]
- aetherspawn 6mo agoCan you use the smaller Gemma 4B model as speculative decoding for the larger 31B model? Why/why not?
- MeetRickAI 6mo ago[dead]
- techpulselab 6mo ago[dead]
- NamlchakKhandro 6mo agoI don't know why people bother with Claude code. It's so jank, there are far superior cli coding harness out there
- loveparade 6mo agoWhat do you recommend? I've tried both pi and opencode and both are better than claude imo, but I wonder if there are others.
- tarruda 6mo agoCodex is the best out-of-box experience, especially due to its builtin sandboxing. Only drawback is that its edit tool requires the LLM to output a diff which only GPTs are trained to do correctly.
- loveparade 6mo agoInteresting, I don't like codex exactly because of its built-in sandboxing. If I need a sandbox I rather do a simple bwrap myself around the agent process, I prefer that over the agent cli doing a bunch of sandboxing magic that gets in my way.
- prettyblocks 6mo agohow is codex sandbox different from /sandbox on claude code?
- z0mghii 6mo agoCan you elaborate what is jank about it?
- threethirtytwo 6mo agoit has visual artifacts when inferencing.
- dimgl 6mo agoVagueposting in Hacker News?
- inzlab 6mo agoawesome, the lighter the hardware running big softwares the more novelty.
- edinetdb 6mo ago[flagged]
- tatrions 6mo ago[flagged]
- mjlee 6mo agoI find MCP beneficial too, but do be aware of token usage. With a naive implementation MCP can use significantly more input tokens (and context) than equivalent skills would. With a handful of third party MCPs I’ve seen tens of thousands of tokens used before I’ve started anything. Here’s an article from Anthropic explaining why, but it is 5 months old so perhaps it's irrelevant ancient history at this point. https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc...
- chappyasel 6mo ago[dead]
- seifbenayed1992 6mo agoLocal models are finally starting to feel pleasant instead of just "possible." The headless LM Studio flow is especially nice because it makes local inference usable from real tools instead of as a demo. Related note from someone building in this space: I've been working on cloclo (https://www.npmjs.com/package/cloclo https://www.npmjs.com/package/cloclo), an open-source coding agent CLI, and this is exactly the direction I'm excited about. It natively supports LM Studio, Ollama, vLLM, Jan, and llama.cpp as providers alongside cloud models, so you can swap between local and hosted backends without changing how you work. Feels like we're getting closer to a good default setup where local models are private/cheap enough to use daily, and cloud models are still there when you need the extra capability.
- SeriousM 6mo agoHow does cloclo differ from pi-mono?
- seifbenayed1992 6mo agopi-mono is a great toolkit — coding agent CLI, unified LLM API, web UI, Slack bot, vLLM pods. cloclo is a runtime for agent toolkits. You plug it into your own agents and it gives them multi-agent orchestration (AICL protocol), 13 providers, skill registry, native browser/docs/phone tools, memory, and an NDJSON bridge. Zero native deps.
- janalsncm 6mo agoQwen3-coder has been better for coding in my experience and has similar sizes. Either way, after a bunch of frustration with the quality and price of CC lately I’m happy there are local options.
- _2fnr 6mo ago[flagged]
- jeremie_strand 6mo ago[dead]
- satvikpendem 6mo agoSounds like the exact opposite, models are being commoditized while the harness and tooling around a model is what actually gets significant gains, especially with RL around specific models. For example, this article was posted recently, Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed [0]. [0] https://news.ycombinator.com/item?id=46988596 https://news.ycombinator.com/item?id=46988596
- bckr 6mo agoI think it’s ALL getting commoditized. The winners here are engineers (who are onboard with the agentic surge) and, hopefully, users who get more and better software.
- vkou 6mo ago> hopefully, users who get more and better software. Users are definitely going to get more software and more features and redesigns in the software they use, but I have strong doubts that it's going to get better. If pre-LLM developer productivity was used to build all sorts of deranged anti-user promo-padding bullshit, imagine how much more of it we can do with a 2x more productive employee base.
- bckr 6mo agoyes. But also some stuff will be amazing.
- chappyasel 6mo ago[dead]
- AbuAssar 6mo agoomlx gives better performance than ollama on apple silicon
- maxbeech 6mo ago[dead]
- smcleod 6mo agoDid you try the MLX model instead? In general MLX tends provide much better performance than GGUF/Llama.cpp on macOS.
- d4rkp4ttern 6mo agoYou can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook: https://pchalasani.github.io/claude-code-tools/integrations/local-llms/ https://pchalasani.github.io/claude-code-tools/integrations/... The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared to Qwen3.5 35BA3B. However the tau2 bench results[1] for this Gemma4 variant lag far behind the Qwen variant (68% vs 81%), so I don’t expect the former to do well on heavy agentic tool-heavy tasks: [1] https://news.ycombinator.com/item?id=47616761 https://news.ycombinator.com/item?id=47616761
- peder 6mo agoDid you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?
- selectodude 6mo agoI’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.
- vlowther 6mo agoSame. Opencode + oMLX (0.3.4) + unsloth-Qwen3-Coder-Next-mlx-8bit on my M5 Max w 128GB is the sweet spot for me locally. The prompt decode caching keeps things coherent and fast even when contexts get north of 100k tokens.
- peder 6mo agoHave you been using `omlx serve`? If so, how are you bumping up the max context size? I'm not seeing a param to go above 32k?
- d4rkp4ttern 6mo ago
- meidad_g 6mo ago[flagged]
- tiku 6mo agoI hate that my M5 with 24 gb has so much trouble with these models. Not getting any good speeds, even with simple models.
- Imanari 6mo agoHow well do the Gemma 4 models perform on agentic coding? What are your impressions?
- ttul 6mo agoI could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.
- jedisct1 6mo agoRunning Gemma 4 with llama.cpp and Swival: $ llama-server --reasoning auto --fit on -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 $ uvx swival --provider llamacpp Done.
- drob518 6mo agoSeems like this might be a great way to do web software testing. We’ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there’s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That’s also a great use for a smaller model like this.
- robot_jesus 6mo agoYeah. I think that's an interesting use case. Especially if I can kick it off or schedule it when I'm not actively working. Inference speed (especially with tool calling involved) won't be great on my machines, but if I schedule nightly usability tests of dev sites while I sleep, that could be really cool.
- drob518 6mo agoYou’re right about inference speed being a concern. I was assuming it’s a small model but even then, one of the browser automation frameworks is going to be faster.
- pseudosavant 6mo agoI want local models to succeed, but today the gap vs cloud models still seems continually too large. Even with a $2k GPU or a $4k MBP, the quality and speed tradeoff usually isn’t sensible. Credit to Google for releasing Gemma 4, though. I’d love to see local models reach the point where a 32 GB machine can handle high quality agentic coding at a practical speed.
- ffsm8 6mo agoFwiw, the real reason we don't have 100+ GB GPUs is because Nvidia likes to segment their markets. They could sell the consumer cards with 200gb gddr RAM on it, they just know that'd eat into their enterprise offering which is quiet literally all their profit margin (which I may add is gargantuan as of 2025)
- alfiedotwtf 6mo agoPSA: For those getting stuck in a repetitive loop or just stopping without completing a task, try the interactive template. I just tried it now and it's blowing my already impressive results out of the water (llama.cpp): --jinja --chat-template-file models/templates/google-gemma-4-31B-it-interleaved.jinja
- bicepjai 6mo agoTotally agree lmstudio headless server on a remote machine but control models from your laptop is an amazing workflow. But Gemma 4 was not a good model atleast in my trials “find me the largest text file in all of the current sub folders” it went on a loopy tool call for ever even with Q8
- aimemobe 6mo ago[flagged]
- ashwanth_megas 6mo agoThe interesting bottleneck I keep running into isn’t just model quality — it’s lifecycle management of models in constrained environments (load → run → unload patterns, plus routing between different models depending on task type). Curious if anyone else is exploring per-request model execution rather than keeping models resident all the time.