8 ms·
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VR
by hypfer 2mo ago
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
- reilly3000 2mo agoThanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
- hypfer 2mo agoAccording to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage. But maybe someone else has experience to share there
- nubg 2mo agojust to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort. put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
- williamcotton 2mo agoI generally agree and expect this to be the case from a legal perspective. Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright. [0] https://en.wikipedia.org/wiki/Structure,_sequence_and_organization https://en.wikipedia.org/wiki/Structure,_sequence_and_organi... [1] https://en.wikipedia.org/wiki/Abstraction–filtration–comparison_test https://en.wikipedia.org/wiki/Abstraction–filtration–compari...
- bilekas 2mo agoYou sound like your trying to reassure yourself of something. I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
- deleted 2mo ago[deleted]
- formerly_proven 2mo ago> I sure hope my boss doesn't think he built my work! Most managers do though?
- bilekas 2mo agoI've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine. But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product. Who created it is still me.
- fragmede 2mo agoDid Steve Jobs create the iPhone?
- D4Ha 2mo agoDo you find it useful or worthwhile to split a large LLM across two GPUs on a desktop? If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU. How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
- deleted 2mo ago[deleted]
- CamperBob2 2mo agoThat's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/hardware/topology.md https://github.com/local-inference-lab/rtx6kpro/blob/master/... . Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
- mips_avatar 2mo agoDepends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
- nullc 2mo agofor layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
- ericd 2mo agoGood point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.
- deleted 2mo ago
- bmitc 2mo agoLol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command. I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
- xienze 2mo agoWell there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
- hypfer 2mo agoWhat makes you say that it would be hard to do that? It's long, I guess, but not cryptic. You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away. You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.
- porphyra 2mo ago> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it. isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
- Aurornis 2mo ago> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks. > --spec-draft-n-max 5 5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
- hypfer 2mo agoYes to both. The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though. As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute. Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
- hedgehog 2mo agoSame, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.
- hypfer 2mo agoI wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions. Though it might turn out that that doesn't offer any benefits. Has it been tried yet? ___ EDIT: LLM pointed me towards this thing I'm not going to read https://arxiv.org/abs/2512.11280 https://arxiv.org/abs/2512.11280 "AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference" Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
- cjbprime 2mo agoHm, I have a 4090 as well, and: $ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off 0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
- Jcowell 2mo agoThere has to be a better way of sharing config , a profile of sorts you can upload and share
- mhitza 2mo agoYes, llama.cpp supports presets (an ini format). Though it's not always clear which llama-server flags it does not support.
- Aurornis 2mo agoThe person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090. If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
- vel0city 2mo ago>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail? That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
- bilsbie 2mo agoAny chance I could run it on a GeForce RTX 5060 Ti 16 GB, (64 gb ram)
- metadat 2mo agoYou may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
- SwellJoe 2mo agoYou could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.
- wgd 2mo agoYes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.
- jacquesm 2mo agoI absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
- secretluver 2mo ago[dead]
- hypfer 2mo agoProblem with that is I think that it quickly devolves into cargo culting, nonsense and noise. Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot. Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future. So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise. And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with. The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.
- jacquesm 2mo agoI've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time. Let me return the favor: Deepseek V4 Flash 0731 Q8 This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04. GGML_CUDA_P2P=1 build/bin/llama-server -m ../DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --alias DeepSeek-V4-Flash --threads 32 --threads-batch 64 --cache-type-k f16 --cache-type-v f16 --metrics --cache-ram 0 --load-mode none --fit on --fit-ctx 1000000 --host 0.0.0.0 -fa on --top-p 0.95 --temp 0.7 --repeat-penalty 1.2 --jinja --tensor-split 1,.8,.1,.15 --cont-batching -b 4096 -ub 4096 -ncmoe 36 --parallel 1 -ot 'blk\.3[67]\.ffn_(gate|up|down)_exps=CPU' I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.
- XCSme 2mo agoWith default config via Ollama and 65k context I get 50tps on a 3090.
- tyfon 2mo agoTo add for me (and claude assisted but tuned myself), here is for 3060 12gb + 5950x 128 gb ram without image support: https://pastebin.com/HsAUBGSY https://pastebin.com/HsAUBGSY Here is the chat template I used (and renamed to qwen.jinja): https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/raw/main/chat_template.jinja https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/r... For image support, drop to NGL 24 and add: --mmproj-url https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/mmproj-F16.gguf \ --no-mmproj-offload \ --image-min-tokens 1024 \ And drop the --no-mmproj
- tommica 2mo agoThanks for this! I got the same card, but not the same CPU nor RAM, but this is a great starting point to start tweaking
- acidburnNSA 2mo agoHow many tokens/second you getting? I have the same CPU but a 5070 and only 64 GB of ram. I just got llama.cpp built and am now hitting a whopping 5-6 tokens/s.
- mrbonner 2mo agoJeez, llama.c++ is becoming the ffmpeg cargo cult CLI now
- cjbprime 2mo agoDoes anyone know how to get this working with Claude Code via llama-server? I'm getting a jinja template error about the system prompt not being the first message.
- qiine 2mo agowhat --no-context-shift is for ?
- pulse7 2mo ago"--context-shift, --no-context-shift ... whether to use context shift on infinite text generation (default: disabled)" From: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
- MrDrMcCoy 2mo agoThat is obvious. Less obvious is what context shifting does and whether you want it or not.
- lelandbatey 2mo agoI can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]: llama-server.exe ^ -m "Qwen3.8-27B-UD-Q2_K_XL.gguf" ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ --fit-ctx 128000 ^ -ctk q4 0 ^ -ctv q4 0 ^ --reasoning-budget -1 ^ --chat-template-kwargs "{\"preserve thinking\": true}" ^ --host 0.0.0.0 ^ --port 8033 [0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL) [1] https://github.com/ggml-org/llama.cpp/releases https://github.com/ggml-org/llama.cpp/releases Instructions if you want to do the same: 1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder. 2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`. 3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window. You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored). Congratulations, you're now running Qwen 3.8 27B. Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
- Almondsetat 2mo agoHave you tried to put the nvidia card into "compute-only mode" since you're not using it for display?
- silentbob7 2mo agoHere's my model.ini for 4090 (using the chat template from froggeric): version = 1 [*] ; Global defaults shared across all presets threads = 16 [Qwen3.8-27B] ; Model files model = ..\Qwen3.8-27B-FP8\Qwen3.8-27B-UD-Q4_K_XL.gguf mmproj = ..\Qwen3.8-27B-FP8\mmproj-BF16.gguf ; Context & compute ctx-size = 114688 parallel = 1 n-gpu-layers = -1 batch-size = 1024 ubatch-size = 512 flash-attn = on cache-type-k = q8_0 cache-type-v = q8_0 ; Speculative decoding (MTP draft) spec-default = true spec-type = draft-mtp spec-draft-n-max = 5 cache-type-k-draft = q4_0 cache-type-v-draft = q4_0 ; Multimodal no-mmproj-offload = true ; Context behavior no-context-shift = true ; Chat template & reasoning jinja = true reasoning = on chat-template-kwargs = {"preserve_thinking":"true","reasoning_effort":"medium"} ; Sampling temp = 1.0 top-n-sigma = 1 top-k = 0 top-p = 1.0 presence-penalty = 0.0 repeat-penalty = 1.0 fit = off ; Auto-load this model when the server starts load-on-startup = true