3 ms·
If you have to ask then your GPU is too small. With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
by trvz 6mo ago
If you have to ask then your GPU is too small.
With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
- FusionX 6mo agoAren't 4bits model decent? Since, this is an MOE model, I'm assuming it should have respectable tk/s, similar to previous MOE models.
- palmotea 6mo ago> If you have to ask then your GPU is too small. What's the minimum memory you need to run a decent model? Is it pretty much only doable by people running Macs with unified memory?
- bfivyvysj 6mo agoA bit like asking how long is a piece of string.
- utilize1808 6mo agoObviously going to depend on your definition of "decent". My impression so far is that you will need between 90GB to 100GB of memory to run medium sized (31B dense or ~110B MoE) models with some quantization enabled.
- cjbgkagh 6mo agoI’m running Gemma4 31B (Q8) on my 2 4090s (48GB) with no problem.
- Glemllksdf 6mo agoI have the same setup but tried paperclip ai with it and it seems to me that either i'm unable to setup it properly or multiply agents struggle with this setup. Especially as it seems that paperclip ai and opencode (used for connection) is blowing up the context to 20-30k Any tips around your setup running this? I use lmstudio with default settings and prioritization instead of split.
- cjbgkagh 6mo agoI asked AI for help setting it up. I use 128k context for 31B and 256k context for 26B4A. Ollama worked out of the box for me but I wanted more control with llama.cpp. My command for llama-server: llama-server -m /models/gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf -ngl 99 -sm layer -ts 10,12 --jinja --flash-attn on --cont-batching -np 1 -c 262144 -b 4096 -ub 512 -ctk q8_0 -ctv q8_0 --host 0.0.0.0 --port 8080 --timeout 18000
- littlestymaar 6mo agoNo, GP is excessively restrictive. Llama.cpp supports RAM offloading out of the box. It's going to be slower than if you put everything on your GPU but it would work. And if it's too slow for your taste you can try the quantized version (some Q3 variant should fit) and see how well it works for you.
- angoragoats 6mo agoMacs with unified memory are economical in terms of $/GB of video memory, and they match an optimized/home built GPU setup in efficiency (W/token), but they are slow in terms of absolute performance. With this model, since the number of active parameters is low, I would think that you would be fine running it on your 16GB card, as long as you have, say 32GB of regular system memory. Temper your expectations about speed with this setup, as your system memory and CPU are multiple times slower than the GPU, so when layers spill over you will slow down. To avoid this, there's no need to buy a Mac -- a second 16GB GPU would do the trick just fine, and the combined dual GPU setup will likely be faster than a cheap mac like a Mac mini. Pay attention to your PCIe slots, but as long as you have at least an x4 slot for the second GPU, you'll be fine (LLM inference doesn't need x8 or x16).
- layer8 6mo agoIt’s also doable with AMD Strix Halo.
- jchw 6mo ago32 GiB of VRAM is possible to acquire for less than $1000 if you go for the Arc Pro B70. I have two of them. The tokens/sec is nowhere near AMD or NVIDIA high end, but its unexpectedly kind of decent to use. (I probably need to figure out vLLM though as it doesn't seem like llama.cpp is able to do them justice even seemingly with split mode = row. But still, 30t/s on Gemma 4 (on 26B MoE, not dense) is pretty usable, and you can do fit a full 256k context.) When I get home today I totally look forward to trying the unsloth variants of this out (assuming I can get it working in anything.) I expect due to the limited active parameter count it should perform very well. It's obviously going to be a long time before you can run current frontier quality models at home for less than the price of a car, but it does seem like it is bound to happen. (As long as we don't allow general purpose computers to die or become inaccessible. Surely...)
- zozbot234 6mo agoNew versions of llama.cpp have experimental split-tensor parallelism, but it really only helps with slow compute and a very fast interconnect, which doesn't describe many consumer-grade systems. For most users, pipeline parallelism will be their best bet for making use of multi-GPU setups.
- jchw 6mo agoYeah, I was doing split tensor and it seemed like a wash. The Arc B70s are not huge on compute. Right now I'm only able to run them in PCI-e 5.0 x8 which might not be sufficient. But, a cheap older Xeon or TR seems silly since PCI-e 4.0 x16 isn't theoretically more bandwidth than PCI-e 5.0 x8. So it seems like if that is really still bottlenecked, I'll just have to bite the bullet and set up a modern HEDT build. With RAM prices... I am not sure there is a world where it could ever be worth it. At that point, seems like you may as well go for an obscenely priced NVIDIA or AMD datacenter card instead and retrofit it with consumer friendly thermal solutions. So... I'm definitely a bit conflicted. I do like the Arc Pro B70 so far. Its not a performance monster, but it's quiet and relatively low power, and I haven't run into any instability. (The AMDGPU drivers have made amazing strides, but... The stability is not legendary. :) I'll have to do a bit of analysis and make sure there really is an interconnect bottleneck first, versus a PEBKAC. Could be dropping more lanes than expected for one reason or another too.
- giobox 6mo agoIt's worth noting now there are other machines than just Apple that combine a powerful SoC with a large pool of unified memory for local AI use: > https://www.dell.com/en-us/shop/cty/pdp/spd/dell-pro-max-fcm1253-micro/xcto_fcm1253_usx https://www.dell.com/en-us/shop/cty/pdp/spd/dell-pro-max-fcm... > https://marketplace.nvidia.com/en-us/enterprise/personal-ai-supercomputers/dgx-spark/ https://marketplace.nvidia.com/en-us/enterprise/personal-ai-... > https://frame.work/products/desktop-diy-amd-aimax300/configuration/new https://frame.work/products/desktop-diy-amd-aimax300/configu... etc. But yes, a modern SoC-style system with large unified memory pool is still one of the best ways to do it.
- TechSquidTV 6mo agoMy Mac Studio with 96GB of RAM is maybe just at the low end of passable. It's actually extremely good for local image generation. I could somewhat replace something like Nano Banana comfortably on my machine. But I don't need Nano Banana very much, I need code. While it can, there's no way I would ever opt to use a local model on my machine for code. It makes so much more sense to spend $100 on Codex, it's genuinely not worth discussing. For non-thinking tasks, it would be a bit slower, but a viable alternative for sure.
- slopinthebag 6mo agoYou just need to adjust your workflow to use the smaller models for coding. It's primarily just a case of holding them wrong if you end up with worse outputs.
- coder543 6mo agoNot true. With a MoE, you can offload quite a bit of the model to CPU without losing a ton of performance. 16GB should be fine to run the 4-bit (or larger) model at speeds that are decent. The --n-cpu-moe parameter is the key one on llama-server, if you're not just using -fit on.
- boppo1 6mo agoI've been way out of the local game for a while now, what's the best way to run models for a fairly technical user? I was using llama.cpp in the command line before and using bash files for prompts.
- adrian_b 6mo agoRunning llama-server (it belongs to llama.cpp) starts a HTTP server on a specified port. You can connect to that port with any browser, for chat. Or you can connect to that port with any application that supports the OpenAI API, e.g. a coding assistant harness.
- gunalx 6mo agoRunning q3 xss with full and quantizised context as options on a 16gb gpu and still has pretty decent quality and fitting fine with up to 64k context.