5 ms·
How much VRAM does it need? I haven't run a local model yet, but I did recently pick up a 16GB GPU, before they were discontinued.
by palmotea 7mo ago
How much VRAM does it need? I haven't run a local model yet, but I did recently pick up a 16GB GPU, before they were discontinued.
- trvz 7mo agoIf you have to ask then your GPU is too small. With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
- FusionX 7mo agoAren't 4bits model decent? Since, this is an MOE model, I'm assuming it should have respectable tk/s, similar to previous MOE models.
- palmotea 7mo ago> If you have to ask then your GPU is too small. What's the minimum memory you need to run a decent model? Is it pretty much only doable by people running Macs with unified memory?
- bfivyvysj 7mo agoA bit like asking how long is a piece of string.
- utilize1808 7mo agoObviously going to depend on your definition of "decent". My impression so far is that you will need between 90GB to 100GB of memory to run medium sized (31B dense or ~110B MoE) models with some quantization enabled.
- cjbgkagh 7mo agoI’m running Gemma4 31B (Q8) on my 2 4090s (48GB) with no problem.
- Glemllksdf 7mo agoI have the same setup but tried paperclip ai with it and it seems to me that either i'm unable to setup it properly or multiply agents struggle with this setup. Especially as it seems that paperclip ai and opencode (used for connection) is blowing up the context to 20-30k Any tips around your setup running this? I use lmstudio with default settings and prioritization instead of split.
- cjbgkagh 7mo agoI asked AI for help setting it up. I use 128k context for 31B and 256k context for 26B4A. Ollama worked out of the box for me but I wanted more control with llama.cpp. My command for llama-server: llama-server -m /models/gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf -ngl 99 -sm layer -ts 10,12 --jinja --flash-attn on --cont-batching -np 1 -c 262144 -b 4096 -ub 512 -ctk q8_0 -ctv q8_0 --host 0.0.0.0 --port 8080 --timeout 18000
- littlestymaar 7mo agoNo, GP is excessively restrictive. Llama.cpp supports RAM offloading out of the box. It's going to be slower than if you put everything on your GPU but it would work. And if it's too slow for your taste you can try the quantized version (some Q3 variant should fit) and see how well it works for you.
- angoragoats 7mo agoMacs with unified memory are economical in terms of $/GB of video memory, and they match an optimized/home built GPU setup in efficiency (W/token), but they are slow in terms of absolute performance. With this model, since the number of active parameters is low, I would think that you would be fine running it on your 16GB card, as long as you have, say 32GB of regular system memory. Temper your expectations about speed with this setup, as your system memory and CPU are multiple times slower than the GPU, so when layers spill over you will slow down. To avoid this, there's no need to buy a Mac -- a second 16GB GPU would do the trick just fine, and the combined dual GPU setup will likely be faster than a cheap mac like a Mac mini. Pay attention to your PCIe slots, but as long as you have at least an x4 slot for the second GPU, you'll be fine (LLM inference doesn't need x8 or x16).
- layer8 7mo agoIt’s also doable with AMD Strix Halo.
- jchw 7mo ago32 GiB of VRAM is possible to acquire for less than $1000 if you go for the Arc Pro B70. I have two of them. The tokens/sec is nowhere near AMD or NVIDIA high end, but its unexpectedly kind of decent to use. (I probably need to figure out vLLM though as it doesn't seem like llama.cpp is able to do them justice even seemingly with split mode = row. But still, 30t/s on Gemma 4 (on 26B MoE, not dense) is pretty usable, and you can do fit a full 256k context.) When I get home today I totally look forward to trying the unsloth variants of this out (assuming I can get it working in anything.) I expect due to the limited active parameter count it should perform very well. It's obviously going to be a long time before you can run current frontier quality models at home for less than the price of a car, but it does seem like it is bound to happen. (As long as we don't allow general purpose computers to die or become inaccessible. Surely...)
- zozbot234 7mo agoNew versions of llama.cpp have experimental split-tensor parallelism, but it really only helps with slow compute and a very fast interconnect, which doesn't describe many consumer-grade systems. For most users, pipeline parallelism will be their best bet for making use of multi-GPU setups.
- jchw 7mo agoYeah, I was doing split tensor and it seemed like a wash. The Arc B70s are not huge on compute. Right now I'm only able to run them in PCI-e 5.0 x8 which might not be sufficient. But, a cheap older Xeon or TR seems silly since PCI-e 4.0 x16 isn't theoretically more bandwidth than PCI-e 5.0 x8. So it seems like if that is really still bottlenecked, I'll just have to bite the bullet and set up a modern HEDT build. With RAM prices... I am not sure there is a world where it could ever be worth it. At that point, seems like you may as well go for an obscenely priced NVIDIA or AMD datacenter card instead and retrofit it with consumer friendly thermal solutions. So... I'm definitely a bit conflicted. I do like the Arc Pro B70 so far. Its not a performance monster, but it's quiet and relatively low power, and I haven't run into any instability. (The AMDGPU drivers have made amazing strides, but... The stability is not legendary. :) I'll have to do a bit of analysis and make sure there really is an interconnect bottleneck first, versus a PEBKAC. Could be dropping more lanes than expected for one reason or another too.
- giobox 7mo agoIt's worth noting now there are other machines than just Apple that combine a powerful SoC with a large pool of unified memory for local AI use: > https://www.dell.com/en-us/shop/cty/pdp/spd/dell-pro-max-fcm1253-micro/xcto_fcm1253_usx https://www.dell.com/en-us/shop/cty/pdp/spd/dell-pro-max-fcm... > https://marketplace.nvidia.com/en-us/enterprise/personal-ai-supercomputers/dgx-spark/ https://marketplace.nvidia.com/en-us/enterprise/personal-ai-... > https://frame.work/products/desktop-diy-amd-aimax300/configuration/new https://frame.work/products/desktop-diy-amd-aimax300/configu... etc. But yes, a modern SoC-style system with large unified memory pool is still one of the best ways to do it.
- TechSquidTV 7mo agoMy Mac Studio with 96GB of RAM is maybe just at the low end of passable. It's actually extremely good for local image generation. I could somewhat replace something like Nano Banana comfortably on my machine. But I don't need Nano Banana very much, I need code. While it can, there's no way I would ever opt to use a local model on my machine for code. It makes so much more sense to spend $100 on Codex, it's genuinely not worth discussing. For non-thinking tasks, it would be a bit slower, but a viable alternative for sure.
- slopinthebag 7mo agoYou just need to adjust your workflow to use the smaller models for coding. It's primarily just a case of holding them wrong if you end up with worse outputs.
- coder543 7mo agoNot true. With a MoE, you can offload quite a bit of the model to CPU without losing a ton of performance. 16GB should be fine to run the 4-bit (or larger) model at speeds that are decent. The --n-cpu-moe parameter is the key one on llama-server, if you're not just using -fit on.
- boppo1 7mo agoI've been way out of the local game for a while now, what's the best way to run models for a fairly technical user? I was using llama.cpp in the command line before and using bash files for prompts.
- adrian_b 7mo agoRunning llama-server (it belongs to llama.cpp) starts a HTTP server on a specified port. You can connect to that port with any browser, for chat. Or you can connect to that port with any application that supports the OpenAI API, e.g. a coding assistant harness.
- gunalx 7mo agoRunning q3 xss with full and quantizised context as options on a 16gb gpu and still has pretty decent quality and fitting fine with up to 64k context.
- WithinReason 7mo agoIt's on the page: Precision Quantization Tag File Size 1-bit UD-IQ1_M 10 GB 2-bit UD-IQ2_XXS 10.8 GB UD-Q2_K_XL 12.3 GB 3-bit UD-IQ3_XXS 13.2 GB UD-Q3_K_XL 16.8 GB 4-bit UD-IQ4_XS 17.7 GB UD-Q4_K_XL 22.4 GB 5-bit UD-Q5_K_XL 26.6 GB 16-bit BF16 69.4 GB
- palmotea 7mo agoThanks! I'd scanned the main content but I'd been blind to the sidebar on the far right.
- Aurornis 7mo agoAdditional VRAM is needed for context. This model is a MoE model with only 3B active parameters per expert which works well with partial CPU offload. So in practice you can run the -A(N)B models on systems that have a little less VRAM than you need. The more you offload to the CPU the slower it becomes though.
- Glemllksdf 7mo agoIsn't that some kind of gambling if you offload random experts onto the CPU? Or is it only layers but that would affect all Experts?
- dragonwriter 7mo agoPretty sure all partial offload systems I’ve seen work by layers, but there might be something else out there.
- namibj 7mo agoSpeculative decoding is already gambling.
- JKCalhoun 7mo ago"16-bit BF16 69.4 GB" Is that (BF16) a 16-bit float?
- zozbot234 7mo agoShould run just fine with CPU-MoE and mmap, but inference might be a bit slow if you have little RAM.
- Ladioss 7mo agoYou can run 25-30b model easily if you use Q3 or Q4 quants and llama-server with a pretty long list of options.
- tommy_axle 7mo agoPick a decent quant (4-6KM) then use llama-fit-params and try it yourself to see if it's giving you what you need.
- gunalx 7mo agoI habe found llama-fit sometimes just selects a way to conservative load with VRAM to spare.