3 ms·

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

charcircuit 2mo ago
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
- leoooodias 2mo ago
  You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
  - revolvingthrow 2mo ago
    The bitrot would be excessive
  - BonerWiener 2mo ago
    IP over Avian Carriers: https://www.rfc-editor.org/info/rfc2549/ https://www.rfc-editor.org/info/rfc2549/
  - chuckadams 2mo ago
    Coincidentally that’s also Google’s new method for distributing Android sources.
- kQq9oHeAz6wLLS 2mo ago
  The key to running in lower amounts of VRAM is patience. It'll be slow, but it'll work.
segbrk 2mo ago
Yes, but unless they support NVLink (they don't), it's quite slow.
sharmajai 2mo ago
I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF.
- Balinares 2mo ago
  Wow, interesting. What KV cache quantization do you use?
- zenoprax 2mo ago
  For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.
xlayn 2mo ago
I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k. llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1 if you have an igpu and want to exclude or just use some gpus you can use --device Vulkan3,Vulkan2,Vulkan1 in my case vulkan because of amd, you can see your devices with llama-server2 --list-devices Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)
gruturo 2mo ago
Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server. If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race. (Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split) Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.
strictnein 2mo ago
Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card. Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.
- jambalaya8 2mo ago
  How many do you recommend?
  - strictnein 2mo ago
    I've run 4, 6, and 8. Adding more video cards increases total available memory, so you can load larger models, but there are definitely some drawbacks. More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them. Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context. And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall. For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models. For hardware: A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-wrx80e-sage-se-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.
walrus01 2mo ago
Yes, google "llama-server split model multiple GPU" for some concrete examples.

Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.

leoooodias 2mo ago
You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
- revolvingthrow 2mo ago
  The bitrot would be excessive
- BonerWiener 2mo ago
  IP over Avian Carriers: https://www.rfc-editor.org/info/rfc2549/ https://www.rfc-editor.org/info/rfc2549/
- chuckadams 2mo ago
  Coincidentally that’s also Google’s new method for distributing Android sources.
kQq9oHeAz6wLLS 2mo ago
The key to running in lower amounts of VRAM is patience. It'll be slow, but it'll work.

segbrk 2mo ago

Yes, but unless they support NVLink (they don't), it's quite slow.

sharmajai 2mo ago

I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF.

Balinares 2mo ago
Wow, interesting. What KV cache quantization do you use?
zenoprax 2mo ago
For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.

xlayn 2mo ago

I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k. llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1 if you have an igpu and want to exclude or just use some gpus you can use --device Vulkan3,Vulkan2,Vulkan1 in my case vulkan because of amd, you can see your devices with llama-server2 --list-devices Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)

gruturo 2mo ago

Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server. If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race. (Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split) Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.

strictnein 2mo ago

Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card. Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.

jambalaya8 2mo ago
How many do you recommend?
- strictnein 2mo ago
  I've run 4, 6, and 8. Adding more video cards increases total available memory, so you can load larger models, but there are definitely some drawbacks. More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them. Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context. And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall. For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models. For hardware: A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-wrx80e-sage-se-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.

walrus01 2mo ago

Yes, google "llama-server split model multiple GPU" for some concrete examples.