3 ms·

I'm building a PC just to run inference on this and the QwQ 32B models. Any suggestions on RAM and GPU I should get?

by henry2023 2y ago

griomnib 2y ago
I have an A6000, it’s about the most affordable for 48 GB VRAM (you can find for a little under $5k sometimes), which is roughly minimum to run a quantized 70b. System RAM doesn’t really matter, but I have 128GB anyway as RAM is pretty cheap.
- paxys 2y ago
  Why not 2 x 4090? Will be cheaper than A6000 if you can manage to find them at msrp, and will perform a lot better.
  - griomnib 2y ago
    My time is worth a lot of money and 2x 4090 is more work, so it’s net more expensive in real terms.
    - BoorishBears 2y ago
      For both inference and training I haven't seen any modern LLM stack take more time for multiple GPUs/tensor parallelism I would take 1 RTX 6000 Ada, but if you mean the pre-Ada 6000, 2x4090 is faster for minimal hassle for most common usecases
      - griomnib 2y ago
        I mean the newest ones. I only do LLM inference, whereas my training load is all DistilBERT models and the A6000 is a beast at cranking those out. Also by “time” I mean my time setting up the machine and doing sys admin. Single card is less hassle.
        BoorishBears 2y ago
        The A6000 predates Ada? There is the RTX 6000 Ada (practically unrelated to the A6000) which has 4090 level performance, that what you're referring to?
        griomnib 2y ago
        This one. https://www.bhphotovideo.com/c/product/1607840-REG/pny_technologies_vcnrtxa6000_pb_nvidia_rtx_a6000_graphic.html https://www.bhphotovideo.com/c/product/1607840-REG/pny_techn...
        zargon 2y ago
        That's an Ampere A6000, one generation older than the Ada A6000. Nvidia decided that confusing model names are a good way to sell old products at a premium.
  - zh3 2y ago
    Running llama3.3:70b here on a pair of eBay Dell RTX3090s in an old (2012!) i3770 workstation - ollama reports 16.67 tokens/sec.
kruxigt 2y ago
[dead]
elorant 2y ago
As a general rule of thumb, for 8-bit quantization you’ll need as much VRAM as the parameters of the model. Half that if you run it at 4-bit. System RAM should be a bit more because you need RAM for the OS too. What most users do is go for second hand 3090s. You get two or three of them depending on quantization.
- pulse7 2y ago
  You also need VRAM for context: the larger the context, the more RAM you need.
loudmax 2y ago
This of course depends on your budget and what you expect to do with these models. For a lot of people, the most cost-effective solution is probably to rent a GPU in the cloud. The limiting factor for running LLMs on consumer grade hardware is generally how much memory your GPU has access to. This is VRAM that's built into the GPU. On non-Apple hardware, the GPU's bandwidth to system RAM is so constrained that you might as well run those operations on the CPU. The cheapest PC solution is usually second-hand RTX 3090's. These can be had for around $700 and they have 24G of VRAM. An RTX 4090 also has 24G of VRAM, but they're about twice as expensive, so for that price you're probably better off getting two 3090's than a single 4090. Llama.cpp runs on the CPU and supports GPU offloading, so you can run a model partly on CPU and partly on GPU. Running anything on the CPU will slow down performance considerably, but it does mean that you can reasonably run a model that's slightly bigger than will fit in VRAM. Quantization works by trimming the least significant digits from the models' parameters, so the model uses less memory at the cost of slight brain damage. A lightly quantized version of QwQ 32B will fit onto a single 3090. A 70B parameter model will need to be quantized down to Q3 or so to run entirely on a 3090. Or you could run a model quantized to Q4 or Q5, but expect only a few tokens per second. We'll need to see how well the quantized versions of this new model behave in practice. Apple's M1-M4 series chips have unified memory so their GPU has access to the system RAM. If you like using a Mac and you were thinking of getting one anyway, they're not a bad choice. But you'll want to get a Mac with as much RAM as you can and they're not cheap.
theropost 2y ago
I've been running this and a quantized version of the QwQ model and comparing the responses - so far, QwQ is working better, though that could change as I use them more and compare the outputs

I'm building a PC just to run inference on this and the QwQ 32B models. Any suggestions on RAM and GPU I should get?

I have an A6000, it’s about the most affordable for 48 GB VRAM (you can find for a little under $5k sometimes), which is roughly minimum to run a quantized 70b. System RAM doesn’t really matter, but I have 128GB anyway as RAM is pretty cheap.

paxys 2y ago
Why not 2 x 4090? Will be cheaper than A6000 if you can manage to find them at msrp, and will perform a lot better.
- griomnib 2y ago
  My time is worth a lot of money and 2x 4090 is more work, so it’s net more expensive in real terms.
  - BoorishBears 2y ago
    For both inference and training I haven't seen any modern LLM stack take more time for multiple GPUs/tensor parallelism I would take 1 RTX 6000 Ada, but if you mean the pre-Ada 6000, 2x4090 is faster for minimal hassle for most common usecases
    - griomnib 2y ago
      I mean the newest ones. I only do LLM inference, whereas my training load is all DistilBERT models and the A6000 is a beast at cranking those out. Also by “time” I mean my time setting up the machine and doing sys admin. Single card is less hassle.
      - BoorishBears 2y ago
        The A6000 predates Ada? There is the RTX 6000 Ada (practically unrelated to the A6000) which has 4090 level performance, that what you're referring to?
        griomnib 2y ago
        This one. https://www.bhphotovideo.com/c/product/1607840-REG/pny_technologies_vcnrtxa6000_pb_nvidia_rtx_a6000_graphic.html https://www.bhphotovideo.com/c/product/1607840-REG/pny_techn...
        zargon 2y ago
        That's an Ampere A6000, one generation older than the Ada A6000. Nvidia decided that confusing model names are a good way to sell old products at a premium.
- zh3 2y ago
  Running llama3.3:70b here on a pair of eBay Dell RTX3090s in an old (2012!) i3770 workstation - ollama reports 16.67 tokens/sec.

kruxigt 2y ago

[dead]

elorant 2y ago

As a general rule of thumb, for 8-bit quantization you’ll need as much VRAM as the parameters of the model. Half that if you run it at 4-bit. System RAM should be a bit more because you need RAM for the OS too. What most users do is go for second hand 3090s. You get two or three of them depending on quantization.

pulse7 2y ago
You also need VRAM for context: the larger the context, the more RAM you need.

loudmax 2y ago

This of course depends on your budget and what you expect to do with these models. For a lot of people, the most cost-effective solution is probably to rent a GPU in the cloud. The limiting factor for running LLMs on consumer grade hardware is generally how much memory your GPU has access to. This is VRAM that's built into the GPU. On non-Apple hardware, the GPU's bandwidth to system RAM is so constrained that you might as well run those operations on the CPU. The cheapest PC solution is usually second-hand RTX 3090's. These can be had for around $700 and they have 24G of VRAM. An RTX 4090 also has 24G of VRAM, but they're about twice as expensive, so for that price you're probably better off getting two 3090's than a single 4090. Llama.cpp runs on the CPU and supports GPU offloading, so you can run a model partly on CPU and partly on GPU. Running anything on the CPU will slow down performance considerably, but it does mean that you can reasonably run a model that's slightly bigger than will fit in VRAM. Quantization works by trimming the least significant digits from the models' parameters, so the model uses less memory at the cost of slight brain damage. A lightly quantized version of QwQ 32B will fit onto a single 3090. A 70B parameter model will need to be quantized down to Q3 or so to run entirely on a 3090. Or you could run a model quantized to Q4 or Q5, but expect only a few tokens per second. We'll need to see how well the quantized versions of this new model behave in practice. Apple's M1-M4 series chips have unified memory so their GPU has access to the system RAM. If you like using a Mac and you were thinking of getting one anyway, they're not a bad choice. But you'll want to get a Mac with as much RAM as you can and they're not cheap.

theropost 2y ago

I've been running this and a quantized version of the QwQ model and comparing the responses - so far, QwQ is working better, though that could change as I use them more and compare the outputs