6 ms·
I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU
by XCSme 3y ago
I am planning to buy a new GPU.
If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well?
Also, does it run considerably better than on a GPU with 12GB of VRAM?
I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).
- jasonjmcghee 3y ago> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version
- kwerk 3y agoI have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note
- _ea1k 3y agoollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
- K0balt 3y agoI run mixtral 6 bit quant very happily on my MacBook with 64 gb.
- deleted 3y ago[deleted]
- ranger_danger 3y agoI'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.
- chpatrick 3y agoThe quantized one works fine on my 24GB 3090.
- XCSme 3y agoSorry, it was from memory. I have those models in Ollama: I have those: dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)
- XCSme 3y agoI have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).
- InitEnabler 3y agoWhat docker flags are you running?
- XCSme 3y agoNone? The default ones from their docs. The Docker also shows minimal usage for the ollama server which is also strange.
- Havoc 3y agoThe smaller quants still require a 24gb card. 16 might work but doubt it
- llm_trw 3y ago>If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? No, it can't run at all. >I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time). That is not mixtral, that is mistral 7b. The 1080ti is slower than running inference on current generation threadripper cpus.
- XCSme 3y agoI have those: dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB) The CPU is 5900x.
- XCSme 3y ago> No, it can't run at all. https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.
- wokwokwok 3y agoDid you check your GPU utilization? Typically when it runs that way it runs on the CPU, not the GPU. Are you sure you're actually offloading any work to the GPU? At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all. The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#additional-options https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.
- XCSme 3y agoI have to check, something does indeed seem weird, especially with the PC freezing like that. Maybe it runs on the CPU. > quantized version Yes, it is 4bit quantized, but still has 24.6GB
- PheonixPharts 3y agoWhile GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where the time to response isn't that big of a deal, and not fine-tuning the models locally (which I also believe GPUs are still superior for). But if your concern is "how big of a model can I run" vs "Can I have close to real time chat", the unified memory approach is superior.
- XCSme 3y agoI already have 128GB of RAM (DDR4), and was wondering if upgrading from a 1080ti (12GB) to a 4070ti super (16GB), would make a big difference. I assume the FP32 and FP16 operations are already a huge improvement, but also the 33% increased VRAM might lead to fewer swaps between VRAM and RAM.
- zozbot234 3y agoThat's system memory, not unified memory. Unified means that all or most of it is going to be directly available to the Apple Silicon GPU.
- giancarlostoro 3y agoThis is the key factor here. I have a 3080, with 16GB of Memory, but still have to run some models on CPU since the memory is not unified at all.
- loudmax 3y agoI have an RTX 3080 with 10GB of VRAM. I'm able to run models larger than 10GB using llama.cpp and offloading to the GPU as much as can fit into VRAM. The remainder of the model runs on CPU + regular RAM. The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. When I run a model bigger than 10GB, say Mixtral or Llama 70B with GPU offloading, my CPU will run full tilt and the VRAM is full, but the GPU processor itself will operate far below full capacity. I think what is happening here is that the model layers that are offloaded to the GPU do their processing, then the GPU spends most of the time waiting for the much slower CPU to do its thing. So in my case, I think upgrading to a faster GPU would make little to no difference when running the bigger models, so long as the VRAM is capped at the same level. But upgrading to a GPU with more VRAM, even a slower GPU, should make the overall speed faster for bigger models because the GPU would spend less time waiting for the CPU. (Of course, models that fit entirely into VRAM will run faster on a faster GPU). In my case, the amount of VRAM absolutely seems to be the performance bottleneck. If I do upgrade, it will be for a GPU with more VRAM, not necessarily a GPU with more processing power. That has been my experience running llama.cpp. YMMV.
- lxe 3y agoGet 2 pre-owned 3090s. You will easily be able to run 70b or even 120b quantized models.
- Zambyte 3y agoI genuinely recommend considering AMD options. I went with a 7900 XTX because it has the most VRAM for any $1000 card (24 GB). NVIDIA cards at that price point are only 16 GB. Ollama and other inference software works on ROCm, generally with at most setting an environment variable now. I've even run Ollama on my Steam Deck with GPU inferencing :)
- XCSme 3y agoI ended up getting a 2nd hand 3090 for 680€. Funnily, I think the card is new (smells new) and unused, most likely a scalper bought it and couldn't sell it.
- Zambyte 3y agoNice, that's definitely a sweet deal
- XCSme 3y agoThanks, I chose a 3090 instead of 4070ti, it was around $200 cheaper and has 24GB vs 16GB VRAM and a similar performance. The only drawback is the 350W TDP. I still struggle with the RAM issue on Ollama, where it uses 128GB/128GB RAM for Mixtral 24.6GB, even though Docker limit is set to 90GB. Docker seems pretty buggy on Windows...
- speedylight 3y agoQuantized models will run well, otherwise inference might be really really slow or the client crashes all together with some CUDA out of memory error.