3 ms·
Also, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B. In Llama V1, 7B was clever but kinda dumb, 13B was still sh
by brucethemoose2 3y ago
Also, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B.
In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format.
16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.
- zacmps 3y agoI don't know, maybe I'm desensitized by GPT4 but I'm running Llama2 70B and I certainly wouldn't call it 'disturbingly good'.
- brucethemoose2 3y agoBase Llama is not good at all, but the finetunes are quite good at their niches. Llama in general is not great for code completion and fact retrieval, from what I have tried
- thewataccount 3y agoCheck out the hugging face leader board and try one of the finetunes. I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.
- zacmps 3y agoThanks! That's useful to know. Unfortunately it looks like a lot of them are non-commercial which rules them out for me but there's a couple that look like they might work.
- brucethemoose2 3y agoIts rather outdated. Search for "70b" on huggingface, sort by date, and you will find more cutting edge models. And you can run this locally with enough combined RAM + VRAM.
- ripvanwinkle 3y agoDoes it have to be exclusively vram to be useful or is shared ram also useful here. I have a system that reports 8gb dedicated to GPU with another 15 that is shared and thus a GPU memory of 23 GB
- brucethemoose2 3y agoDriver shared memory is effectively useless. The way it accesses CPU memory is (for now) extremely slow. First some background: llama is divided into prompt ingestion code, and "layers" for actually generating the tokens. There are different offloading schemes, but what llama.cpp specifically does is map the layers to different devices. For example, ~7GB of the model could reside on the 8GB GPU, and the other ~9GB would live on the CPU. During runtime, the prompt is ingested by the GPU all at once (which doesn't take much VRAM), and then for each word, the layers are run sequentially. So the first ~half of a word would run on your GPU, and the last half would run on your CPU, and the alternation repeats till all the words are generated. The beauty of llama.cpp is that its cpu token generation is (compared to other llama runtimes) extremely fast. So offloading even half or two thirds of a model to a decent CPU is not so bad.
- ripvanwinkle 3y agoThank you for the explanation