4 ms·

You link to Guanaco-33B, but Guanaco-65B is much more capable. CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML https://huggingface.co/TheBloke/gu

You link to Guanaco-33B, but Guanaco-65B is much more capable. CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ

wing-_-nuts 3y ago
It irritates me to no end that people don't list the system requirements of various models. How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed? edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?
- orost 3y ago
  You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
- evanchisholm 3y ago
  It's fairly simple to estimate ram requirements based on parameter count: https://blog.eleuther.ai/transformer-math/ https://blog.eleuther.ai/transformer-math/
- TheUninformed 3y ago
  A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models) You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there. https://old.reddit.com/r/LocalLLaMA/wiki/models https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.
- ac29 3y ago
  The first link there gives very specific RAM requirements for each of the models. "Reasonable speed" is subjective, though.

wing-_-nuts 3y ago

It irritates me to no end that people don't list the system requirements of various models. How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed? edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?

orost 3y ago
You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
evanchisholm 3y ago
It's fairly simple to estimate ram requirements based on parameter count: https://blog.eleuther.ai/transformer-math/ https://blog.eleuther.ai/transformer-math/
TheUninformed 3y ago
A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models) You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there. https://old.reddit.com/r/LocalLLaMA/wiki/models https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.
ac29 3y ago
The first link there gives very specific RAM requirements for each of the models. "Reasonable speed" is subjective, though.