To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it,

To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way. Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.