https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g-GGML https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPT... Try this model, wh

https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g-GGML https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPT... Try this model, which uses excludes some of the "ethics" filtering. Also, IDK what you mean by a "small" GPU, but I am getting ~3.5 tokens/sec on a laptop 2060, and I think I can squeeze more out of it. And there are methods for splitting up the load in small VRAM pools.