The 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized. There are already efforts underway with GPTQ libraries but I have found they incur a subs

The 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized. There are already efforts underway with GPTQ libraries but I have found they incur a substantial performance penalty, with the benefit of consuming much lower VRAM. EDIT: I had a look at the repo, it appears the Vicuna model is using 3bit quantization.