4 ms·Are you using quantization? I’ve gotten very good results from the float16 13B vicuna model.by valine 4y agoAre you using quantization? I’ve gotten very good results from the float16 13B vicuna model.tarruda 4y agoDid you use GPU inference? If so, how much memory is required?valine 4y agoYeah I'm using GPU inference. The vicuna 13B model uses 26.3GB of VRAM with my setup. I'm running it split between two rtx4090s which gives me about 20 token/s.