2 ms·
Did you use GPU inference? If so, how much memory is required?
by tarruda 4y ago
Did you use GPU inference? If so, how much memory is required?
- valine 4y agoYeah I'm using GPU inference. The vicuna 13B model uses 26.3GB of VRAM with my setup. I'm running it split between two rtx4090s which gives me about 20 token/s.