3 ms·
> If you have a linux machine with GPUs How much VRAM one needs to run inference with llama 2 on a GPU approximately?
by krisoft 3y ago
> If you have a linux machine with GPUs
How much VRAM one needs to run inference with llama 2 on a GPU approximately?
- lolinder 3y agoDepends on which model. I haven't bothered doing it on my 8GB because the only model that would fit is the 7B model quantized to 4 bits, and that model at that size is pretty bad for most things. I think you could have fun with 13B with 12GB VRAM. The full size model would require >35GB even quantized.
- novaRom 3y ago16Gb is minimum to run 7B model with float16 weights; out of the box, with no further efforts.
- zacmps 3y ago70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB). Quantization still seems to have some issues for the 70B model.