3 ms·
> 7B models can easily fit on a single NVIDIA A10 GPU (24GB memory) or NVIDIA A100 GPU (40GB memory), while 70B doesn’t in either Float16 or even Int8 precision
by brucethemoose2 3y ago
> 7B models can easily fit on a single NVIDIA A10 GPU (24GB memory) or NVIDIA A100 GPU (40GB memory), while 70B doesn’t in either Float16 or even Int8 precision. For the 70B parameter model Llama2-70B-Chat, it’s necessary to distribute the workload across multiple GPUs.
Why aren't they using 4 bit quantization? That should be enough for a single A100.