4 ms·

Llama2-70B with MosaicML Inference

• brucethemoose2 3y ago
  > 7B models can easily fit on a single NVIDIA A10 GPU (24GB memory) or NVIDIA A100 GPU (40GB memory), while 70B doesn’t in either Float16 or even Int8 precision. For the 70B parameter model Llama2-70B-Chat, it’s necessary to distribute the workload across multiple GPUs. Why aren't they using 4 bit quantization? That should be enough for a single A100.