4 ms·
70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB). Quantization still seems to have some issues for the 70B model.
by zacmps 3y ago
70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB).
Quantization still seems to have some issues for the 70B model.