4 ms·
Llama 7B will squeeze on a 6GB GPU quantized. Maybe even less with EX2 quantization. Foundational model trainers dont seem to worry about quantization much, th
by brucethemoose2 3y ago
Llama 7B will squeeze on a 6GB GPU quantized. Maybe even less with EX2 quantization.
Foundational model trainers dont seem to worry about quantization much, they just throw the base model out there and then let the community take care of easing the runtime requirements.