4 ms·
For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.
by philoko 3y ago
For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.
- aarnphm 3y agoCurrently on main, 8bit and 4bit quant is supported One can simply do ```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4``` Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization