3 ms·
How much vram would you need for a 180B model? IIRC the 32B models will barely fit in 24gb of vram on the latest flagship cards. I'd guess 180B is out of reac
by wing-_-nuts 3y ago
How much vram would you need for a 180B model? IIRC the 32B models will barely fit in 24gb of vram on the latest flagship cards. I'd guess 180B is out of reach on everything but A/H100's?
- SparkyMcUnicorn 3y agoAccording to the huggingface blog[0], this model requires 640GB for BF16/FP16 and 320GB for GPTQ/int4. [0] https://huggingface.co/blog/falcon-180b https://huggingface.co/blog/falcon-180b
- sodality2 3y ago> To run inference with the model in full bfloat16 precision you need approximately 8xA100 80GB or equivalent. https://huggingface.co/tiiuae/falcon-180B https://huggingface.co/tiiuae/falcon-180B
- Keyframe 3y agoHalf-joking, but not really, at that size I'd ask how much kW(H) would you need to run this model?