4 ms·
Can I run a fp8 quant with 96gb VRAM?
by isatty 1mo ago
Can I run a fp8 quant with 96gb VRAM?
- cpburns2009 1mo agoOnly VRAM? Unlikely unless you can also load the whole model into regular RAM. The previous 3.5 release was 250gb at BF16, so FP8 would likely be around 125gb. Your best best is FP4/Q4.
- isatty 1mo agoVery sad. I try not to go below q8.