3 ms·
I’m new to neural nets, when should one use fp8 vs fp16 vs fp32?
by bobsyourbuncle 2y ago
I’m new to neural nets, when should one use fp8 vs fp16 vs fp32?
- deleted 2y ago[deleted]
- reissbaker 2y agoBasically no one uses FP32 at inference time. BF16/FP16 is typically considered unquantized, whereas FP8 is lightly quantized. That being said there's pretty minimal quality loss at FP8 compared to 16-bit typically; Llama 3.1 405b, for example, only benchmarks around ~1% worse when run at FP8: https://blog.vllm.ai/2024/07/23/llama31.html https://blog.vllm.ai/2024/07/23/llama31.html Every major inference provider other than Hyperbolic Labs runs Llama 3.1 405b at FP8, FWIW (e.g. Together, Fireworks, Lepton), so to compare against FP32 is misleading to say the least. Even Hyperbolic runs it at BF16. Pretraining is typically done in FP32, although some labs (e.g. Character AI, RIP) apparently train in INT8: https://research.character.ai/optimizing-inference/ https://research.character.ai/optimizing-inference/
- tarasglek 2y agoSambaNova does bf16
- ericlewis 2y agoHigher the precision the better. Use what works within your memory constraints.
- jasonjmcghee 2y agoWith serious diminishing returns. At inference time, no reason to use fp64 and should probably use fp8 or less. The accuracy loss is far less than you'd expect. AFAIK Llama 3.2 3B fp4 will outperform Llama 3.2 1B at fp32 in accuracy and speed, despite 8x precision.