2 ms·
> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For so
by alecco 3y ago
> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available
Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK).
Some people report throttling issues in some cases, though.
- iAkashPaul 3y agoI think PyTorch recently added FP8 & even Intel's Neural-Compressor has experimental support. But yeah most HF LLM models are currently just INT8 or FP4/NF4 loaded thanks to BitsandBytes by Tim Dettmers.