3 ms·
Kind of a weird article, as 8-bit quantization is widely used in production for neural networks for a number of years now. The title of the article is a bit mis
by eklitzke 4y ago
Kind of a weird article, as 8-bit quantization is widely used in production for neural networks for a number of years now. The title of the article is a bit misleading since it's widely known that 8-bit quantization does work and is extremely effective at improving inference throughput and latency. I'm not 100% sure if I'm reading the article correctly since it's a bit oblique, but it seems like the news here is that work is being done to formally specify a cross-vendor FP8 standard, as what exists right now is de facto standards from different vendors.
- FL33TW00D 4y agoINT8 quantisation has been used in production for years. FP8 has not.
- Dylan16807 4y agoFP8 provides some nice accuracy benefits over INT8 but if you swap it out that doesn't affect your overhead.
- ftufek 4y agoThe article mentions the 8 bit quantization, I believe this is about training in fp8 as native format. The latest GPUs provide huge flops for those, Tim Dettmers updated his gpu article and he talks about this, the claim is 0.66 PFLOPS for an RTX 4090.
- tysam_and 4y agoThis is very different, FP8 is a new and active area of research that has new hardware that is just supporting it. You are thinking of (U)INT8 quantization