4 ms·
> can potentially reduce 95% energy cost by elementwise floating point tensor multiplications and 80% energy cost of dot products It this were about convolutio
by visarga 2y ago
> can potentially reduce 95% energy cost by elementwise floating point tensor multiplications and 80% energy cost of dot products
It this were about convolutional nets then optimizing compute would be a much bigger deal. Transformers are lightweight on compute and heavy on memory. The weakest link in the chain is fetching the model weights into the cores. The 95% and 80% energy reductions cited are for the multiplication operations in isolation, not for the entire inference process.
- lifthrasiir 2y agoI'm also sure that fp8 is small enough that multiplication can really be done in a much simpler circuit than larger fp formats. Even smaller formats like fp4 would be able to just use a lookup table, and that makes them more like sort-of-standardized quantization schemes.
- tankenmate 2y agoi suspect that you could do fp8 with log tables and interpolation if you really wanted to (compared to the memory required for the model it's peanuts), it just turns into a LUT (log table look up) and bit shift (interpolation). so again, memory bandwidth is the limiting factor for transformers (as far as energy is concerned).
- lifthrasiir 2y agoThis time though LUT exists in a circuit, which is much more efficient than typical memory lookup. Such LUT would have to exist per each ALU though, so it can't be too large.
- brilee 2y agofp4/fp8 for neural networks don't work the way you think they do - they are merely compression formats - a set of, say, 256 fp32 weights from 1 neuron are lossily turned into 1 max value (stored in fp32 precision) and 256 fp4/fp8 numbers. Those compressed numbers are multiplied by the fp32 number at runtime to restore the original weights and full fp32 multiplication + additions are executed.
- imjonse 2y agoWith w8a8 quantization the hw (>= hopper) can do the heavy math in fp8 twice as fast as fp16.
- SuchAnonMuchWow 2y agoThe goal of this type of quantization is to move the multiplication by the fp32 rescale factor outside of the dot-product accumulation. So the multiplications+additions are done on fp8/int8/int4/whatever (when the hardware support those operators of course) and accumulated in a fp32 or similar, and only the final accumulator is multiplied by the rescale factor in fp32.
- lifthrasiir 2y agoYou are correct that the accumulation (i.e. additions in dot products) has to be done in a higher precision, however the multiplication can still be done via LUT. (Source: I currently work at a hardware-accelerated ML hardware startup.)
- rajnathani 2y agoThat's how Nvidia's mixed precision training worked with FP32-FP16, but it isn't the case for Bfloat16 on TPUs and maybe (I'm not sure) FP8 training on Nvidia Hopper GPUs.
- bee_rider 2y agoWhat is fp4? 3 bits of exponent and one of mantissa?
- wruza 2y agoSEEM (sign, exp, mantissa)
- bee_rider 2y agoInteresting… I guess it must be biased, m*2^ee would leave like half of the limited space wasted, so 1.m*2^ee? I always wonder with these tiny formats if 0 should even be represented…
- wruza 2y agoI’m not a binary guy that much, but iirc all floats are 1.m*2^e — “1.” is always there except for subnormals. There’s also SEEE FP4 which is basically +-2^([u?]int3). https://medium.com/@harrietfiagbor/floating-points-and-deep-learning-dl-supports-8ee35053ea01 https://medium.com/@harrietfiagbor/floating-points-and-deep-...
- kendalf89 2y agoMaybe this technique can be used for training then since that is a lot more compute intensive?
- SuchAnonMuchWow 2y agoIts worse than that: the energy gains are when comparing computations made with fp32, but for fp8 the multipliers are really tiny and the adder/shifters represent a largest part of the operators (energy-wise and area-wise) and this paper will only have small gains. On fp8, the estimated gate count of fp8 multipliers is 296 vs. 157 with their technique, so the power gain on the multipliers will be much lower (50% would be a more reasonable estimation), but again for fp8 the additions in the dot products are a large part of the operations. Overall, its really disingenuous to claim 80% power gain and small drop in accuracy, when the power gain is only for fp32 operations and the small drop in accuracy is only for fp8 operators. They don't analyze the accuracy drop in fp32, and don't present the power saved for fp8 dot product.
- bobsyourbuncle 2y agoI’m new to neural nets, when should one use fp8 vs fp16 vs fp32?
- deleted 2y ago[deleted]
- reissbaker 2y agoBasically no one uses FP32 at inference time. BF16/FP16 is typically considered unquantized, whereas FP8 is lightly quantized. That being said there's pretty minimal quality loss at FP8 compared to 16-bit typically; Llama 3.1 405b, for example, only benchmarks around ~1% worse when run at FP8: https://blog.vllm.ai/2024/07/23/llama31.html https://blog.vllm.ai/2024/07/23/llama31.html Every major inference provider other than Hyperbolic Labs runs Llama 3.1 405b at FP8, FWIW (e.g. Together, Fireworks, Lepton), so to compare against FP32 is misleading to say the least. Even Hyperbolic runs it at BF16. Pretraining is typically done in FP32, although some labs (e.g. Character AI, RIP) apparently train in INT8: https://research.character.ai/optimizing-inference/ https://research.character.ai/optimizing-inference/
- tarasglek 2y agoSambaNova does bf16
- imjonse 2y agoThat is true for single user/light inference only. For training and batch inference you can get compute bound fast enough.
- saagarjha 2y agoThat really depends on what you're doing. Trying to feed a tensor core is pretty hard–they're really fast.
- woadwarrior01 2y agoPre-fill (even in the single batch case) and multi-batch decoding are still compute dominated. The oft repeated trope of "decoder only transformer inference is bottle-necked on memory bandwidth" is only strictly true in the single batch decoding case, because you're mostly doing vector matrix mults when the batch size is one.
- ein0p 2y agoNot even single batch. If you want reasonable latency per token (TPOT) even larger batches do not give you high compute utilization during extend. It’s only when you don’t care about TPOT at all, and your model is small enough to leave space for a large batch on an 8 GPU host, that’s when you could get decent utilization. That’s extend only - it’s easy to get high utilization in prefill.
- mikewarot 2y agoImagine if you had a systolic array large enough that all the weights would only have to be loaded once at startup. Eliminating the memory-compute bottleneck of the von Neumann architecture could make this quite a bit more efficient.
- api 2y agoSounds like the awesome architecture for transformers would be colocation of memory and compute.
- Joker_vD 2y agoYes, that's why we generally run them on GPUs.
- moffkalast 2y agoGPUs that pull a kilowatt when running yes. This might actually work on an FPGA if the addition doesn't take too many clock cycles compared to matmuls which were too slow.
- phkahler 2y agoThat's why we need a row of ALUs in RAM chips. Read a row of DRAM and use it in a vector operation. With the speed of row reading, the ALU could take many cycles per operation to limit area.
- namibj 2y agoThe big problem is that DRAM is extremely secretive about their processes, and they largely don't do that well for logic.
- api 2y agoGPUs are better but I'm thinking of even tighter coupling, like an integrated architecture.
- h_tbob 2y agoBro... they are NOT lightweight on compute!