3 ms·
What happens on overflow or 0/0 with this format?
by Athas 5y ago
What happens on overflow or 0/0 with this format?
- pm215 5y agoAlternative-halfprec format only applies for conversions (ie to/from single and double prec); data-processing operations on fp16 ignore the AHP bit and always assume IEEE format. If you convert a NaN from single/double into AHP you get a zero with the same sign as the input; if you convert an infinity you get the max/min representable number; in both cases the InvalidOp fp exception bit is set. The upshot is that it's basically an in-memory storage format, and all the actual data-processing gets done at either single or double precision.