3 ms·
What would it mean though? If the information is embedded in the network, and we find some performance figure of merit, wouldn't it probably be about the same p
by mensetmanusman 3y ago
What would it mean though? If the information is embedded in the network, and we find some performance figure of merit, wouldn't it probably be about the same performance when normalized to power utilization?
Maybe it's about optimizing every clock cycle?
- eru 3y agoWhy would you expect performance to be constant when normalized to power utilisation? If your 16 bit floats perform about the same as 32 bit floats in an absolute sense, then they will probably perform even better when normalised for power utilisation. And if 16 bit work, 15 bit floats might perform well, too, for all we know. That's what the original commenter was getting at, I think.
- em3rgent0rdr 3y agoYes, that is what I was getting at. Floating point hardware with fewer bits have less complexity and so have smaller transistor die area and power consumption.
- WithinReason 3y agoIf I recall correctly, area/power is proportional to the square of the length of the mantissa.
- mensetmanusman 3y agoI'm guessing that "about the same" will be hard to measure, and that at some point, thermodynamics will dictate the maximum performance per power output (assuming fixed transistor architecture).
- eru 3y agoThermodynamics will dictate the performance in some sense of bit-operations per Joule sense. The more important performance metric is not the number of bit-operations, but the quality of the neural network output. The hypothesis is that fewer bits in your numbers give you the same or nearly the same output quality, but at drastically fewer bit operations performed, and thus less Joule spent.