3 ms·

Can't they just switch to all exponential numbers (no mantissa)? There's plenty of precedent in image processing where the pixel values are usually supposed to

by rini17 4y ago

Can't they just switch to all exponential numbers (no mantissa)? There's plenty of precedent in image processing where the pixel values are usually supposed to be exponentiated by gamma factor before doing any linear operations.

PaulHoule 4y ago
I think there is something wrongheaded about the "tiny float" approach, but it seems better than the alternatives. If you are working with the log of numbers then you don't get negative numbers, which most neural networks want. I guess you could use logs and still have a sign bit. Alternatively it seems you could scale your numbers appropriately so that you can just use fixed point everywhere. I find it hard to believe that a particular activation level is going to have a meaningful transition between 5×10⁻⁴ and 6×10⁻⁴ and also a meaningful transition between 3x10⁷ and 4x10⁷... And if it does there is something bad going on. (It's a "bad smell" in linear models that you are seeing large-magnitude positive and negative coefficients, these days people would say it is a sign of "overfitting" and one way to fight it is to set a cost for large coefficients in the model... I can't imagine it's really that different in neural models) One of the great things you learn at the graduate level in physics is that a complex calculation (say Chandrasekhar's calculations about how stars work) should be partitioned into a set of scaling factors on the outside and an internal complex calculation which is in dimensionless units which never get as large or small as the lifetime of the universe or the mass of a proton but rather you tend to get ratios that aren't so huge in magnitude, and from that viewpoint you don't need such a crazy range of exponents in the interior calculation... Adding to my feeling that floating point is as much a problem as a solution.

by rini17 4y ago

PaulHoule 4y ago

I think there is something wrongheaded about the "tiny float" approach, but it seems better than the alternatives. If you are working with the log of numbers then you don't get negative numbers, which most neural networks want. I guess you could use logs and still have a sign bit. Alternatively it seems you could scale your numbers appropriately so that you can just use fixed point everywhere. I find it hard to believe that a particular activation level is going to have a meaningful transition between 5×10⁻⁴ and 6×10⁻⁴ and also a meaningful transition between 3x10⁷ and 4x10⁷... And if it does there is something bad going on. (It's a "bad smell" in linear models that you are seeing large-magnitude positive and negative coefficients, these days people would say it is a sign of "overfitting" and one way to fight it is to set a cost for large coefficients in the model... I can't imagine it's really that different in neural models) One of the great things you learn at the graduate level in physics is that a complex calculation (say Chandrasekhar's calculations about how stars work) should be partitioned into a set of scaling factors on the outside and an internal complex calculation which is in dimensionless units which never get as large or small as the lifetime of the universe or the mass of a proton but rather you tend to get ratios that aren't so huge in magnitude, and from that viewpoint you don't need such a crazy range of exponents in the interior calculation... Adding to my feeling that floating point is as much a problem as a solution.