2 ms·
Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.
by piker 3y ago
Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.
- xpe 3y agoAllocating more bits for each parameter increases precision, by definition. But that doesn’t come for free.* So it is useful to optimize network performance for a given number of total parameter bytes. I haven’t done a recent literature review, but my hand-wavy guessplanation is that a NN (as a whole) can adapt to relatively low precision parameters. Up to a point. * In general. Given actual hardware designs, there are places where you have slack in the system. So adding some extra parameters, e.g. to fully utilize a GPU’s core’s threads (e.g. 32), might actually cost you nothing.