3 ms·

> This theoretical exploration offers perspective that is distinct from the literature which attributes the success of low-precision neural networks to its regu

by zmmmmm 3y ago

> This theoretical exploration offers perspective that is distinct from the literature which attributes the success of low-precision neural networks to its regularization effect They only tease in the abstract what their "perspective" is, defining only by what it is not. I can't see it in the conclusion either. Unfortunately not up for reading through their formalisation to try and understand the point. I have a denoising autoencoder that manages to ridiculously overfit a complex data set despite bottlenecking on a tiny set of neurons, which I attribute to it managing to exploit all the bits of precision within the bottleneck to effectively make it hold far more information than you would naively think. So I'm sceptical if they are saying this is not a real effect.

bckr 3y ago
> Remark. It is important to acknowledge that previous research has highlighted the ability of neural networks to tolerate noise and even act as a regularization method. However, our proposed analysis differs in two ways. Firstly, it is the first comparison of its kind that focuses on pure 16-bit neural networks. Secondly, our analysis offers a unique quantification approach that distinguishes it from previous explanations related to regularization. Our approach examines floating-point errors and formalizes the concept of tolerance, allowing for its quantification as a measure of comparison with floating-point errors. > Explanation of the Lemma.... in the worst case.... But as long as... the two classifiers M16 and M32 must have the same classification result on x. So my rough understanding is that they're saying "Hey, it's not/not only because of regularization; it's because fp16 is really as good as fp32 for what we need it to do (but more efficient?"
tysam_and 3y ago
This is sort of my party trick I do in a number of places in diagnosing NNs from scare environmental data, let me give it a whirl. It sounds like your network has far too many degrees of freedom. Reducing it via residuals or some other method will likely help with that. Additionally, some L2 weight decay, switching to SGD+OneCycle, don't forget to BatchNorm before every activation as well. If this is a newer-style attention-Unet ala StyleGAN then that would be a confusing result as transformers seem to be pretty okay with not immediately collapsing to that kind of thing if I understand correctly. Barring all of that, swapped labels can be a surprising reason for complex data to overfit as it forces the network into a memorization-only mode with very little chance for generalization. Let me know if I got it correct/close for you. :) :D <3 :))))

by zmmmmm 3y ago

bckr 3y ago

> Remark. It is important to acknowledge that previous research has highlighted the ability of neural networks to tolerate noise and even act as a regularization method. However, our proposed analysis differs in two ways. Firstly, it is the first comparison of its kind that focuses on pure 16-bit neural networks. Secondly, our analysis offers a unique quantification approach that distinguishes it from previous explanations related to regularization. Our approach examines floating-point errors and formalizes the concept of tolerance, allowing for its quantification as a measure of comparison with floating-point errors. > Explanation of the Lemma.... in the worst case.... But as long as... the two classifiers M16 and M32 must have the same classification result on x. So my rough understanding is that they're saying "Hey, it's not/not only because of regularization; it's because fp16 is really as good as fp32 for what we need it to do (but more efficient?"

tysam_and 3y ago

This is sort of my party trick I do in a number of places in diagnosing NNs from scare environmental data, let me give it a whirl. It sounds like your network has far too many degrees of freedom. Reducing it via residuals or some other method will likely help with that. Additionally, some L2 weight decay, switching to SGD+OneCycle, don't forget to BatchNorm before every activation as well. If this is a newer-style attention-Unet ala StyleGAN then that would be a confusing result as transformers seem to be pretty okay with not immediately collapsing to that kind of thing if I understand correctly. Barring all of that, swapped labels can be a surprising reason for complex data to overfit as it forces the network into a memorization-only mode with very little chance for generalization. Let me know if I got it correct/close for you. :) :D <3 :))))