4 ms·

> Networks which generalise better are harder to break yep, that was my whole thesis dissertation. also you can actually include some confusion during the trai

> Networks which generalise better are harder to break yep, that was my whole thesis dissertation. also you can actually include some confusion during the training itself by adding noise to the weights or the transfer function and the learning algorithm will find more stable solutions that are more robust on its own. we used that to produce network that were more resilient to being quantized when moved to low power devices having only fixed point arithmetic available.

pilooch 9y ago
Interesting, any public pointer to your dissertation ?
- LoSboccacc 9y ago
  I've lost the original files, I digitalized some of the important part back, here https://imgur.com/a/w2BCy https://imgur.com/a/w2BCy it was 2004-2006, we used weight quantization as noise function and a modified levemberg-marquad for training, matlab was the hot stuff back then and by today standard everything is ancient, so there's that - the training alghorithm was changed to train toward a flatter area of the noise function, so it used the second derivative to hold gradient descent and overfitting. I can answer specific questions if you want.
  - pilooch 9y ago
    Many thanks for the thorough answer! Will read and ask questions if any. We have customers we help quantize the models for their FPGAs and this may be an avenue we could test.
    - p1esk 9y ago
      How do you currently quantize the models?
maffydub 9y ago
Isn't this the reason that dropout works so well as a regularizer? In dropout, you select 50% (say) of the nodes in one layer of your network at random and force them to pass on no signal (neither positive nor negative). You then train the network to still return the result. This is effectively training the network to be robust to failures (i.e. "harder to break"). While writing this, I've realized that we've both assumed that not only "networks which generalize better are harder to break" but also "networks which are harder to break generalize better". I think that's true, but worth being explicit. (Still developing my intuition for machine learning, so I could be wrong...)
- arimorcos 9y ago
  Author here. Indeed, our work is closely related to dropout. However, as we discuss in the paper (https://arxiv.org/abs/1803.06959 https://arxiv.org/abs/1803.06959), dropout doesn't really encourage the network to be robust to deletion generally; it only encourages this robustness up until the dropout fraction used in training. So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is. Hope that was helpful!
  - LoSboccacc 9y ago
    > but the network could completely fail once 51% of the units are deleted also the network could have weird results for dropout values around 20%-30% depending on how the robustness was 'learned'
  - maffydub 9y ago
    Thanks for your response - that's very interesting! I haven't read the paper yet, but have added it to my queue.

pilooch 9y ago

Interesting, any public pointer to your dissertation ?

LoSboccacc 9y ago
I've lost the original files, I digitalized some of the important part back, here https://imgur.com/a/w2BCy https://imgur.com/a/w2BCy it was 2004-2006, we used weight quantization as noise function and a modified levemberg-marquad for training, matlab was the hot stuff back then and by today standard everything is ancient, so there's that - the training alghorithm was changed to train toward a flatter area of the noise function, so it used the second derivative to hold gradient descent and overfitting. I can answer specific questions if you want.
- pilooch 9y ago
  Many thanks for the thorough answer! Will read and ask questions if any. We have customers we help quantize the models for their FPGAs and this may be an avenue we could test.
  - p1esk 9y ago
    How do you currently quantize the models?

maffydub 9y ago

Isn't this the reason that dropout works so well as a regularizer? In dropout, you select 50% (say) of the nodes in one layer of your network at random and force them to pass on no signal (neither positive nor negative). You then train the network to still return the result. This is effectively training the network to be robust to failures (i.e. "harder to break"). While writing this, I've realized that we've both assumed that not only "networks which generalize better are harder to break" but also "networks which are harder to break generalize better". I think that's true, but worth being explicit. (Still developing my intuition for machine learning, so I could be wrong...)

arimorcos 9y ago
Author here. Indeed, our work is closely related to dropout. However, as we discuss in the paper (https://arxiv.org/abs/1803.06959 https://arxiv.org/abs/1803.06959), dropout doesn't really encourage the network to be robust to deletion generally; it only encourages this robustness up until the dropout fraction used in training. So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is. Hope that was helpful!
- LoSboccacc 9y ago
  > but the network could completely fail once 51% of the units are deleted also the network could have weird results for dropout values around 20%-30% depending on how the robustness was 'learned'
- maffydub 9y ago
  Thanks for your response - that's very interesting! I haven't read the paper yet, but have added it to my queue.