3 ms·

Yes, I read that. The problem is that you need to train the bias to get comparable results. Also, please note that you cannot have zero mean outputs of residua

by thesz 7y ago

Yes, I read that. The problem is that you need to train the bias to get comparable results. Also, please note that you cannot have zero mean outputs of residual in the res x = x + relu (Ax + b) and you obviously can have zero mean outputs in the subtraction case (res x = x - relu (Ax + b)). The very fact that in one case you have zero mean outputs and in other case you don't brings me to necessity to point you to SELU: https://towardsdatascience.com/selu-make-fnns-great-again-snn-8d61526802a9 https://towardsdatascience.com/selu-make-fnns-great-again-sn... This SELU paper demonstrates, in my opinion, the benefits of having zero mean outputs. (I have to say that in my experiments SELU was not all that beneficial, but other means that bring zero means into existence were) I think that residual neural network is capable to route around the case of having to learn non-zero means in inputs. So you are right in stating that these two cases will be indistinguishable. I just have to say that having subtraction instead of addition helps neural net to train faster and get better accuracy just because training process have less things to learn.

bonoboTP 7y ago
I'm fairly sure I was right in the above analysis which would imply the minus version cannot have any substantial advantages. > please note that you cannot have zero mean outputs of residual in the res x = x + relu (Ax + b) and you obviously can have zero mean outputs in the subtraction case (res x = x - relu (Ax + b) This is incorrect. You seem to assume x is positive an therefore adding something relu'd onto it will take it further from zero, while your subtractive one can pull it towards and beyond zero. The problem in this reasoning is that in your version you sequentially subtract the residuals so you have the exact symmetric effect, getting further away from zero. It's like a left hand and a right hand. Not the same, but have the same effect. I said all I could at this point. If you still have your experiments set up, just try negating your input to the network and initialize randomly. The accuracies observed will be indistinguishable from using your variant. The network is not the same but the training procedure yields a sample from the same distribution.
- thesz 7y ago
  Sorry. I consulted the code and it turned out I used relu(Ax + b) - x as a modified residual layer. I hope it clears things up.

by thesz 7y ago

bonoboTP 7y ago

I'm fairly sure I was right in the above analysis which would imply the minus version cannot have any substantial advantages. > please note that you cannot have zero mean outputs of residual in the res x = x + relu (Ax + b) and you obviously can have zero mean outputs in the subtraction case (res x = x - relu (Ax + b) This is incorrect. You seem to assume x is positive an therefore adding something relu'd onto it will take it further from zero, while your subtractive one can pull it towards and beyond zero. The problem in this reasoning is that in your version you sequentially subtract the residuals so you have the exact symmetric effect, getting further away from zero. It's like a left hand and a right hand. Not the same, but have the same effect. I said all I could at this point. If you still have your experiments set up, just try negating your input to the network and initialize randomly. The accuracies observed will be indistinguishable from using your variant. The network is not the same but the training procedure yields a sample from the same distribution.

thesz 7y ago
Sorry. I consulted the code and it turned out I used relu(Ax + b) - x as a modified residual layer. I hope it clears things up.