3 ms·
> The "depth-three" model in this paper is a bit strange in that their second layer has only one output, so the third linear layer doesn't have any effect. I wo
by jkabrg 9y ago
> The "depth-three" model in this paper is a bit strange in that their second layer has only one output, so the third linear layer doesn't have any effect. I would have called this "depth two"; but it is internally consistent with their definition of depth.
No, it does have an effect: It takes a linear combination of the outputs of the previous layer, and then applies a non-linearity $\sigma'$. If $\sigma'$ is the logistic function, then the output of the last layer is a probability.
- yorwba 9y agoNo, the last layer is just a linear layer, there's no non-linearity. That's what's strange about their definition. The depth-three network only applies a non-linearity twice, which would conventionally be labeled as depth two.
- jkabrg 9y agoYou're wrong. See section 5.1. I've drawn a graph of it that shows it to be a classical NN with a single output node.
- yorwba 9y agoYes, it is a classical NN with a single output node. I'm not disputing that, I just think their calculation of depth is strange. The network only applies the sigmoid function twice, and would ordinarily be regarded as having a depth of two. The third linear layer is fixed to multiplying by 1, which is what I meant by "has no effect". (Did you miss that I was talking about the third layer?)
- jkabrg 9y agoHere's a diagram [1] of a one-hidden-layer NN. It has 2 activation functions. Their NN is of the same type. [1] - https://raw.githubusercontent.com/qingkaikong/blog/master/40_ANN_part3_step_by_step_MLP/figures/figure4_hidden_perceptron.jpg https://raw.githubusercontent.com/qingkaikong/blog/master/40...
- yorwba 9y agoThe question is, how deep is that network?