4 ms·
This was a really fun read-thank you! If you don't mind answering, I've got a few questions: 1. "Vanishing gradients after 2-3 layers"-does this mean that the
by SatvikBeri 14y ago
This was a really fun read-thank you! If you don't mind answering, I've got a few questions:
1. "Vanishing gradients after 2-3 layers"-does this mean that the partial derivatives tend to be smaller on the higher layers, and therefore the network finds local minima that aren't very useful?
2. Step 3 (p 18) mentions that the outputs are not continuous variables, they're binary. What's the reasoning behind that?
- tansey 14y ago1. Basically. It means that the network has a hard time pulling itself in any direction since the weights in the deeper layers are never really adjusted by very much. 2. It's been a while since I read the paper, but I believe that the justification has to do with the proof of convergence of Gibbs sampling. I haven't tried using continuous values, so I can't give an intuition for what happens in those cases.