3 ms·
Initialization? > W = np.random.normal(0, np.sqrt(2/(h.shape[0] + layer_dim[i])), size = (layer_dim[i], h.shape[0])) A N(0, sqrt(2/width)) would produce nega
by gwern 9y ago
Initialization?
> W = np.random.normal(0, np.sqrt(2/(h.shape[0] + layer_dim[i])), size = (layer_dim[i], h.shape[0]))
A N(0, sqrt(2/width)) would produce negative values.
- chillee 9y agoI was talking about the graphs here: https://i.imgur.com/M6P71aC.jpg https://i.imgur.com/M6P71aC.jpg I missed that he's not storing activations for those graphs, he's storing activations+batch norm. See my edit.