3 ms·
> I wonder if you could also receive similar results by simply turning some connections "off" for a while, only training the rest of the connections. Then turn
by visarga 9y ago
> I wonder if you could also receive similar results by simply turning some connections "off" for a while, only training the rest of the connections. Then turn the missing connections on a gain and, randomly turning a few other connections off, and continue training.
Isn't that almost like Dropout? Dropout does what you say but it changes the dropped set of neurons for each individual example.
- vanderZwan 9y agohttps://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf Looks pretty close, yeah. So this is basically dividing the network into two groups, and give each group a second weight: one for those that are on, zero for those that are off. Has anyone tried more variants of this? Using random bundling + various levels of amplification of such bundles, so that backpropagation affects various networks differently?
- visarga 9y agoThere is also this: Mixture of experts. A massive neural net from Google, with multiple submodules that are activated depending on input. https://medium.com/@thoszymkowiak/google-brains-new-super-fast-and-highly-accurate-ai-the-mixture-of-experts-layer-dd3972c25663 https://medium.com/@thoszymkowiak/google-brains-new-super-fa...