3 ms·
This idea is at least partially in use with regularisation and dropout. The difference at least with dropout is that the "killed" neurons are then massaged back
by antome 10y ago
This idea is at least partially in use with regularisation and dropout. The difference at least with dropout is that the "killed" neurons are then massaged back into the network in order become useful again.
- shmageggy 10y agoAgreed that this is another way of framing the problem of regularizing a network. Rather than starting with a big network and penalizing complexity, they are starting with a simple network and adding complexity. To that end, I'd've liked to see a comparison to dropout or L1/L2 regularization.
- visarga 10y agoBiological neurons themselves are stochastic so they have an internal "dropout" that doesn't seem to hurt, on the contrary, these perturbations and imperfect communication increase learning ability.