4 ms·
The sparsity encouraged by ReLUs is one reason they are used but not to the exclusion of all other activations. ReLU variants can indeed outperform standard ReL
by jimfleming 10y ago
The sparsity encouraged by ReLUs is one reason they are used but not to the exclusion of all other activations. ReLU variants can indeed outperform standard ReLU[0] but sometimes sparsity is a desired property. For example, in generative models.
[0] https://arxiv.org/abs/1505.00853 https://arxiv.org/abs/1505.00853