4 ms·
Using ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited
by rudyl313 10y ago
Using ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited about ReLU units, he was still showing huge improvements just by using pretraining and later by using dropout, which shouldn't change the cost function.
- argonaut 10y agoDropout helps with convergence/optimization, sure. The existence of a global minimum says nothing about the time required to reach it. Important to note that dropout isn't as common anymore; it's not a huge win.