4 ms·
But the tricks/advancements I mentioned are not changing the function. They changed the initial weights and how the cost function was explored.
by rudyl313 10y ago
But the tricks/advancements I mentioned are not changing the function. They changed the initial weights and how the cost function was explored.
- aab0 10y agoBut they do change the function. You use RELU now, instead of pretraining.
- rudyl313 10y agoUsing ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited about ReLU units, he was still showing huge improvements just by using pretraining and later by using dropout, which shouldn't change the cost function.
- argonaut 10y agoDropout helps with convergence/optimization, sure. The existence of a global minimum says nothing about the time required to reach it. Important to note that dropout isn't as common anymore; it's not a huge win.