3 ms·
Yes, but as he also says, the best hyperparameters are near the boundary. You want to be a bit greedy here.
by vintermann 3y ago
Yes, but as he also says, the best hyperparameters are near the boundary. You want to be a bit greedy here.
- alexmolas 3y agoThe way the author defines best hyperparameters is also a bit weird. To assign a "score" to each pair of HP the author sums all the losses during training (ie score = ∑ᵀₜ₌₀ lₜ) instead of just taking the value of the loss at the last epoch (score = l_T). This makes the converging solutions with high learning rates appear better since they have lower losses since the beginning (so the sum is smaller), but it doesn't mean that the final value of the loss is better than the ones with smaller learning rates.