3 ms·
Wow, that implies that given a large enough model, early stopping is actually a mistake! Who’d a thunk it?
by _0ffh 7y ago
Wow, that implies that given a large enough model, early stopping is actually a mistake! Who’d a thunk it?
- gwern 7y agoBut maybe it's not a big one (the reduction on the other side is still fairly small), and it may not be one at all (do you have the compute budget? look at how many epoches it takes to reach the other side and actually realize any gains!). More theoretically interesting than practical advice, I'd say.
- _0ffh 7y agoYup, I've seen on the graph it looks like at least an order of magnitude more training time until it starts to get interesting. Obviously this is only one example, so that factor might vary mightily. It might be an option for someone who is really serious about squeezing out the last drop of generalisation performance. I guess we'll see how things work out, because some folk are bound to try.