4 ms·
I won an ML lottery ticket, then squandered it
- sillysaurusx 4y agoThis is maybe more of a curiosity than a worthwhile submission, but today I was shocked to discover that my run from two days ago was a winning lottery ticket -- it achieved a loss almost 10x better than anything we've trained so far. It was also >4x better than a model with 16x the parameter count. It's the first time in my career that I've seen evidence in the field that tiny ML models can outperform much larger ones, if you happen to win the initialization lottery. FWIW, I was very careful to make sure all of the hyperparams and architecture were exactly the same, otherwise I'd suspect I was fooling myself. It's a one-block autoencoder for RGB frames of 64x32 resolution. Yesterday's run: loss 0.0002 Today's run: loss 0.0018 16-block run: loss 0.0012 It hurts knowing I had a chance to study this phenomenon and let it slip. But at least I can document it in case it's helpful to others.
- kristianp 4y agohttps://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635