10 ms·
Does this work on any other dataset than Cifar-10 with ResNets? I ask because I worked on reproducing this paper for ICLR 2018 Reproducibility Challenge and the
by Permit 8y ago
Does this work on any other dataset than Cifar-10 with ResNets? I ask because I worked on reproducing this paper for ICLR 2018 Reproducibility Challenge and the paper noted that this was the only setup in which super-convergence would be observed.
The paper's authors didn't succeed with Adam (which this article seems to have overcome) so I'm curious if they attempted this training method on any other datasets?
From the last paragraph of: https://openreview.net/forum?id=H1A5ztj3b https://openreview.net/forum?id=H1A5ztj3b
> Our experiments with Densenets and all our experiments with the Imagenet dataset for a wide variety
of architectures failed to produce the super-convergence phenomenon. Here we will list some of the
experiments we tried that failed to produce the super-convergence behavior. Super-convergence did not occur when training with the Imagenet dataset; that is, we ran Imagenet experiments with Resnets,
ResNeXt, GoogleNet/Inception, VGG, AlexNet, and Densenet without success. Other architectures
we tried with Cifar-10 that did not show super-convergence capability included ResNeXt, Densenet,
and a bottleneck version of Resnet.
EDIT: I see now that they mention a few other datasets: Cars Stanford Dataset and Wikitext-2.
- jph00 8y agoThanks for your work on the reproducibility challenge - we read your results and found them interesting. The good news is that we had a lot more success, although it took many months of work to really make it sing, and bringing in AdamW was an important part of the success. Note that the super-convergence paper has been greatly improved through the recent 1cycle work, which includes a lot of important points to make it work in practice. We tried the most divergent datasets and architectures we could think of, including even AWD-LSTM and QRNN. We got great results for pretty much everything we tried. (We didn't try ResNeXt, since it's so slow in practice, or VGG or AlexNet, since they're largely obsolete. We did look at inception-resnet-v2; I'll have to go back to check to see the results, but IIRC it worked quite well.)