4 ms·
Seriously... What's surprising is that Google has an enormous dataset called JFT where they could have tested this without this confound. Just shrink the image
by dontreact 6y ago
Seriously...
What's surprising is that Google has an enormous dataset called JFT where they could have tested this without this confound. Just shrink the images and you can make something cifar-like and something cifar-5m-like
- choppaface 6y agoor even just an ablation on Imagenet or some other “large” dataset. Did this paper get accepted because it has five pages of citations and invokes Vapnik in the second sentence?
- preetum 6y agoWe have ImageNet experiments in Section 4 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127
- choppaface 6y agoYou're not ablating anything there. What happens when Train Infinity (Train 150K) doubles in size (to Train Infinity_2 -> 300K)? What happens when you add an unseen class? These are real-world conditions that hamper existing theoretical estimation of the generalization gap-- the "ideal world" always gets larger. In Bengio's group paper (Predicting the Generalization Gap https://arxiv.org/pdf/1810.00113.pdf https://arxiv.org/pdf/1810.00113.pdf ) they actually do these sorts of ablations. Also, you use K (thousands) and $K$ (latex K) interchangeably; it's really hard to decipher is K is a variable or what you mean.
- 6gvONxR4sf7o 6y agoWhy would you criticize them so hard for not doing imagenet experiments when they did do them?
- choppaface 6y agoThis article was promoted by Google's PR blog and the conference is ultra selective (probably excessively selective). Hyped results deserve extra criticism. And no, they didn't do any useful ablation experiments.