3 ms·

Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv

Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127 To clarify some other comments on this post: In all settings, we compare "Real World" and "Ideal World" for the same underlying distribution. Eg, we never compare CIFAR-10 and CIFAR-5m, we only compare "Real World CIFAR-5m" vs "Ideal World CIFAR-5m".

choppaface 6y ago
But who actually uses your ImageNet DogBird dataset in an existing study? Part of the problem is that you're inventing your own benchmark. And yet you could have used SVHN and/or the same fixtures (i.e. datasets and ablations) of some of the work you cite e.g. as in "Fantastic Generalization Measures and Where to Find Them" https://arxiv.org/pdf/1912.02178.pdf https://arxiv.org/pdf/1912.02178.pdf The CIFAR-5m result is interesting, but it's misleading to the reader since the dataset is so contrived. And yet you lead with it on your front page. There's wayyyy too much hype going on here.