7 ms·

> We trained a generative model on CIFAR-10, which we then used to generate ~6 million images. The scale of the dataset was chosen to ensure that it is “virtual

> We trained a generative model on CIFAR-10, which we then used to generate ~6 million images. The scale of the dataset was chosen to ensure that it is “virtually infinite” from the model’s perspective, so that the model never resamples the same data. I don't think so, Tim. They observe identical performance between cifar-10 and cifar-5m, because the generative model for cifar-5m learned to replicate the distribution that cifar-10 was sampled from. It's the same dataset.

kingsleyopara 6y ago
This stood out to me too and puts a big question mark over the whole framework.
dontreact 6y ago
Seriously... What's surprising is that Google has an enormous dataset called JFT where they could have tested this without this confound. Just shrink the images and you can make something cifar-like and something cifar-5m-like
- choppaface 6y ago
  or even just an ablation on Imagenet or some other “large” dataset. Did this paper get accepted because it has five pages of citations and invokes Vapnik in the second sentence?
  - preetum 6y ago
    We have ImageNet experiments in Section 4 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127
    - choppaface 6y ago
      You're not ablating anything there. What happens when Train Infinity (Train 150K) doubles in size (to Train Infinity_2 -> 300K)? What happens when you add an unseen class? These are real-world conditions that hamper existing theoretical estimation of the generalization gap-- the "ideal world" always gets larger. In Bengio's group paper (Predicting the Generalization Gap https://arxiv.org/pdf/1810.00113.pdf https://arxiv.org/pdf/1810.00113.pdf ) they actually do these sorts of ablations. Also, you use K (thousands) and $K$ (latex K) interchangeably; it's really hard to decipher is K is a variable or what you mean.
  - 6gvONxR4sf7o 6y ago
    Why would you criticize them so hard for not doing imagenet experiments when they did do them?
    - choppaface 6y ago
      This article was promoted by Google's PR blog and the conference is ultra selective (probably excessively selective). Hyped results deserve extra criticism. And no, they didn't do any useful ablation experiments.
deleted 6y ago
[deleted]
habitue 6y ago
This looked suspect to me at first as well, but on reflection it doesn't seem so bad. The important aspect here is that they have some large dataset that a model will converge on in less than one epoch. The benefit of generating it from cifar-10 is just that they already have multiple reasonable models to compare with, and they already have the hyperparameters for them. I haven't read the paper, but my guess is that the 50K images in the real-world epoch are not just real images from the cifar-10 dataset, they're 50K random images from cifar-5m. I'm also guessing they don't ever compare performance between a model trained on cifar-10 vs. cifar-5m, they only compare performance of real vs. ideal. So in effect, you can ignore the cifar-10 dataset.
- chroem- 6y ago
  I've had the opportunity to train on an 88 billion example dataset, and while you do see diminishing returns with each additional sample, it is still important to try to cover as much of the dataset as possible. The limiting factor on training on datasets that large seems to be some sort of hysteresis overfit on the earlier samples that prevents the model from generalizing to the entire dataset. Normally it goes unnoticed, but there is still a small amount of overfit that accumulates the longer you train the model, even if every sample is new.
  - ShamelessC 6y ago
    Thanks this insight helped me understand an issues I'm currently having with training actually!
  - gwern 6y ago
    n = 88 billion is larger than I can think of for any DL research I've read about. Can you (ahem) enlarge on that?
    - chroem- 6y ago
      It's an ultra high resolution sales forecast model for a national retailer. It's made somewhat more manageable by the fact that the data is 1D instead of 2D though.
cambalache 6y ago
I dont know if it is depressing or encouraging that a team of learned guys who get paid mid 6-figures come up with this kind of strategy.
ansk 6y ago
I glanced over the paper and it appears that the generative model used for cifar-5m wasn't even trained with a proper train/test split. It was simply trained until the FID on the training set stopped decreasing. That's a pretty good way to overfit a model -- especially on a dataset as small as cifar-10 -- so it's hard to trust that cifar-5m is a decent proxy for the underlying cifar-10 data distribution.
preetum 6y ago
Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127 To clarify some other comments on this post: In all settings, we compare "Real World" and "Ideal World" for the same underlying distribution. Eg, we never compare CIFAR-10 and CIFAR-5m, we only compare "Real World CIFAR-5m" vs "Ideal World CIFAR-5m".
- choppaface 6y ago
  But who actually uses your ImageNet DogBird dataset in an existing study? Part of the problem is that you're inventing your own benchmark. And yet you could have used SVHN and/or the same fixtures (i.e. datasets and ablations) of some of the work you cite e.g. as in "Fantastic Generalization Measures and Where to Find Them" https://arxiv.org/pdf/1912.02178.pdf https://arxiv.org/pdf/1912.02178.pdf The CIFAR-5m result is interesting, but it's misleading to the reader since the dataset is so contrived. And yet you lead with it on your front page. There's wayyyy too much hype going on here.

kingsleyopara 6y ago

This stood out to me too and puts a big question mark over the whole framework.

dontreact 6y ago

Seriously... What's surprising is that Google has an enormous dataset called JFT where they could have tested this without this confound. Just shrink the images and you can make something cifar-like and something cifar-5m-like

choppaface 6y ago
or even just an ablation on Imagenet or some other “large” dataset. Did this paper get accepted because it has five pages of citations and invokes Vapnik in the second sentence?
- preetum 6y ago
  We have ImageNet experiments in Section 4 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127
  - choppaface 6y ago
    You're not ablating anything there. What happens when Train Infinity (Train 150K) doubles in size (to Train Infinity_2 -> 300K)? What happens when you add an unseen class? These are real-world conditions that hamper existing theoretical estimation of the generalization gap-- the "ideal world" always gets larger. In Bengio's group paper (Predicting the Generalization Gap https://arxiv.org/pdf/1810.00113.pdf https://arxiv.org/pdf/1810.00113.pdf ) they actually do these sorts of ablations. Also, you use K (thousands) and $K$ (latex K) interchangeably; it's really hard to decipher is K is a variable or what you mean.
- 6gvONxR4sf7o 6y ago
  Why would you criticize them so hard for not doing imagenet experiments when they did do them?
  - choppaface 6y ago
    This article was promoted by Google's PR blog and the conference is ultra selective (probably excessively selective). Hyped results deserve extra criticism. And no, they didn't do any useful ablation experiments.

deleted 6y ago

[deleted]

habitue 6y ago

This looked suspect to me at first as well, but on reflection it doesn't seem so bad. The important aspect here is that they have some large dataset that a model will converge on in less than one epoch. The benefit of generating it from cifar-10 is just that they already have multiple reasonable models to compare with, and they already have the hyperparameters for them. I haven't read the paper, but my guess is that the 50K images in the real-world epoch are not just real images from the cifar-10 dataset, they're 50K random images from cifar-5m. I'm also guessing they don't ever compare performance between a model trained on cifar-10 vs. cifar-5m, they only compare performance of real vs. ideal. So in effect, you can ignore the cifar-10 dataset.

chroem- 6y ago
I've had the opportunity to train on an 88 billion example dataset, and while you do see diminishing returns with each additional sample, it is still important to try to cover as much of the dataset as possible. The limiting factor on training on datasets that large seems to be some sort of hysteresis overfit on the earlier samples that prevents the model from generalizing to the entire dataset. Normally it goes unnoticed, but there is still a small amount of overfit that accumulates the longer you train the model, even if every sample is new.
- ShamelessC 6y ago
  Thanks this insight helped me understand an issues I'm currently having with training actually!
- gwern 6y ago
  n = 88 billion is larger than I can think of for any DL research I've read about. Can you (ahem) enlarge on that?
  - chroem- 6y ago
    It's an ultra high resolution sales forecast model for a national retailer. It's made somewhat more manageable by the fact that the data is 1D instead of 2D though.

cambalache 6y ago

I dont know if it is depressing or encouraging that a team of learned guys who get paid mid 6-figures come up with this kind of strategy.

ansk 6y ago

I glanced over the paper and it appears that the generative model used for cifar-5m wasn't even trained with a proper train/test split. It was simply trained until the FID on the training set stopped decreasing. That's a pretty good way to overfit a model -- especially on a dataset as small as cifar-10 -- so it's hard to trust that cifar-5m is a decent proxy for the underlying cifar-10 data distribution.

preetum 6y ago

Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv.org/abs/2010.08127 https://arxiv.org/abs/2010.08127 To clarify some other comments on this post: In all settings, we compare "Real World" and "Ideal World" for the same underlying distribution. Eg, we never compare CIFAR-10 and CIFAR-5m, we only compare "Real World CIFAR-5m" vs "Ideal World CIFAR-5m".

choppaface 6y ago
But who actually uses your ImageNet DogBird dataset in an existing study? Part of the problem is that you're inventing your own benchmark. And yet you could have used SVHN and/or the same fixtures (i.e. datasets and ablations) of some of the work you cite e.g. as in "Fantastic Generalization Measures and Where to Find Them" https://arxiv.org/pdf/1912.02178.pdf https://arxiv.org/pdf/1912.02178.pdf The CIFAR-5m result is interesting, but it's misleading to the reader since the dataset is so contrived. And yet you lead with it on your front page. There's wayyyy too much hype going on here.