3 ms·

> as long as you have a sane set of regularization techniques (or have we even ditched those too?). Maybe. In the context of natural language at least, Transf

> as long as you have a sane set of regularization techniques (or have we even ditched those too?). Maybe. In the context of natural language at least, Transformers require less and less data to reach the same result as you increase the number of parameters. No regularization needed. See Figure 2 in the paper Scaling Laws for Neural Language Models (2001.08361). It's quite odd. Who knows if that will hold for other domains, like protein folding. It may very well be the case though, since AFAIK DeepMind's folding model used attention to reach these landmark results.

dnautics 6y ago
I remember reading somewhere that this was likely to be the consequence of how distance metrics work in high-dimensional manifolds.
rprenger 6y ago
In figure 2, those curves are all from one epoch each right? Which is to say the model has never seen the same data twice? It's true you don't need regularization if you've never seen the same data twice, but that's a similar regularization to early stopping. You'd expect the larger number of parameters would make the training error drop faster with fewer tokens as well due to improved optimizability. But rather than "larger models need less data", I'd say the take away is more "larger models need fewer steps to optimize training error". None of the models get "good" until they've seen a number of tokens similar to the number of parameters. Unless you've run many epochs on small data sets and seen the same results, in which case that's pretty weird/cool.
- martingoodson 6y ago
  Figure 4 is for dataset size and seems to shows the same thing.
  - rprenger 6y ago
    Yeah, figure 4 is more clear. This is the early stopped loss though, so the regularization is more explicit. If you trained the large models to completion on small data sets they would do much worse on test error due to overfitting. It is interesting that larger models with regularization (early stopping) seems to work better than than training smaller models to convergence though.

dnautics 6y ago

I remember reading somewhere that this was likely to be the consequence of how distance metrics work in high-dimensional manifolds.

rprenger 6y ago

In figure 2, those curves are all from one epoch each right? Which is to say the model has never seen the same data twice? It's true you don't need regularization if you've never seen the same data twice, but that's a similar regularization to early stopping. You'd expect the larger number of parameters would make the training error drop faster with fewer tokens as well due to improved optimizability. But rather than "larger models need less data", I'd say the take away is more "larger models need fewer steps to optimize training error". None of the models get "good" until they've seen a number of tokens similar to the number of parameters. Unless you've run many epochs on small data sets and seen the same results, in which case that's pretty weird/cool.

martingoodson 6y ago
Figure 4 is for dataset size and seems to shows the same thing.
- rprenger 6y ago
  Yeah, figure 4 is more clear. This is the early stopped loss though, so the regularization is more explicit. If you trained the large models to completion on small data sets they would do much worse on test error due to overfitting. It is interesting that larger models with regularization (early stopping) seems to work better than than training smaller models to convergence though.