4 ms·

There was this very interesting paper out of Stanford this last September about pretraining under the unlimited compute but limited data paradigm[0]. Pretty muc

There was this very interesting paper out of Stanford this last September about pretraining under the unlimited compute but limited data paradigm[0]. Pretty much exactly the same thing but with ~200M training tokens instead. [0] https://www.alphaxiv.org/abs/2509.14786 https://www.alphaxiv.org/abs/2509.14786

sdpmas 7mo ago
yeah, we do incorporate some of the findings from the paper in our repo! like aggressive regularization and ensembling.
- _0ffh 7mo ago
  I see you already mention diffusion - iirc there was a result not too long ago that diffusion models keep improving with more epochs for longer than AR models do.
  - sdpmas 7mo ago
    diffusion is promising, but still an open question how much data efficient they are compared to AR. in practice, you can also train AR forever with high enough regularization, so let's see.
    - _0ffh 7mo ago
      Yes, it could go either way of course. Still, just for reference, here's the paper I remembered: https://arxiv.org/pdf/2507.15857 https://arxiv.org/pdf/2507.15857
      - sdpmas 7mo ago
        thanks, here's another one: https://arxiv.org/abs/2511.03276 https://arxiv.org/abs/2511.03276