4 ms·

> Maybe 10T? 100T? I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate trai

> Maybe 10T? 100T? I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.

MichaelRazum 3y ago
Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.
- mrkeen 3y ago
  > they are able to generate data if needed. If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.
  - astrange 3y ago
    The point of generating data is so you can teach the model its distribution, not so it can memorize it.