3 ms·

Was thinking the same, so 1T might bring you to the league of GPT-4. Actually in the best case, since it seems that meta, google, openai and so on have the mos

Was thinking the same, so 1T might bring you to the league of GPT-4. Actually in the best case, since it seems that meta, google, openai and so on have the most talent. Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?

lossolo 3y ago
> Maybe 10T? 100T? I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
- MichaelRazum 3y ago
  Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.
  - mrkeen 3y ago
    > they are able to generate data if needed. If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.
    - astrange 3y ago
      The point of generating data is so you can teach the model its distribution, not so it can memorize it.