3 ms·

I feel this is one of the major ways that most pundits failed with their “the data is going to run out” predictions. First and foremost a chatbot generates ple

I feel this is one of the major ways that most pundits failed with their “the data is going to run out” predictions. First and foremost a chatbot generates plenty of new data (plus feedback!), but you can also commission new high-quality content. Karpathy recently commented that GPT-3 needs so many parameters because most of the training set is garbage, and that he expects eventually a GPT-2 sized model could reach GPT-3 level, if trained exclusively on high-quality textbooks. This is one of the ways you get textbooks to push the frontier capabilities.

from-nibly 2y ago
At a good cost though? Last time I checked generating good data costs a tiny bit more than an http request to somebody elses website.
- theptip 2y ago
  If the cost is not “good enough”, why are the big guys buying a lot of it?
llm_trw 2y ago
I've not done pre-training for LLMs, but years ago I generated a completely synthetic dataset for table recognition using an off the shelf document segmentation model, raw TeX, a random table generator, a discriminator and an evolutionary algorithm to generate different styles of tables. The project got killed due to management but I still got results on that dataset better than state of the art in 2023 with no human annotation. The Venn diagram of people who know TeX well enough to write a modular script for table generation with metadata and people who know how to train LLMs has an intersection of a dozen people I imagine.
- theptip 2y ago
  This is a great example. Areas where an expert can write a synthetic data generator (eg code, physics simulators, etc) are the dream scenario here. It seems to me there is a huge amount of unharvested low-hanging fruit here, for example IIUC GPT is not trained on synthetic code in languages other than Python (and maybe JS, I don’t recall).

from-nibly 2y ago

At a good cost though? Last time I checked generating good data costs a tiny bit more than an http request to somebody elses website.

theptip 2y ago
If the cost is not “good enough”, why are the big guys buying a lot of it?

llm_trw 2y ago

I've not done pre-training for LLMs, but years ago I generated a completely synthetic dataset for table recognition using an off the shelf document segmentation model, raw TeX, a random table generator, a discriminator and an evolutionary algorithm to generate different styles of tables. The project got killed due to management but I still got results on that dataset better than state of the art in 2023 with no human annotation. The Venn diagram of people who know TeX well enough to write a modular script for table generation with metadata and people who know how to train LLMs has an intersection of a dozen people I imagine.

theptip 2y ago
This is a great example. Areas where an expert can write a synthetic data generator (eg code, physics simulators, etc) are the dream scenario here. It seems to me there is a huge amount of unharvested low-hanging fruit here, for example IIUC GPT is not trained on synthetic code in languages other than Python (and maybe JS, I don’t recall).