3 ms·
One thing I noticed is that they have a table in the appendix where they show that some of the datasets used for finetuning were already included in the pretrai
by sebzim4500 3y ago
One thing I noticed is that they have a table in the appendix where they show that some of the datasets used for finetuning were already included in the pretraining.
Is this the first time they have published that there was synthetic data in the GPT-4 pretraining dataset? Maybe that's part of the secret sauce behind why no one can catch GPT-4? I don't recall anyone else mentioning synthetic data.
It certainly explains why there are so many people on the data team at OpenAI.
- visarga 3y agoIn the future I expect synthetic data will take the lion's share because it is easier to scale. In order to produce it you need a good way to filter out the junk, this can be done in many ways, some cheap and other expensive.
- sebzim4500 3y agoI guess, but surely if you run the 20 million books on z-library (I guess there is some duplication) through 4 epoch you get way more than enough data to train current models? You get like 100 trillion tokens, admittedly less after you deduplicate.