7 ms·
In the future I expect synthetic data will take the lion's share because it is easier to scale. In order to produce it you need a good way to filter out the jun
by visarga 3y ago
In the future I expect synthetic data will take the lion's share because it is easier to scale. In order to produce it you need a good way to filter out the junk, this can be done in many ways, some cheap and other expensive.
- sebzim4500 3y agoI guess, but surely if you run the 20 million books on z-library (I guess there is some duplication) through 4 epoch you get way more than enough data to train current models? You get like 100 trillion tokens, admittedly less after you deduplicate.