3 ms·

I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data

by ftxbro 3y ago

I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help. A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.

hooande 3y ago
Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.
- ftxbro 3y ago
  > Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
- jasmer 3y ago
  [dead]
- imtringued 3y ago
  Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.
ilaksh 3y ago
I think TinyStories is a promising direction. I just wish we had an alternative to GPT-4 because supposedly you can't use it to train other models. Or maybe clarification of people publicly saying they are going to ignore that restriction.

by ftxbro 3y ago

hooande 3y ago

Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.

ftxbro 3y ago
> Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
jasmer 3y ago
[dead]
imtringued 3y ago
Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.

ilaksh 3y ago

I think TinyStories is a promising direction. I just wish we had an alternative to GPT-4 because supposedly you can't use it to train other models. Or maybe clarification of people publicly saying they are going to ignore that restriction.