3 ms·
Dataset counts, but also the number of total parameters in the network, i.e. capacity.
by kookamamie 3y ago
Dataset counts, but also the number of total parameters in the network, i.e. capacity.
- visarga 3y agoAgreed, it's in my first phrase "as long as you can efficiently train models of the same size, what counts is the dataset". But useful sizes are just a few. 7, 13, 35, 70, 120B - because they are targeted to various families of GPUs. A 2T model I can't run or too expensive to use on APIs is of no use. Not just dataset size, but data quality matters just as much, and diversity. I believe LLMs will train mostly on synthetic data engineered to have extreme diversity and very high quality. This kind of data confers 5x gains in efficiency as demonstrated by Microsoft in the Phi-1.5 paper.
- whimsicalism 3y agoThere are really 3 axes - the number of tokens, the number of parameters, and the number of flops per pass.
- p1esk 3y agoThis says nothing about data quality.
- Salgat 3y agoTo add, if you have a model that's too large for the dataset, you end up in a situation where it just memorizes the training dataset and performs poorly on the test data. That's why GPT works so good, you only pass the training data through it once, and there's so much good training data that it works in spite of the model being absolutely massive.
- whimsicalism 3y agoit only memorizes the dataset if you give it multiple passes and are bad at regularizing overparameterization is fine