4 ms·
The advancements since GPT-3 version 1 have mostly been better training. (Well, RLHF is more data as well, but not in the same sense.) There's also this DeepMi
by astrange 4y ago
The advancements since GPT-3 version 1 have mostly been better training. (Well, RLHF is more data as well, but not in the same sense.)
There's also this DeepMind paper (https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training https://www.deepmind.com/publications/an-empirical-analysis-...) saying people need to focus on better training more than data. Of course, there's not much more freely available text available anyway.