4 ms·
I think it's more accurate to say that chatGPT was trained than written. The advancements in this space have been primarily pushingb the parameter sets of the m
by zaphar 4y ago
I think it's more accurate to say that chatGPT was trained than written. The advancements in this space have been primarily pushingb the parameter sets of the model high enough that you hit a threshold where you unlock an order of magnitude more capability than before. It's mostly a matter of how much compute and training data you can throw at it.
- astrange 4y agoThe advancements since GPT-3 version 1 have mostly been better training. (Well, RLHF is more data as well, but not in the same sense.) There's also this DeepMind paper (https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training https://www.deepmind.com/publications/an-empirical-analysis-...) saying people need to focus on better training more than data. Of course, there's not much more freely available text available anyway.