12 ms·

Super-human cognition? Hard to say. GPT-4 does raise the possibility of a machine writing smarter text than a human. What perplexes me is that since GPT is a p

Super-human cognition? Hard to say. GPT-4 does raise the possibility of a machine writing smarter text than a human. What perplexes me is that since GPT is a predictor, it shouldn’t be able to write the smartest text - it should write the average text (since that has the largest frequency in the training set). Yet this does not seem to be the case. Is it inevitable that despite the quality of the data, better models output text which supercedes its training, or could the GPT-4 secret sauce be RLHF weighing intelligent answers higher?

dehrmann 4y ago
It's just a really good cover band.
bitL 4y ago
I think you misunderstood how the generation of text works. For each new token it samples probabilities given previous tokens, not averages, then chooses some token from the top k as the next one with rules that penalize repetition of some order. Moreover, there is no upper bound for transformers found yet, i.e. the larger the model is and the more data is used for training, the better it performs. It's literally about who is able to throw more money at it at this point, with some closely guarded secrets like warm up steps, training schedules etc. There is also the overfitting effect where one pushes training far beyond overfitting (validation loss growing again) as with transformers at some point the overfitting stops, validation loss starts dropping again and that's when the magic starts happening and money are burnt for scale.
- circuit10 4y ago
  You missed the point they were making, which is that the probabilities it’s predicting are based on what it expects the average text in its training set to look like. The loss you’re talking about is how closely its answers match the training set, not how clever the answers sound (though with RLHF it’s different). A model producing better text than what’s in its training set would be penalised for not matching it closely and quickly learn to not do that
withinboredom 4y ago
Clearly written “average” text will always be better than unclear “smart text”
espadrine 4y ago
> could the GPT-4 secret sauce be RLHF weighing intelligent answers higher? That part is one of the rare things that the technical report addresses. In Appendix B[0], they show that RLHF does not improve capabilities on human tasks. It does improve alignment. To me, this is an indication that they performed better scaling analysis and pretrained until it no longer improved. As the Chinchilla paper showed, GPT-3 was undertrained, so any fine-tuning also improved its capabilities. To address your question though, consider two things: first, there are many more ways to be incorrect than to be correct, so even just prediction will find correct answers more likely than incorrect ones. Second, the corpus goes through a significant filtering process; they didn't just feed the raw Twitter firehose to it. [0]: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.)
- minzi 4y ago
  > (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.) I find this to be particularly egregious. Such an obvious false front is a red flag to me.
selfhoster11 4y ago
It's not that it's smarter, it's that it's faster. If I have to choose between a larger quantity of code or higher quality of code while keeping the time constant, then I'd prefer the former.

dehrmann 4y ago

It's just a really good cover band.

bitL 4y ago

I think you misunderstood how the generation of text works. For each new token it samples probabilities given previous tokens, not averages, then chooses some token from the top k as the next one with rules that penalize repetition of some order. Moreover, there is no upper bound for transformers found yet, i.e. the larger the model is and the more data is used for training, the better it performs. It's literally about who is able to throw more money at it at this point, with some closely guarded secrets like warm up steps, training schedules etc. There is also the overfitting effect where one pushes training far beyond overfitting (validation loss growing again) as with transformers at some point the overfitting stops, validation loss starts dropping again and that's when the magic starts happening and money are burnt for scale.

circuit10 4y ago
You missed the point they were making, which is that the probabilities it’s predicting are based on what it expects the average text in its training set to look like. The loss you’re talking about is how closely its answers match the training set, not how clever the answers sound (though with RLHF it’s different). A model producing better text than what’s in its training set would be penalised for not matching it closely and quickly learn to not do that

withinboredom 4y ago

Clearly written “average” text will always be better than unclear “smart text”

espadrine 4y ago

> could the GPT-4 secret sauce be RLHF weighing intelligent answers higher? That part is one of the rare things that the technical report addresses. In Appendix B[0], they show that RLHF does not improve capabilities on human tasks. It does improve alignment. To me, this is an indication that they performed better scaling analysis and pretrained until it no longer improved. As the Chinchilla paper showed, GPT-3 was undertrained, so any fine-tuning also improved its capabilities. To address your question though, consider two things: first, there are many more ways to be incorrect than to be correct, so even just prediction will find correct answers more likely than incorrect ones. Second, the corpus goes through a significant filtering process; they didn't just feed the raw Twitter firehose to it. [0]: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.)

minzi 4y ago
> (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.) I find this to be particularly egregious. Such an obvious false front is a red flag to me.

selfhoster11 4y ago

It's not that it's smarter, it's that it's faster. If I have to choose between a larger quantity of code or higher quality of code while keeping the time constant, then I'd prefer the former.