4 ms·

> it doesn't know if it's supposed to guess a probable word from a Wikipedia article, an Onion article, a Project Gutenberg manuscript, or an Archive Of Our Own

> it doesn't know if it's supposed to guess a probable word from a Wikipedia article, an Onion article, a Project Gutenberg manuscript, or an Archive Of Our Own fanfic. So you get a bit of all that. This is true of base LLM models that are just trained on missing-word prediction on the training corpus, but one of the main points of RLHF[1] is to tune this model to make these kind of inferences the way a human would expect. For example if you asked an untuned model to write a poem in the style of ... etc., a valid internet response might be "hmm no thanks, you go first", you need to steer the model away from replying like this. I'm not saying it's perfect, but it's wrong to say e.g. GPT-4 has had no information about the difference between a good and bad response and is just generating internet-like text at random, the big players have made progress on this already. [1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...

jameshart 3y ago
Right. Reinforcement learning trains them that question and answer sessions contain answers which statistically correlate with factual statements in their broader learning corpus. When formulating answers, this leads them to formulate answers that reflect the factual information on which they were trained. My point is that the source data contains a far muddier range of information than just unarguable facts. We largely want LLM based Q&A bots to answer questions about fictional or mythical characters in their own terms. As I said, those questions above all have reasonably ‘correct’ answers. The fact that from all that LLMs do as well as they do is remarkable. But it also seems like it requires us to assume that LLMs are capable of a remarkable degree of cultural nuance, media literacy and contextual awareness for them to figure out the different authorship, salience, trustworthiness, agenda, biases, and assumptions of all the gigareams of text they’ve ingested.

jameshart 3y ago

Right. Reinforcement learning trains them that question and answer sessions contain answers which statistically correlate with factual statements in their broader learning corpus. When formulating answers, this leads them to formulate answers that reflect the factual information on which they were trained. My point is that the source data contains a far muddier range of information than just unarguable facts. We largely want LLM based Q&A bots to answer questions about fictional or mythical characters in their own terms. As I said, those questions above all have reasonably ‘correct’ answers. The fact that from all that LLMs do as well as they do is remarkable. But it also seems like it requires us to assume that LLMs are capable of a remarkable degree of cultural nuance, media literacy and contextual awareness for them to figure out the different authorship, salience, trustworthiness, agenda, biases, and assumptions of all the gigareams of text they’ve ingested.