3 ms·
> Don't think of it as "less optimal" For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume
by skizm 2y ago
> Don't think of it as "less optimal"
For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?
- tsumnia 2y agoThe issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.