3 ms·

Don't think of it as "less optimal", but rather other next tokens that have smaller, but still high, probabilties of being selected. If your largest next token

Don't think of it as "less optimal", but rather other next tokens that have smaller, but still high, probabilties of being selected. If your largest next token has a probability of (arbitrarily) 25% of being selected, this doesn't make it optimal - just highest probable answer. If the second most probable has a 24% chance, that would now account for 49% of probable responses. Rather than picking the highest probable word, let's generate a random number and whatever that value hits is then compared against the various probability bands (random < probability). Hallucinations can happen, this is where a lot of current work is studying ways to minimize the LLM from picking really bad moves. Using things like critics and chain of thought and meant to help keep the probability machine within the realm of reasonable answers. This is also why jailbreaking techniques like spamming umlauts (ü) has worked. They create very rare tokens where probabilties on what comes next is limited. Once every next token gets equal probability, the LLMs goal is to just try and improve its probability and will output anything, including things against its directive, to get back to 'normal'.

skizm 2y ago
> Don't think of it as "less optimal" For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?
- tsumnia 2y ago
  The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.

skizm 2y ago

> Don't think of it as "less optimal" For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?

tsumnia 2y ago
The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.