The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal onl

The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.