3 ms·

General LLM question: a lot of people ask specific knowledge based questions to LLMs. Isn't one of the defining features of modern NPL (and therefore LLMs) that

by skizm 2y ago

General LLM question: a lot of people ask specific knowledge based questions to LLMs. Isn't one of the defining features of modern NPL (and therefore LLMs) that it is designed to be non-deterministic? Meaning that it will intentionally select "less optimal" next tokens at some random rate in order to make it sound less like a robot answering the same thing over and over. If this is the case, isn't it pretty much guaranteed to get knowledge based questions wrong a decent amount of the time? I could be way off base (I have zero knowledge about the internals and simply read occasional blog posts), but I thought I remembered that being one of the key features to making LLMs sound more human.

marcosdumay 2y ago
> Isn't one of the defining features of modern NPL (and therefore LLMs) that it is designed to be non-deterministic? Not of all NPL, but it is absolutely a property of LLMs. But I doubt this even enters into the total figure if you go search for the causes of LLMs giving wrong answers.
tsumnia 2y ago
Don't think of it as "less optimal", but rather other next tokens that have smaller, but still high, probabilties of being selected. If your largest next token has a probability of (arbitrarily) 25% of being selected, this doesn't make it optimal - just highest probable answer. If the second most probable has a 24% chance, that would now account for 49% of probable responses. Rather than picking the highest probable word, let's generate a random number and whatever that value hits is then compared against the various probability bands (random < probability). Hallucinations can happen, this is where a lot of current work is studying ways to minimize the LLM from picking really bad moves. Using things like critics and chain of thought and meant to help keep the probability machine within the realm of reasonable answers. This is also why jailbreaking techniques like spamming umlauts (ü) has worked. They create very rare tokens where probabilties on what comes next is limited. Once every next token gets equal probability, the LLMs goal is to just try and improve its probability and will output anything, including things against its directive, to get back to 'normal'.
- skizm 2y ago
  > Don't think of it as "less optimal" For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?
  - tsumnia 2y ago
    The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.
hn_throwaway_99 2y ago
What you are referring to is called "temperature" with respect to LLMs, and it is a parameter that can be tweaked at inference time. Google's AI Overview gives a pretty good summary IMO: > The temperature parameter in a large language model (LLM) controls the amount of randomness in the model's output. It's a numerical value that influences the model's choices, and is often set between 0 and 1. > The temperature setting affects the quality of the output by balancing exploration and exploitation: > Low temperature > The model is more likely to exploit patterns it's already learned, resulting in more deterministic and conservative outputs. This is ideal for tasks that require accuracy, such as data analysis, technical writing, and summarizing research papers. > High temperature > The model is encouraged to explore, resulting in more varied and creative outputs. This is ideal for creative tasks, such as brainstorming, artistic content, and storytelling. However, adjusting the temperature too high can result in non-sensical outputs.

by skizm 2y ago

marcosdumay 2y ago

> Isn't one of the defining features of modern NPL (and therefore LLMs) that it is designed to be non-deterministic? Not of all NPL, but it is absolutely a property of LLMs. But I doubt this even enters into the total figure if you go search for the causes of LLMs giving wrong answers.

tsumnia 2y ago

Don't think of it as "less optimal", but rather other next tokens that have smaller, but still high, probabilties of being selected. If your largest next token has a probability of (arbitrarily) 25% of being selected, this doesn't make it optimal - just highest probable answer. If the second most probable has a 24% chance, that would now account for 49% of probable responses. Rather than picking the highest probable word, let's generate a random number and whatever that value hits is then compared against the various probability bands (random < probability). Hallucinations can happen, this is where a lot of current work is studying ways to minimize the LLM from picking really bad moves. Using things like critics and chain of thought and meant to help keep the probability machine within the realm of reasonable answers. This is also why jailbreaking techniques like spamming umlauts (ü) has worked. They create very rare tokens where probabilties on what comes next is limited. Once every next token gets equal probability, the LLMs goal is to just try and improve its probability and will output anything, including things against its directive, to get back to 'normal'.

skizm 2y ago
> Don't think of it as "less optimal" For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?
- tsumnia 2y ago
  The issue is that an LLM by itself does not try to be correct or incorrect, only respond with tokens that have a high probability of appearing next. Optimal only becomes a metric for consideration when humans were added to rate the "goodness" of a response. A good explanation of this can be seen in "The True Story of How GPT-2 Became Maximally Lewd" (https://www.youtube.com/watch?v=qV_rOlHjvvs https://www.youtube.com/watch?v=qV_rOlHjvvs). The 'critics' that assisted in training GPT focused on producing coherent sentences that users liked. When the bug occurred, it maintained coherence but became incredibly vulgar - because the "morality critic" kept saying these negative reviews were good. Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.

hn_throwaway_99 2y ago

What you are referring to is called "temperature" with respect to LLMs, and it is a parameter that can be tweaked at inference time. Google's AI Overview gives a pretty good summary IMO: > The temperature parameter in a large language model (LLM) controls the amount of randomness in the model's output. It's a numerical value that influences the model's choices, and is often set between 0 and 1. > The temperature setting affects the quality of the output by balancing exploration and exploitation: > Low temperature > The model is more likely to exploit patterns it's already learned, resulting in more deterministic and conservative outputs. This is ideal for tasks that require accuracy, such as data analysis, technical writing, and summarizing research papers. > High temperature > The model is encouraged to explore, resulting in more varied and creative outputs. This is ideal for creative tasks, such as brainstorming, artistic content, and storytelling. However, adjusting the temperature too high can result in non-sensical outputs.