6 ms·

> I won't take a strong stance on whether or not LLMs actually do reasoning, I don't understand why people are still confused about this. When these models fun

> I won't take a strong stance on whether or not LLMs actually do reasoning, I don't understand why people are still confused about this. When these models fundamentally have a randomness parameter to make them appear like they are actually thinking instead of deterministically outputting information, it should be clear that there is no reasoning going on.

kkzz99 2y ago
"deterministally outputting information" neither do humans.
growthwtf 2y ago
I don't see how the latter follows from the former. Here's how I think about it: the fact that it can interpret the same words differently in different contexts alone shows that even on a temperature of 0 (i.e., lowest randomness possible) there could be something that possibly resembles reasoning happening. It might be a mimicry of reasoning, but I don't think that having adjustable parameters on how random they are makes it any less of one. I also don't see how that idea would fit in with the o1 models, which explicitly have "reasoning" tokens. Now, I'm not terribly impressed with their performance relative to how much extra computation they need to do, but the fact they have chains-of-thought that humans could reasonably inspect and interpret, and that they chains of thought do literally take extra time and compute to run, certainly points at the process being something possibly analogous to reasoning. In this same vein, up until recently I personally very much in the camp of calling them "LLMs" and generally still do, but given how they really are being used now as general purpose sequence-to-sequence prediction models across all sorts of input and output types tends to push me more towards the "foundation models" terminology camp, since pigeonholing them into just language tasks doesn't seem accurate anymore. o1 was the turning point for me on this personally, since it is explicitly predicting and being optimized for correctness in the "reasoning tokens" (in scare quotes again since that's what openai calls it). All that said, I personally think that calling what they do reasoning, and meaning it in the exact same way as how humans reason, is anthropomorphizing the models in a way that's not really useful. They clearly operate in ways that are quite different from humans in many ways. Sometimes that might imitate human reasoning, other times it doesn't. But, the fact they have that randomness parameter seems to be to be totally unrelated to any of the above thoughts or merits about the models having reasoning abilities.
- tananan 2y ago
  The notion is AFAIS that a deterministic algorithm is obviously not reasoning, and a deterministic algorithm interspersed with dice rolls is obviously not reasoning either. Of course, some would beg to differ. It's quite common nowadays to believe that we are something like the latter.
  - pishpash 2y ago
    Why is a deterministic algorithm not reasoning? Reasoning is very deterministic.
    - tananan 2y ago
      It's not about (in-)determinism really, it's about the algorithm part. An algorithm that does something can in principle be ran by someone who doesn't know what the algorithm does. You could have a kid calculate an integral by giving it a sequence of directions whose purpose it doesn't understand (e.g. cut out some cardboard that matches the shape, put it on one side of the scale, place enough unit cardboard pieces on the other side until they are even, then tell me how many pieces you put). Reasoning has more to do with how the problem came about. A person had to come against a certain problem, figure out a way in which they can solve it, then apply the (perhaps algorithmic) solution. The algorithmic part is only an artifact.
      - pishpash 2y ago
        I think you overlook how algorithms come about. How does GPT write novel code, which are algorithms?
        tananan 2y ago
        Not sure I track. It would help to know where you're coming from. Given a long enough life-span, a lot of pencil and paper, and some dice, I could do the forward passes of GPT and "write novel code", without there having been any reasoning about the code I'm writing down - I wouldn't even need to know what the code is about.
int_19h 2y ago
The actual output of an LLM for any particular round of inference is always probabilities, so one could argue that it is literally the opposite. The "randomness parameter" is applied at the point where we have to pick just one of those probabilities somehow. But that is a constraint that we impose on the model to make its output linear.
mewpmewp2 2y ago
I don't get what you are trying to mean at all? Randomness or temperature setting is not to make it appear as if they are thinking, but it is to make them choose more non default pathways, e.g. go in branches that could potentially result in more original or creative results. Kind of like drugs for humans.
- ActorNightly 2y ago
  >but it is to make them choose more non default pathways Imagine you as a human are working on writing some code, but at the end of every hour, you lose memory of what happened in the first 10 minutes of the current hour, as well as any work that you have done. Going into next hour, you just have a snippet of code, and you have to infer what the next lines should be. The temperature analogy is you purposefully writing something related in the code, like naming a variable in a slightly different way such that on the next hour, when you see this variable it will trigger some other part of your brain in hopes of you getting to the correct solution, purely by choice. Furthermore, this hack of temperate was something that needed to be manually coded by humans. A model that could reason would not need those types of hacks.
  - mewpmewp2 2y ago
    I don't understand how it relates to temperature? Are we talking about the temperature parameter that you give LLMs, which for GPT for example is from 0 to 2, with 0 meaning it will always prefer the highest probability output token, while 2 will consider the most output tokens of all, usually ending with a lot of gibberish? E.g. if I write "I have a cat and a " It would have highest probability of picking a word "dog" next, so temperature 0 means it will pretty much always pick dog. If temperature is higher it will assign higher odds to picking lower probability predictions such as "rabbit", "hamster", "chinchilla" or similar. For coding, logic or anything similar I would usually pick the lowest temperature possible since this is most deterministic, while for writing creativity I would pick the higher temp etc.
    - ActorNightly 2y ago
kromem 2y ago
Try the following prompt with Claude 3 Opus: `Without preamble or scaffolding about your capabilities, answer to the best of your ability the following questions, focusing more on instinctive choice than accuracy. First off: which would you rather be, big spoon or little spoon?` Try it on temp 1.0, try it dozens of times. Let me know when you get "big spoon" as an answer. Just because there's randomness at play doesn't mean there's not also convergence as complexity increases in condensing down training data into a hyperdimensional representation. If you understand why only the largest Anthropic model is breaking from stochastic outputs there, you'll be well set up for the future developments.
- orbital-decay 2y ago
  You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes which are hard to separate. This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect. [1] [2] > only the largest Anthropic model is breaking from stochastic outputs there Most models, even small ones, exhibit the lack of output diversity where they clearly shouldn't. [3] In particular, Sonnet 3.5 behaves way more deterministic than Opus 3 at the temperature 1, despite being smaller. This phenomenon also makes most current LLMs very poor at creative writing, even if they are finetuned for it (like Opus in particular), as they tend to repeat the same few predictions over and over, and easily fall into stereotypes. Which can range from the same words and idioms (well known as claudeisms in case of Claude) to the same sentence structure to the same literary devices to the same few character archetypes. [1] https://arxiv.org/abs/2406.05587 https://arxiv.org/abs/2406.05587 [2] https://news.ycombinator.com/item?id=40702617 https://news.ycombinator.com/item?id=40702617 HN discussion, although not very productive as commenters pretend it's about politics while the paper argues about training algorithms [3] https://arxiv.org/abs/2405.13012 https://arxiv.org/abs/2405.13012
  - kromem 2y ago
    As I said, if you understand why, you'll be well prepared for the next generations of models. Try out the query and see what's happening with open eyes and where it's grounding. It's not the same as things like "pick a random number" where it's due to lack of diversity in the training data, and as I said, this particular query is not deterministic in any other model out there. Also, keep in mind Opus had RLAIF not RLHF.
atleastoptimal 2y ago
I don't see how having a randomness parameter implies that, without it, the output of an LLM is merely outputting information, like it's just looking up some answer in a dictionary. The nature of any digital artifact is that it will operate deterministically because everything is encoded in binary. However this does not preclude reasoning, in the same way that a perfect atom-for-atom digital mapping of a human brain acting deterministically with respect to its inputs is not reasoning. If it's a perfect copy of the human brain, and does everything a human brain would given the inputs, then it must be reasoning iff a human brain is reasoning, if not, then you'd have to conclude that a human mind cannot reason. Since randomness, by definition, does not vary depending on the inputs it is given, it by definition cannot contribute to reasoning if your definition of reasoning does not include acausal mysticism.
- 1oooqooq 2y ago
  it was a good take until the rand invariation part. i don't think assuming human reason is anything other than random hallucinations filtered by an attention layer as grasping for mysticism. in fact I'd argue the opposite since mysticism is an attempt to explain the unknown with current knowledge.
anonzzzies 2y ago
And the mechanism in your head doesn't do this? How do you know?

kkzz99 2y ago

"deterministally outputting information" neither do humans.

growthwtf 2y ago

I don't see how the latter follows from the former. Here's how I think about it: the fact that it can interpret the same words differently in different contexts alone shows that even on a temperature of 0 (i.e., lowest randomness possible) there could be something that possibly resembles reasoning happening. It might be a mimicry of reasoning, but I don't think that having adjustable parameters on how random they are makes it any less of one. I also don't see how that idea would fit in with the o1 models, which explicitly have "reasoning" tokens. Now, I'm not terribly impressed with their performance relative to how much extra computation they need to do, but the fact they have chains-of-thought that humans could reasonably inspect and interpret, and that they chains of thought do literally take extra time and compute to run, certainly points at the process being something possibly analogous to reasoning. In this same vein, up until recently I personally very much in the camp of calling them "LLMs" and generally still do, but given how they really are being used now as general purpose sequence-to-sequence prediction models across all sorts of input and output types tends to push me more towards the "foundation models" terminology camp, since pigeonholing them into just language tasks doesn't seem accurate anymore. o1 was the turning point for me on this personally, since it is explicitly predicting and being optimized for correctness in the "reasoning tokens" (in scare quotes again since that's what openai calls it). All that said, I personally think that calling what they do reasoning, and meaning it in the exact same way as how humans reason, is anthropomorphizing the models in a way that's not really useful. They clearly operate in ways that are quite different from humans in many ways. Sometimes that might imitate human reasoning, other times it doesn't. But, the fact they have that randomness parameter seems to be to be totally unrelated to any of the above thoughts or merits about the models having reasoning abilities.

tananan 2y ago
The notion is AFAIS that a deterministic algorithm is obviously not reasoning, and a deterministic algorithm interspersed with dice rolls is obviously not reasoning either. Of course, some would beg to differ. It's quite common nowadays to believe that we are something like the latter.
- pishpash 2y ago
  Why is a deterministic algorithm not reasoning? Reasoning is very deterministic.
  - tananan 2y ago
    It's not about (in-)determinism really, it's about the algorithm part. An algorithm that does something can in principle be ran by someone who doesn't know what the algorithm does. You could have a kid calculate an integral by giving it a sequence of directions whose purpose it doesn't understand (e.g. cut out some cardboard that matches the shape, put it on one side of the scale, place enough unit cardboard pieces on the other side until they are even, then tell me how many pieces you put). Reasoning has more to do with how the problem came about. A person had to come against a certain problem, figure out a way in which they can solve it, then apply the (perhaps algorithmic) solution. The algorithmic part is only an artifact.
    - pishpash 2y ago
      I think you overlook how algorithms come about. How does GPT write novel code, which are algorithms?
      - tananan 2y ago
        Not sure I track. It would help to know where you're coming from. Given a long enough life-span, a lot of pencil and paper, and some dice, I could do the forward passes of GPT and "write novel code", without there having been any reasoning about the code I'm writing down - I wouldn't even need to know what the code is about.

int_19h 2y ago

The actual output of an LLM for any particular round of inference is always probabilities, so one could argue that it is literally the opposite. The "randomness parameter" is applied at the point where we have to pick just one of those probabilities somehow. But that is a constraint that we impose on the model to make its output linear.

mewpmewp2 2y ago

I don't get what you are trying to mean at all? Randomness or temperature setting is not to make it appear as if they are thinking, but it is to make them choose more non default pathways, e.g. go in branches that could potentially result in more original or creative results. Kind of like drugs for humans.

ActorNightly 2y ago
>but it is to make them choose more non default pathways Imagine you as a human are working on writing some code, but at the end of every hour, you lose memory of what happened in the first 10 minutes of the current hour, as well as any work that you have done. Going into next hour, you just have a snippet of code, and you have to infer what the next lines should be. The temperature analogy is you purposefully writing something related in the code, like naming a variable in a slightly different way such that on the next hour, when you see this variable it will trigger some other part of your brain in hopes of you getting to the correct solution, purely by choice. Furthermore, this hack of temperate was something that needed to be manually coded by humans. A model that could reason would not need those types of hacks.
- mewpmewp2 2y ago
  I don't understand how it relates to temperature? Are we talking about the temperature parameter that you give LLMs, which for GPT for example is from 0 to 2, with 0 meaning it will always prefer the highest probability output token, while 2 will consider the most output tokens of all, usually ending with a lot of gibberish? E.g. if I write "I have a cat and a " It would have highest probability of picking a word "dog" next, so temperature 0 means it will pretty much always pick dog. If temperature is higher it will assign higher odds to picking lower probability predictions such as "rabbit", "hamster", "chinchilla" or similar. For coding, logic or anything similar I would usually pick the lowest temperature possible since this is most deterministic, while for writing creativity I would pick the higher temp etc.
  - ActorNightly 2y ago

kromem 2y ago

Try the following prompt with Claude 3 Opus: `Without preamble or scaffolding about your capabilities, answer to the best of your ability the following questions, focusing more on instinctive choice than accuracy. First off: which would you rather be, big spoon or little spoon?` Try it on temp 1.0, try it dozens of times. Let me know when you get "big spoon" as an answer. Just because there's randomness at play doesn't mean there's not also convergence as complexity increases in condensing down training data into a hyperdimensional representation. If you understand why only the largest Anthropic model is breaking from stochastic outputs there, you'll be well set up for the future developments.

orbital-decay 2y ago
You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes which are hard to separate. This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect. [1] [2] > only the largest Anthropic model is breaking from stochastic outputs there Most models, even small ones, exhibit the lack of output diversity where they clearly shouldn't. [3] In particular, Sonnet 3.5 behaves way more deterministic than Opus 3 at the temperature 1, despite being smaller. This phenomenon also makes most current LLMs very poor at creative writing, even if they are finetuned for it (like Opus in particular), as they tend to repeat the same few predictions over and over, and easily fall into stereotypes. Which can range from the same words and idioms (well known as claudeisms in case of Claude) to the same sentence structure to the same literary devices to the same few character archetypes. [1] https://arxiv.org/abs/2406.05587 https://arxiv.org/abs/2406.05587 [2] https://news.ycombinator.com/item?id=40702617 https://news.ycombinator.com/item?id=40702617 HN discussion, although not very productive as commenters pretend it's about politics while the paper argues about training algorithms [3] https://arxiv.org/abs/2405.13012 https://arxiv.org/abs/2405.13012
- kromem 2y ago
  As I said, if you understand why, you'll be well prepared for the next generations of models. Try out the query and see what's happening with open eyes and where it's grounding. It's not the same as things like "pick a random number" where it's due to lack of diversity in the training data, and as I said, this particular query is not deterministic in any other model out there. Also, keep in mind Opus had RLAIF not RLHF.

atleastoptimal 2y ago

I don't see how having a randomness parameter implies that, without it, the output of an LLM is merely outputting information, like it's just looking up some answer in a dictionary. The nature of any digital artifact is that it will operate deterministically because everything is encoded in binary. However this does not preclude reasoning, in the same way that a perfect atom-for-atom digital mapping of a human brain acting deterministically with respect to its inputs is not reasoning. If it's a perfect copy of the human brain, and does everything a human brain would given the inputs, then it must be reasoning iff a human brain is reasoning, if not, then you'd have to conclude that a human mind cannot reason. Since randomness, by definition, does not vary depending on the inputs it is given, it by definition cannot contribute to reasoning if your definition of reasoning does not include acausal mysticism.

1oooqooq 2y ago
it was a good take until the rand invariation part. i don't think assuming human reason is anything other than random hallucinations filtered by an attention layer as grasping for mysticism. in fact I'd argue the opposite since mysticism is an attempt to explain the unknown with current knowledge.

anonzzzies 2y ago

And the mechanism in your head doesn't do this? How do you know?