3 ms·

LLMs don't think, and LLMs don't have strategies. Maybe it could be argued that LLMs have "derived meaning", but all LLMs do is predict the next token. Even RL

LLMs don't think, and LLMs don't have strategies. Maybe it could be argued that LLMs have "derived meaning", but all LLMs do is predict the next token. Even RL just tweaks the next-token prediction process, but the math that drives an LLM makes it impossible for there to be anything that could reasonably be called thought.

yawnxyz 2y ago
rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. Those types of systems are inherently interpretable because you can just look at how they work. They're not black boxes. LLMs are black boxes, and if anything, interpretability systems show us what the heck is going on inside them. Especially useful when half the world is using these already, and we have no idea how t hey work
- kazinator 2y ago
  Water doesn't think, yet if you inject it into the entrance of a maze, it will soon come gushing out of the exit.
- LoganDark 2y ago
  > rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. That doesn't mean the water itself has strategies, just that you can use water in an implementation of strategy... it's fairly well known at this point that LLMs can be used as part of strategies (see e.g. "agents"), they just don't intrinsically have any.
ajkdhcb2 2y ago
True. People use completely unjustified anthropomorphised terminology for marketing reasons and it bothers me a lot. I think it actually holds back understanding how it works. "Hallucinate" is the worst - it's an error and undesired result, not a person having a psychotic episode
kazinator 2y ago
A chess program from 1968 has "strategy", so why deny that to an LLM. LLMs are built on neural networks which are encoding a kind of strategy function through their training. The strategy in an LLM isn't necessarily that it "thinks" about the specific problem described in your prompt and develops a strategy tailored to that problem, but rather its statistical strategy for cobbing together the tokens of the answer. From that, it can seem as if it's making a strategy to a problem also. Certainly, the rhetoric that LLMs put out can at times seem very convincing of that. You can't be sure whether that's not just something cribbed out of the terabytes of text, in which discussions of something very similar to your problem have occurred.
- dev_throwaway 2y ago
  This is not a bad way of looking at it, if I may add a bit, the llm is a solid state system. The only thing that survives from one iteration to the next is the singular highest ranking token, the entire state and "thought process" of the network cannot be represented by a single token, which means that every strategy is encoded in it during training, as a lossy representation of the training data. By definition that is a database, not a thinking system, as the strategy is stored, not actively generated during usage. The anthropomorphization of llms bother me, we don't need to pretend they are alive and thinking, at best that is marketing, at worst, by training the models to output human sounding conversations we are actively taking away the true potential these models could achieve by being ok with them being "simply a tool". But pretending that they are intelligent is what brings in the investors, so that is what we are doing. This paper is just furthering that agenda.
  - kazinator 2y ago
    People anthropomorphize LLMs because that's the most succinct language for describing what they seem to be doing. To avoid anthropomorphizing, you will have to use more formal language which would obfuscate the concepts. Anthropo language has been woven into AI from the early beginnings. AI programs were said to have goals, and to plan and hypothesize. They were given names like "Conniver". The word "expert system" anthropomorphizes! It's literally saying that some piece of logic programming loaded with a base of rules and facts about medical diagnosis is a medical expert.

yawnxyz 2y ago

rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. Those types of systems are inherently interpretable because you can just look at how they work. They're not black boxes. LLMs are black boxes, and if anything, interpretability systems show us what the heck is going on inside them. Especially useful when half the world is using these already, and we have no idea how t hey work

kazinator 2y ago
Water doesn't think, yet if you inject it into the entrance of a maze, it will soon come gushing out of the exit.
LoganDark 2y ago
> rivers don't think and water doesn't have strategies, yet you can build intricate logic-gated tools using the power of water. That doesn't mean the water itself has strategies, just that you can use water in an implementation of strategy... it's fairly well known at this point that LLMs can be used as part of strategies (see e.g. "agents"), they just don't intrinsically have any.

ajkdhcb2 2y ago

True. People use completely unjustified anthropomorphised terminology for marketing reasons and it bothers me a lot. I think it actually holds back understanding how it works. "Hallucinate" is the worst - it's an error and undesired result, not a person having a psychotic episode

kazinator 2y ago

A chess program from 1968 has "strategy", so why deny that to an LLM. LLMs are built on neural networks which are encoding a kind of strategy function through their training. The strategy in an LLM isn't necessarily that it "thinks" about the specific problem described in your prompt and develops a strategy tailored to that problem, but rather its statistical strategy for cobbing together the tokens of the answer. From that, it can seem as if it's making a strategy to a problem also. Certainly, the rhetoric that LLMs put out can at times seem very convincing of that. You can't be sure whether that's not just something cribbed out of the terabytes of text, in which discussions of something very similar to your problem have occurred.

dev_throwaway 2y ago
This is not a bad way of looking at it, if I may add a bit, the llm is a solid state system. The only thing that survives from one iteration to the next is the singular highest ranking token, the entire state and "thought process" of the network cannot be represented by a single token, which means that every strategy is encoded in it during training, as a lossy representation of the training data. By definition that is a database, not a thinking system, as the strategy is stored, not actively generated during usage. The anthropomorphization of llms bother me, we don't need to pretend they are alive and thinking, at best that is marketing, at worst, by training the models to output human sounding conversations we are actively taking away the true potential these models could achieve by being ok with them being "simply a tool". But pretending that they are intelligent is what brings in the investors, so that is what we are doing. This paper is just furthering that agenda.
- kazinator 2y ago
  People anthropomorphize LLMs because that's the most succinct language for describing what they seem to be doing. To avoid anthropomorphizing, you will have to use more formal language which would obfuscate the concepts. Anthropo language has been woven into AI from the early beginnings. AI programs were said to have goals, and to plan and hypothesize. They were given names like "Conniver". The word "expert system" anthropomorphizes! It's literally saying that some piece of logic programming loaded with a base of rules and facts about medical diagnosis is a medical expert.