4 ms·

But the LLM doesn't "think" - it just spits out a sequence of tokens (forever, literally). The chat models have been fine-tuned to eventually emit an "end" toke

by sottol 2y ago

But the LLM doesn't "think" - it just spits out a sequence of tokens (forever, literally). The chat models have been fine-tuned to eventually emit an "end" token (when the training examples would have been answered) that the UI/backend interprets and cuts off the LLM to stop it from generating more output. But the model would gladly go on forever and devolve into more and more obscure output the further the original query moves out of the context window.

kelseyfrog 2y ago
Sorry, but thinking is a social category. Debating it is pointless because it assumes the existence of social facts. Once you're aware of the existence of the meta category of social facts as reified ideas, all of them become unjustifiable. The line of argumentation is already dead.
- BoxedEmpathy 2y ago
  If I understand currently, you're pointing out 'thinking' wasn't meant literally in your comment? Or did I miss?
  - kelseyfrog 2y ago
    I'm saying there isn't a distinction between literally and figuratively thinking or anything of the sort. "Thinking" isn't scientifically grounded, it's socially grounded, and barely so, given that it seems everyone has a differing justification for what constitutes "thinking". The variability in construing "thinking" is a product of how we each fashion our own internal sense of what it means, which if anything is evidence for the anti-realism of "thinking". Continuing, we largely fool ourselves when we play the shell game of hunting for scientific justification of "social facts". Ie: when we assume a social fact, then hunt for scientific evidence to ground it in then use the scientific evidence to justify the existence and validity of the social fact we assumed. This happens a LOT and it's epistemically invalid. A LOT of folks dismissal of thinking machines reminds me of Feuerbach's The Essence of Christianity, specifically the section on anthropomorphism[1], but in an inverted form - the reason LLMs can't "think" is that it would dilute what we hold dear about ourselves: our ability to think. It's an ego protection mechanism, but no one's jumping to admit that. https://www.gutenberg.org/files/47025/47025-h/47025-h.htm#pb17 https://www.gutenberg.org/files/47025/47025-h/47025-h.htm#pb...
    - BoxedEmpathy 2y ago
      Thank you very much for the elaboration! I think I follow better. Thinking doesn't have a rigorous and testable definition, so it isn't scientific. We all have our own colloquial understand. Since it's not testable and the definition varies person to person, it's not useful when reasoning about AI or intelligence. Also I wanted to say I love how you write!
      - kelseyfrog 2y ago
        Even if it had a rigorous and testable definition it would still be suspect by virtue of us having searched for evidence of it under the guise of science. To explain it by analogy, we had the concept of women and men for at least thousands of years. Socially construed by physical appearance and behavior. Then in 1905 Nettie Stevens discovered sex chromosomes. Her discovery "found" the existence of men in women in the epistemological universe of science. But then a curious thing happened, people began placing people in social categories based on sex chromosomes. It's an incestuous loop of logic that hoists itself into validity simply because folks forget how it occurred. If we went searching for a scientifically rigorous definition of "thinking" we would forget we made the same fatal leap of logic.
        BoxedEmpathy 2y ago
        Interesting way to look at it. I suppose I have trouble seeing it that way because I view words as pointers to meaning. The word itself holds no intrinsic value; it's the meaning humans attach to it and the context in which it's used that gives it significance. When we give "thinking" a scientifically testable definition we can reason around it in a more academic way. I've read psych papers that defined "thinking" as cognition, measured by action potentials. A participant was said to be "thinking more" when their EEG showed increased activity. It's important to understand that "thinking" here was only defined within the context of the paper. To use an analogy, many people say their computer is thinking when it's undergoing a heavy processor load. Strictly speaking this is not the case, their computer cannot think, but they aren't wrong or incorrect because in that context 'thinking' literally means 'processing'.
- sottol 2y ago
  That seems a little reductionist. We're just exchanging ideas and trying to interpret imprecise terms in-context (ideally in good faith). We're not really discussing whether LLMs are "thinking" in the sense of the "social category" in the abstract sense - that's just the one word that this discussion got hung up on. Concretely, we're discussing whether giving LLMs a larger time-budget would yield better results - OP suggested that letting LLMs iteratively refine or enrich an answer might lead to better results. The term "thinking" was used as an analogy or quick crutch to convey an analogy, not 100% equivalence, between a hypothetic LLM output evaluation algo and people's supposed "fast" and "slow" modes of thinking (if they even exist, whatever). Even if people don't have these two modes of thought, as you said they might be post-fact reifications of sociological ideas, the algorithm/idea of iteratively reflecting on and self-refining LLM outputs might still apply to LLMs as LLMs are not actually thinking. Imo current LLM architecture and/or training data facilitate (mostly?) the automatic non-iterative mode but OpenAI is trying to do more of the iterative, reflective mode with O1 afaict. Imo and currently, there's not much to be gained by iteratively feeding an a current-generation LLM it's own output to iteratively analyze and augment it. At least not in general. Even enriching the input data via RAG doesn't always lead to better results. Imo, there's more fundamental work necessary - and maybe OpenAI is doing that.
bob1029 2y ago
Anthropomorphization of the technology is my new favorite canary for when something might be going wrong.

by sottol 2y ago

kelseyfrog 2y ago

Sorry, but thinking is a social category. Debating it is pointless because it assumes the existence of social facts. Once you're aware of the existence of the meta category of social facts as reified ideas, all of them become unjustifiable. The line of argumentation is already dead.

BoxedEmpathy 2y ago
If I understand currently, you're pointing out 'thinking' wasn't meant literally in your comment? Or did I miss?
- kelseyfrog 2y ago
  I'm saying there isn't a distinction between literally and figuratively thinking or anything of the sort. "Thinking" isn't scientifically grounded, it's socially grounded, and barely so, given that it seems everyone has a differing justification for what constitutes "thinking". The variability in construing "thinking" is a product of how we each fashion our own internal sense of what it means, which if anything is evidence for the anti-realism of "thinking". Continuing, we largely fool ourselves when we play the shell game of hunting for scientific justification of "social facts". Ie: when we assume a social fact, then hunt for scientific evidence to ground it in then use the scientific evidence to justify the existence and validity of the social fact we assumed. This happens a LOT and it's epistemically invalid. A LOT of folks dismissal of thinking machines reminds me of Feuerbach's The Essence of Christianity, specifically the section on anthropomorphism[1], but in an inverted form - the reason LLMs can't "think" is that it would dilute what we hold dear about ourselves: our ability to think. It's an ego protection mechanism, but no one's jumping to admit that. https://www.gutenberg.org/files/47025/47025-h/47025-h.htm#pb17 https://www.gutenberg.org/files/47025/47025-h/47025-h.htm#pb...
  - BoxedEmpathy 2y ago
    Thank you very much for the elaboration! I think I follow better. Thinking doesn't have a rigorous and testable definition, so it isn't scientific. We all have our own colloquial understand. Since it's not testable and the definition varies person to person, it's not useful when reasoning about AI or intelligence. Also I wanted to say I love how you write!
    - kelseyfrog 2y ago
      Even if it had a rigorous and testable definition it would still be suspect by virtue of us having searched for evidence of it under the guise of science. To explain it by analogy, we had the concept of women and men for at least thousands of years. Socially construed by physical appearance and behavior. Then in 1905 Nettie Stevens discovered sex chromosomes. Her discovery "found" the existence of men in women in the epistemological universe of science. But then a curious thing happened, people began placing people in social categories based on sex chromosomes. It's an incestuous loop of logic that hoists itself into validity simply because folks forget how it occurred. If we went searching for a scientifically rigorous definition of "thinking" we would forget we made the same fatal leap of logic.
      - BoxedEmpathy 2y ago
        Interesting way to look at it. I suppose I have trouble seeing it that way because I view words as pointers to meaning. The word itself holds no intrinsic value; it's the meaning humans attach to it and the context in which it's used that gives it significance. When we give "thinking" a scientifically testable definition we can reason around it in a more academic way. I've read psych papers that defined "thinking" as cognition, measured by action potentials. A participant was said to be "thinking more" when their EEG showed increased activity. It's important to understand that "thinking" here was only defined within the context of the paper. To use an analogy, many people say their computer is thinking when it's undergoing a heavy processor load. Strictly speaking this is not the case, their computer cannot think, but they aren't wrong or incorrect because in that context 'thinking' literally means 'processing'.
sottol 2y ago
That seems a little reductionist. We're just exchanging ideas and trying to interpret imprecise terms in-context (ideally in good faith). We're not really discussing whether LLMs are "thinking" in the sense of the "social category" in the abstract sense - that's just the one word that this discussion got hung up on. Concretely, we're discussing whether giving LLMs a larger time-budget would yield better results - OP suggested that letting LLMs iteratively refine or enrich an answer might lead to better results. The term "thinking" was used as an analogy or quick crutch to convey an analogy, not 100% equivalence, between a hypothetic LLM output evaluation algo and people's supposed "fast" and "slow" modes of thinking (if they even exist, whatever). Even if people don't have these two modes of thought, as you said they might be post-fact reifications of sociological ideas, the algorithm/idea of iteratively reflecting on and self-refining LLM outputs might still apply to LLMs as LLMs are not actually thinking. Imo current LLM architecture and/or training data facilitate (mostly?) the automatic non-iterative mode but OpenAI is trying to do more of the iterative, reflective mode with O1 afaict. Imo and currently, there's not much to be gained by iteratively feeding an a current-generation LLM it's own output to iteratively analyze and augment it. At least not in general. Even enriching the input data via RAG doesn't always lead to better results. Imo, there's more fundamental work necessary - and maybe OpenAI is doing that.

bob1029 2y ago

Anthropomorphization of the technology is my new favorite canary for when something might be going wrong.