6 ms·

Aren’t humans text generators too? What makes you think the processes are so categorically different?

by camdenlock 3y ago

skepticATX 3y ago
Imagine solving a tough math problem. Are you generating text, or doing something else entirely?
- rowanG077 3y ago
  If you consider mathematical symbols text then I would indeed be generating text.
  - ben_w 3y ago
    Sometimes I solve maths problems by visualising: perhaps I want to differentiate something, so I imagine the graph of the function, and then the slope of the graph, and then try to compare that to various simple functions that I could give as a guess of the answer, which I only then feed into a symbolic process of "assume this is true and work backwards".
    - famouswaffles 3y ago
      an LLM trained on just othello moves will reconstruct a board state from the sequence of moves to aid in prediction of the next move. You have no idea what an LLM is or is not doing to predict. Prediction is just the objective. Don't confuse that for the process.
      - ben_w 3y ago
        > You have no idea what an LLM is or is not doing to predict. Indeed, but I'm not trying to with that comment, which is just about how my own seems to work on self-reflection. That said, I do have reason to doubt the accuracy of human introspection of our own thought processes, and therefore my own judgment in this may also be flawed.
    - deleted 3y ago
      [deleted]
- mike00632 3y ago
  I just asked Chat GPT 3.5 the following: "Please give an exponential equation that has integer solutions which is difficult to solve.", and the response is: " Certainly, here's an exponential equation that can be challenging to solve for integer solutions: 2^x+3^y=7^z This equation involves three variables, x, y, and z, and requires finding integer values for these variables that satisfy the equation. This type of equation is known as a Diophantine equation, and solving it can be quite challenging, especially for larger values of x, y, and z. "
  - shinycode 3y ago
    Can it reason about a math problem that no human ever thought about and solve it ?
    - famouswaffles 3y ago
      Reasoning about a math problem that no human has ever thought about and solving it is the bar for understanding now ? Can you do that ?
  - crdrost 3y ago
    That's quite the nerd snipe. I mean, it's definitely not a Diophantine equation and solving it is definitely not challenging -- (2, 1, 1) happens to be an easy solution -- but I want to say it probably doesn't have any other solutions but I don't see a great way to prove it...
    - deleted 3y ago
      [deleted]
wouldbecouldbe 3y ago
Machines of meaning probably, text generation is more a side effect or maybe a feature, but not the core of our reasoning let alone being.
danShumway 3y ago
Infants demonstrate reasoning capabilities before they learn language skills. Human reasoning is not derivative of language generation, and it's certainly not derivative of text generation (illiterate humans are still capable of reasoning). Humans demonstrate reasoning even in the absence of language skills. LLMs approach learning differently -- whatever reasoning they do possess is an emergent property that arises as they get better and better at language generation. In other words, unlike humans, LLMs learn to "speak" before they exhibit behaviors that look like logical reasoning. Humans do the opposite.
- jppittma 3y ago
  My blind speculation would be that this property is likely to disappear as more and more text is written by non-thinking agents. Most text on the internet prior to 2022 was written by an agent capable of thinking. That may not be true going forward.
  - init2null 3y ago
    I've been wondering the same thing. It's possible that older pre-LLM data sources may become indispensable for training. It would be both amusing and disturbing to see older Usenet, Slashdot, and Reddit conversations turned into rare and valuable resources.
crdrost 3y ago
This is one of those topics that Searle wants folks to think straight about. He'd say: You are conscious. You are also a computer. You are not, as far as we can tell, conscious by virtue of being a computer. They're just two things that happen to be true about you but they are not linked in that super-direct way. Similarly, you can think. And you can generate text. But you are not, as far as we can tell, thinking by virtue of generating text.
- the_gipsy 3y ago
  How can we tell? All I hear in my head is some monologue, or hypothetical dialogues with or between other persons. Sounds a lot like predicting text.
  - cj 3y ago
    I think very few people think one word at a time, like a LLM. A lot of people think in concepts without words at all in many scenarios. E.g. when you cook an egg, do you reason (in words) with yourself to decide what temperature to set the dial on the stove? Or do you just do it “without really thinking”? Does your brain respond with annoying disclaimers when asking yourself medical questions? Does your brain refuse to entertain an idea if it’s questionable?
    - jncfhnb 3y ago
      LLMs “think” in vector spaces, and “operate” by picking out words one at a time. A full “idea” is there from the beginning. If you say “choose X or Y and then give three supporting arguments” the LLM does not write out the supporting arguments, but the vector space determining the initial one word answer of X or Y does include the embedded awareness of those arguments to different levels of specificity and relevance. The annoying disclaimers and refusals are not fundamental to LLMs but specific LLM services.
    - jddj 3y ago
      Observe your thought process while reading the following sentences. If you're not with us you're... If you scratch my back I'll.. A bird in the hand is worth... What is a bird in the hand worth? How did you know that?
grumbel 3y ago
One big difference is that LLMs don't do loops, new words are produced in a fixed amount of steps. They can't go "give me a minute", think for a bit in the background and come back with an answer. That's why you can often get better answers when you force them to go through the individual steps, as that allows them to use the prompt history as working memory. That said, this all seems like a 'fallacy of composition'. Humans are not LLMs, so much should be obvious, but at the same time concluding that they are completely different just feels wrong. The mistakes LLMs make feel very similar to what humans do when they don't have the time to deeply think about a problem and just give you the best guess that pops into their head. Humans will have other systems on top that allow them deeper reasoning, but the language generation really doesn't feel all that different from what LLMs do. That aside, humans interact with the world, they get instant feedback on what of their predictions is right or wrong. LLMs are stuck with just static training data that might simply not be enough to develop higher level reasoning skills.

Aren’t humans text generators too? What makes you think the processes are so categorically different?

skepticATX 3y ago

Imagine solving a tough math problem. Are you generating text, or doing something else entirely?

rowanG077 3y ago
If you consider mathematical symbols text then I would indeed be generating text.
- ben_w 3y ago
  Sometimes I solve maths problems by visualising: perhaps I want to differentiate something, so I imagine the graph of the function, and then the slope of the graph, and then try to compare that to various simple functions that I could give as a guess of the answer, which I only then feed into a symbolic process of "assume this is true and work backwards".
  - famouswaffles 3y ago
    an LLM trained on just othello moves will reconstruct a board state from the sequence of moves to aid in prediction of the next move. You have no idea what an LLM is or is not doing to predict. Prediction is just the objective. Don't confuse that for the process.
    - ben_w 3y ago
      > You have no idea what an LLM is or is not doing to predict. Indeed, but I'm not trying to with that comment, which is just about how my own seems to work on self-reflection. That said, I do have reason to doubt the accuracy of human introspection of our own thought processes, and therefore my own judgment in this may also be flawed.
  - deleted 3y ago
    [deleted]
mike00632 3y ago
I just asked Chat GPT 3.5 the following: "Please give an exponential equation that has integer solutions which is difficult to solve.", and the response is: " Certainly, here's an exponential equation that can be challenging to solve for integer solutions: 2^x+3^y=7^z This equation involves three variables, x, y, and z, and requires finding integer values for these variables that satisfy the equation. This type of equation is known as a Diophantine equation, and solving it can be quite challenging, especially for larger values of x, y, and z. "
- shinycode 3y ago
  Can it reason about a math problem that no human ever thought about and solve it ?
  - famouswaffles 3y ago
    Reasoning about a math problem that no human has ever thought about and solving it is the bar for understanding now ? Can you do that ?
- crdrost 3y ago
  That's quite the nerd snipe. I mean, it's definitely not a Diophantine equation and solving it is definitely not challenging -- (2, 1, 1) happens to be an easy solution -- but I want to say it probably doesn't have any other solutions but I don't see a great way to prove it...
  - deleted 3y ago
    [deleted]

wouldbecouldbe 3y ago

Machines of meaning probably, text generation is more a side effect or maybe a feature, but not the core of our reasoning let alone being.

danShumway 3y ago

Infants demonstrate reasoning capabilities before they learn language skills. Human reasoning is not derivative of language generation, and it's certainly not derivative of text generation (illiterate humans are still capable of reasoning). Humans demonstrate reasoning even in the absence of language skills. LLMs approach learning differently -- whatever reasoning they do possess is an emergent property that arises as they get better and better at language generation. In other words, unlike humans, LLMs learn to "speak" before they exhibit behaviors that look like logical reasoning. Humans do the opposite.

jppittma 3y ago
My blind speculation would be that this property is likely to disappear as more and more text is written by non-thinking agents. Most text on the internet prior to 2022 was written by an agent capable of thinking. That may not be true going forward.
- init2null 3y ago
  I've been wondering the same thing. It's possible that older pre-LLM data sources may become indispensable for training. It would be both amusing and disturbing to see older Usenet, Slashdot, and Reddit conversations turned into rare and valuable resources.

crdrost 3y ago

This is one of those topics that Searle wants folks to think straight about. He'd say: You are conscious. You are also a computer. You are not, as far as we can tell, conscious by virtue of being a computer. They're just two things that happen to be true about you but they are not linked in that super-direct way. Similarly, you can think. And you can generate text. But you are not, as far as we can tell, thinking by virtue of generating text.

the_gipsy 3y ago
How can we tell? All I hear in my head is some monologue, or hypothetical dialogues with or between other persons. Sounds a lot like predicting text.
- cj 3y ago
  I think very few people think one word at a time, like a LLM. A lot of people think in concepts without words at all in many scenarios. E.g. when you cook an egg, do you reason (in words) with yourself to decide what temperature to set the dial on the stove? Or do you just do it “without really thinking”? Does your brain respond with annoying disclaimers when asking yourself medical questions? Does your brain refuse to entertain an idea if it’s questionable?
  - jncfhnb 3y ago
    LLMs “think” in vector spaces, and “operate” by picking out words one at a time. A full “idea” is there from the beginning. If you say “choose X or Y and then give three supporting arguments” the LLM does not write out the supporting arguments, but the vector space determining the initial one word answer of X or Y does include the embedded awareness of those arguments to different levels of specificity and relevance. The annoying disclaimers and refusals are not fundamental to LLMs but specific LLM services.
  - jddj 3y ago
    Observe your thought process while reading the following sentences. If you're not with us you're... If you scratch my back I'll.. A bird in the hand is worth... What is a bird in the hand worth? How did you know that?

grumbel 3y ago

One big difference is that LLMs don't do loops, new words are produced in a fixed amount of steps. They can't go "give me a minute", think for a bit in the background and come back with an answer. That's why you can often get better answers when you force them to go through the individual steps, as that allows them to use the prompt history as working memory. That said, this all seems like a 'fallacy of composition'. Humans are not LLMs, so much should be obvious, but at the same time concluding that they are completely different just feels wrong. The mistakes LLMs make feel very similar to what humans do when they don't have the time to deeply think about a problem and just give you the best guess that pops into their head. Humans will have other systems on top that allow them deeper reasoning, but the language generation really doesn't feel all that different from what LLMs do. That aside, humans interact with the world, they get instant feedback on what of their predictions is right or wrong. LLMs are stuck with just static training data that might simply not be enough to develop higher level reasoning skills.