4 ms·

> just find the most probable word that follows next Well, if in all situations you can predict which word Einstein would probably say next, then I think you'r

> just find the most probable word that follows next Well, if in all situations you can predict which word Einstein would probably say next, then I think you're in a good spot. This "most probable" stuff is just absurd handwaving. Every prompt of even a few words is unique, there simply is no trivially "most probable" continuation. Probable given what? What these machines learn to do is predicting what intelligence would do, which is the same as being intelligent.

qsera 8mo ago
>Probable given what? The training data.. >predicting what intelligence would do No, it just predict what the next word would be if an intelligent entity translated its thoughts to words. Because it is trained on the text that are written by intelligent entities. If it was trained on text written by someone who loves to rhyme, you would be getting all rhyming responses. It imitates the behavior -- in text -- of what ever entity that generated the training data. Here the training data was made by intelligent humans, so we get an imitation of the same. It is a clever party trick that works often enough.
- throw310822 8mo ago
  > The training data If the prompt is unique, it is not in the training data. True for basically every prompt. So how is this probability calculated?
  - qsera 8mo ago
    Just using a scaled up and cleverly tweaked version of linear regression analysis...
    - red75prime 8mo ago
      That is, the probability distribution that the network should learn is defined by which probability distribution the network has learned. Brilliant!
  - cbovis 8mo ago
    The prompt is unique but the tokens aren't. Type "owejdpowejdojweodmwepiodnoiwendoinw welidn owindoiwendo nwoeidnweoind oiwnedoin" into ChatGPT and the response is "The text you sent appears to be random or corrupted and doesn’t form a clear question." because the prompt doesnt correlate to training data.
    - hmmmmmmmmmmmmmm 8mo ago
      ...? what is the response supposed to be here?
    - HDThoreaun 8mo ago
      Or because the text you send was random and doesnt form a clear quesiton?
    - newswasboring 8mo ago
      > The prompt is unique but the tokens aren't. The tokens aren't unique, but the sequence is. Every input this model sees in unique. Even tokens are not as simple as they seem If you type "ejst os th xspitsl of fermaby?" in ChatGPT it responds with > It looks like you typed “ejst os th xspitsl of fermaby?”, which seems like a garbled version of: > "What is the capital of Germany?” > The capital of Germany is Berlin. > If you meant to ask something else, feel free to clarify!" edit: formatting
    - ajam1507 8mo ago
      The prompt does correlate to its training data. In this case, since you sent random text, it generated the most likely response to random text.
  - hmmmmmmmmmmmmmm 8mo ago
    Hamiltonian paths and previous work by Donald Knuth is more than likely in the training data.
    - red75prime 8mo ago
      The specific sequence of tokens that comprise the Knuth's problem with an answer to it is not in the training data. A naive probability distribution based on counting token sequences that are present in the training data would assign 0 probability to it. The trained network represents extremely non-naive approach to estimating the ground-truth distribution (the distribution that corresponds to what a human brain might have produced).
      - qsera 8mo ago
        >the distribution that corresponds to what a human brain might have produced.. But the human brain (or any other intelligent brain) does not work by generating probability distribution of the next word. Even beings that does not have a language can think and act intelligent.
        astrange 8mo ago
        LLMs also don't work by generating probability distributions of the next word. Your explanation isn't able to explain why they can generate words, let alone sentences.
        qsera 8mo ago
        That is exactly how they work.
        astrange 8mo ago
        No, a token is not a word.
        8mo ago
- empath75 8mo ago
  It is impossible to accurately imitate the action of intelligent beings without being intelligent. To believe otherwise is to believe that intelligence is a vacuous property.
  - qsera 8mo ago
    >It is impossible to accurately imitate the action of intelligent beings without being intelligent. Wait what? So a robot who is accurately copying the actions of an intelligent human, is intelligent?
    - empath75 8mo ago
      That was probably phrased poorly. If a robot can independently accurately do what an intelligent person would do when placed in a novel situation, then yes, I would say it is intelligent. If it's just basically being a puppet, then no. You tell me what claude code is more like, a puppet, or a person?
      - qsera 8mo ago
        It is neither puppet or a person. It is a computer program.
        throw310822 8mo ago
        As much as a bundle of an mp3 decoder and a terabyte of mp3 music are "just a program".
    - UltraSane 8mo ago
      How can you distinguish intelligence form a sufficiently accurate imitation of intelligence?
      - slopinthebag 8mo ago
        By "sufficiently accurate" do you mean identical? Because if so, it's not an imitation of intelligence at all, and the question is thus nonsensical.

qsera 8mo ago

>Probable given what? The training data.. >predicting what intelligence would do No, it just predict what the next word would be if an intelligent entity translated its thoughts to words. Because it is trained on the text that are written by intelligent entities. If it was trained on text written by someone who loves to rhyme, you would be getting all rhyming responses. It imitates the behavior -- in text -- of what ever entity that generated the training data. Here the training data was made by intelligent humans, so we get an imitation of the same. It is a clever party trick that works often enough.

throw310822 8mo ago
> The training data If the prompt is unique, it is not in the training data. True for basically every prompt. So how is this probability calculated?
- qsera 8mo ago
  Just using a scaled up and cleverly tweaked version of linear regression analysis...
  - red75prime 8mo ago
    That is, the probability distribution that the network should learn is defined by which probability distribution the network has learned. Brilliant!
- cbovis 8mo ago
  The prompt is unique but the tokens aren't. Type "owejdpowejdojweodmwepiodnoiwendoinw welidn owindoiwendo nwoeidnweoind oiwnedoin" into ChatGPT and the response is "The text you sent appears to be random or corrupted and doesn’t form a clear question." because the prompt doesnt correlate to training data.
  - hmmmmmmmmmmmmmm 8mo ago
    ...? what is the response supposed to be here?
  - HDThoreaun 8mo ago
    Or because the text you send was random and doesnt form a clear quesiton?
  - newswasboring 8mo ago
    > The prompt is unique but the tokens aren't. The tokens aren't unique, but the sequence is. Every input this model sees in unique. Even tokens are not as simple as they seem If you type "ejst os th xspitsl of fermaby?" in ChatGPT it responds with > It looks like you typed “ejst os th xspitsl of fermaby?”, which seems like a garbled version of: > "What is the capital of Germany?” > The capital of Germany is Berlin. > If you meant to ask something else, feel free to clarify!" edit: formatting
  - ajam1507 8mo ago
    The prompt does correlate to its training data. In this case, since you sent random text, it generated the most likely response to random text.
- hmmmmmmmmmmmmmm 8mo ago
  Hamiltonian paths and previous work by Donald Knuth is more than likely in the training data.
  - red75prime 8mo ago
    The specific sequence of tokens that comprise the Knuth's problem with an answer to it is not in the training data. A naive probability distribution based on counting token sequences that are present in the training data would assign 0 probability to it. The trained network represents extremely non-naive approach to estimating the ground-truth distribution (the distribution that corresponds to what a human brain might have produced).
    - qsera 8mo ago
      >the distribution that corresponds to what a human brain might have produced.. But the human brain (or any other intelligent brain) does not work by generating probability distribution of the next word. Even beings that does not have a language can think and act intelligent.
      - astrange 8mo ago
        LLMs also don't work by generating probability distributions of the next word. Your explanation isn't able to explain why they can generate words, let alone sentences.
        qsera 8mo ago
        That is exactly how they work.
        astrange 8mo ago
        No, a token is not a word.
        8mo ago
empath75 8mo ago
It is impossible to accurately imitate the action of intelligent beings without being intelligent. To believe otherwise is to believe that intelligence is a vacuous property.
- qsera 8mo ago
  >It is impossible to accurately imitate the action of intelligent beings without being intelligent. Wait what? So a robot who is accurately copying the actions of an intelligent human, is intelligent?
  - empath75 8mo ago
    That was probably phrased poorly. If a robot can independently accurately do what an intelligent person would do when placed in a novel situation, then yes, I would say it is intelligent. If it's just basically being a puppet, then no. You tell me what claude code is more like, a puppet, or a person?
    - qsera 8mo ago
      It is neither puppet or a person. It is a computer program.
      - throw310822 8mo ago
        As much as a bundle of an mp3 decoder and a terabyte of mp3 music are "just a program".
  - UltraSane 8mo ago
    How can you distinguish intelligence form a sufficiently accurate imitation of intelligence?
    - slopinthebag 8mo ago
      By "sufficiently accurate" do you mean identical? Because if so, it's not an imitation of intelligence at all, and the question is thus nonsensical.