17 ms·

I think referring to ChatGPT as an advanced autocomplete is too much of a reduction, to the point of leading people to incorrect conclusions; Or at least conclu

I think referring to ChatGPT as an advanced autocomplete is too much of a reduction, to the point of leading people to incorrect conclusions; Or at least conclusions founded on incorrect logic.

azinman2 3y ago
It’s more correct than not. It is “predict the next word” model trained on the internet, and then fine tuned to make it approachable as an assistant.
- Closi 3y ago
  And computers are just lots of really fast logic gates. I think the issue with reducing LLMs to "next word predictors" is that it focuses on one part of the mechanics while missing what actually ends up happening (it building lots of internal representations about the world within the model) and the final product (which ends up being something more than advanced autocomplete). Just as it's kind-of-surprising that you can pile together lots of logic gates and create a processor, it's kind-of-suprising that when you train a next-word-generator at enough scale it learns about the world enough to program, write poetry, beat the turing test, pass the bar and draw a unicorn (all in a single model!).
  - enterprise_cog 3y ago
    Poor analogy given logic gates are deterministic while an LLM is not.
    - viraptor 3y ago
      LLM implementation may be deterministic or not. The idea/tech itself does not restrict this in any way.
    - baobabKoodaa 3y ago
      You can coerce deterministic and reproducible outputs from an LLM
    - Closi 3y ago
      What do you think LLMs are made of?
      - Grimblewald 3y ago
        poor question, we know what human brains are made of, doesn't help us understand them all too much.
    - RugnirViking 3y ago
      LLMs are deterministic though? like with 0 temprature you always get the same output (temprature is literally injected randomness because by default they are "too" deterministic)
      - enterprise_cog 3y ago
        A 0 temp LLM is just an autocomplete. Not even spicy.
- CamperBob2 3y ago
  As you type your reply to this post, consider this: where do your words come from? Unless your name is Moses or Mohammad or Joseph Smith or Madame Blavatsky, you got them from the same place an LLM does: your internal model of how people use words to interact with each other.
- Buttons840 3y ago
  Am I a predict-the-next-word model? Whenever I think of the words of a song I have to do it in order. Sometimes I want to remember a certain part, but I can't just jump there in my mind unless I sing the words in order.
  - azinman2 3y ago
    No you’re not. You think in often non-verbal semantic frames versus individual words, aided by many senses and context that isn’t limited to the words spoken to you just before, with your own autonomous goals and plans that span short and long terms beyond any one conversation or context, with the ability to absorb new information that fundamentally changes how you then act, experience emotion and grounding of language to your environment, and countless other differences. I tire of this trope that is so obviously is not the case.
    - Grimblewald 3y ago
      'non-verbal semantic frames vs individual words' ok i'd agree for things using lstm style architecture but that goes out the window with transformers.
- Grimblewald 3y ago
  if we found out tomorrow that all we do is 'complete the word' based on context would you still hold that mechanic to such a lowly opinion?
  - azinman2 3y ago
    But we don’t. See my other comment.
    - Grimblewald 3y ago
      the "non-verbal semantic frames vs individual words" one? Not true for all people, if any truly are. Interrupt most people, they need to start the sentence again for example. Similar things may be true for transformer networks, which is why these can do high quality zero-shot translation between languages for which these were shown no translation example using only their 'understanding' of that language and the knowledge of how to translate between some other languages. This is because once a LLM has to be polyglottal it is more efficient to gain an 'understanding' of concepts then separately an understanding of language. This way it can save on space for learning new concepts and simply learn to map its 'personal understanding' if you will to the target language.

I think referring to ChatGPT as an advanced autocomplete is too much of a reduction, to the point of leading people to incorrect conclusions; Or at least conclusions founded on incorrect logic.

azinman2 3y ago

It’s more correct than not. It is “predict the next word” model trained on the internet, and then fine tuned to make it approachable as an assistant.

Closi 3y ago
And computers are just lots of really fast logic gates. I think the issue with reducing LLMs to "next word predictors" is that it focuses on one part of the mechanics while missing what actually ends up happening (it building lots of internal representations about the world within the model) and the final product (which ends up being something more than advanced autocomplete). Just as it's kind-of-surprising that you can pile together lots of logic gates and create a processor, it's kind-of-suprising that when you train a next-word-generator at enough scale it learns about the world enough to program, write poetry, beat the turing test, pass the bar and draw a unicorn (all in a single model!).
- enterprise_cog 3y ago
  Poor analogy given logic gates are deterministic while an LLM is not.
  - viraptor 3y ago
    LLM implementation may be deterministic or not. The idea/tech itself does not restrict this in any way.
  - baobabKoodaa 3y ago
    You can coerce deterministic and reproducible outputs from an LLM
  - Closi 3y ago
    What do you think LLMs are made of?
    - Grimblewald 3y ago
      poor question, we know what human brains are made of, doesn't help us understand them all too much.
  - RugnirViking 3y ago
    LLMs are deterministic though? like with 0 temprature you always get the same output (temprature is literally injected randomness because by default they are "too" deterministic)
    - enterprise_cog 3y ago
      A 0 temp LLM is just an autocomplete. Not even spicy.
CamperBob2 3y ago
As you type your reply to this post, consider this: where do your words come from? Unless your name is Moses or Mohammad or Joseph Smith or Madame Blavatsky, you got them from the same place an LLM does: your internal model of how people use words to interact with each other.
Buttons840 3y ago
Am I a predict-the-next-word model? Whenever I think of the words of a song I have to do it in order. Sometimes I want to remember a certain part, but I can't just jump there in my mind unless I sing the words in order.
- azinman2 3y ago
  No you’re not. You think in often non-verbal semantic frames versus individual words, aided by many senses and context that isn’t limited to the words spoken to you just before, with your own autonomous goals and plans that span short and long terms beyond any one conversation or context, with the ability to absorb new information that fundamentally changes how you then act, experience emotion and grounding of language to your environment, and countless other differences. I tire of this trope that is so obviously is not the case.
  - Grimblewald 3y ago
    'non-verbal semantic frames vs individual words' ok i'd agree for things using lstm style architecture but that goes out the window with transformers.
Grimblewald 3y ago
if we found out tomorrow that all we do is 'complete the word' based on context would you still hold that mechanic to such a lowly opinion?
- azinman2 3y ago
  But we don’t. See my other comment.
  - Grimblewald 3y ago
    the "non-verbal semantic frames vs individual words" one? Not true for all people, if any truly are. Interrupt most people, they need to start the sentence again for example. Similar things may be true for transformer networks, which is why these can do high quality zero-shot translation between languages for which these were shown no translation example using only their 'understanding' of that language and the knowledge of how to translate between some other languages. This is because once a LLM has to be polyglottal it is more efficient to gain an 'understanding' of concepts then separately an understanding of language. This way it can save on space for learning new concepts and simply learn to map its 'personal understanding' if you will to the target language.