5 ms·

(2019) Note that this article precedes both ChatGPT and GPT-3. When it was written, Leibniz' idea of a machine reasoning by manipulating symbols was still scie

(2019) Note that this article precedes both ChatGPT and GPT-3. When it was written, Leibniz' idea of a machine reasoning by manipulating symbols was still science fiction. Now it is very much reality.

mensetmanusman 3y ago
Are the billions of parameters all symbols, or are you referencing math manipulation as symbol manipulation?
- cubefox 3y ago
  Both the inputs and outputs are symbols: text.
  - mensetmanusman 3y ago
    The machine is considering text the same way as RGB pixel values in the annealing process. Are these pixel values symbols? Maybe it’s all information and symbols are a small way to grasp at it but not the complete way.
WillAdams 3y ago
Does ChatGPT actually manipulate symbols, or does it string together strings of characters based on what most frequently occurs next? I haven't seen anything that indicates actual working w/ symbols/logic/ideas.
- kangalioo 3y ago
  s/frequently/likely (Large language models are not plain Markov chains, contrary to popular belief)
  - WillAdams 3y ago
    Having parameters for "temperature" and so forth, while it might provide sometimes useful functionality, does not change the basic nature.
- gberger 3y ago
  "string together strings of characters" = "manipulate symbols", no?
  - WillAdams 3y ago
    There is a big difference between "War and Peace" as a string of ISO-Latin1 characters which are frequently found in that order, and "novel by Tolstoy which contains many insights into the human condition" _and_ understanding the symbolism of said insights. A grade school student who has read and understood _War and Peace_ would be able to write a paper which is original to the degree that it did not previously exist as that exact sequence of characters, even if it had no original insights, while ChatGPT would regurgitate the most frequent combinations of characters which the metadata and so forth indicate are in the context of writing about that novel.
  - MichaelMoser123 3y ago
    i think it is working with embeddings - meaning that each word (or word stem) is represented by a very long vector of floating point numbers. These vectors are the result of something like word2vec. The input text is transformed into these embedding vectors and lined up in sequence (that's called the context window), then this gargantuan ML model works starts to process this gargantuan input sequence.
    - WillAdams 3y ago
      Yes, but it's a processing by association, not understanding.
deleted 3y ago
[deleted]

mensetmanusman 3y ago

Are the billions of parameters all symbols, or are you referencing math manipulation as symbol manipulation?

cubefox 3y ago
Both the inputs and outputs are symbols: text.
- mensetmanusman 3y ago
  The machine is considering text the same way as RGB pixel values in the annealing process. Are these pixel values symbols? Maybe it’s all information and symbols are a small way to grasp at it but not the complete way.

WillAdams 3y ago

Does ChatGPT actually manipulate symbols, or does it string together strings of characters based on what most frequently occurs next? I haven't seen anything that indicates actual working w/ symbols/logic/ideas.

kangalioo 3y ago
s/frequently/likely (Large language models are not plain Markov chains, contrary to popular belief)
- WillAdams 3y ago
  Having parameters for "temperature" and so forth, while it might provide sometimes useful functionality, does not change the basic nature.
gberger 3y ago
"string together strings of characters" = "manipulate symbols", no?
- WillAdams 3y ago
  There is a big difference between "War and Peace" as a string of ISO-Latin1 characters which are frequently found in that order, and "novel by Tolstoy which contains many insights into the human condition" _and_ understanding the symbolism of said insights. A grade school student who has read and understood _War and Peace_ would be able to write a paper which is original to the degree that it did not previously exist as that exact sequence of characters, even if it had no original insights, while ChatGPT would regurgitate the most frequent combinations of characters which the metadata and so forth indicate are in the context of writing about that novel.
- MichaelMoser123 3y ago
  i think it is working with embeddings - meaning that each word (or word stem) is represented by a very long vector of floating point numbers. These vectors are the result of something like word2vec. The input text is transformed into these embedding vectors and lined up in sequence (that's called the context window), then this gargantuan ML model works starts to process this gargantuan input sequence.
  - WillAdams 3y ago
    Yes, but it's a processing by association, not understanding.

deleted 3y ago

[deleted]