4 ms·

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which i

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which isn't statistics ----- I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn't to say they haven't learned something during training, at least according to the colloquial meaning we typically ascribe to even lower models like MNIST classification. I'm even kind of okay with saying that it reasons about things in the same colloquial sense. In a lot of ways, we just don't have a good definition of what 'reasoning' is. Is it just bad at reasoning because it's input/output/modeling/training is insufficient? Humans struggle to learn multiplication tables when we're young. Are those humans not reasoning because they get the math wrong? But there isn't plasticity, there isn't adaptability, it's unclear to me that you can effectively inform it how to embed truly novel information - surely something that is possible, with some neurons existing for routing and activating other learned embeddings. Anyway, interesting stuff.

deleted 3y ago
[deleted]
jbay808 3y ago
I'm glad to see someone express this view, because I think this gets to the heart of the question. How does a stochastic parrot learn to sort lists? Embeddings are part of the compression-by-abstraction that I'm explaining in the first two parts, but the embeddings generated by an LLM go beyond the normal word2vec picture that most people have of embeddings, and I believe are closer to whatever "understanding" means if it could be formally defined. It would be quite a coincidence if GPT-4 happened to solve the riddle merely by virtue of "Moonling" and "cabbage" being closely-located vectors.
- eximius 3y ago
  Eh. I still consider them stochastic parrots. My concessions lie elsewhere, primarily in the vocabulary. We refer to algorithms like quicksort as 'reasoning' about the input. So it's fine to use the same sense of the word to apply to stochastic parrots. The difference between an LLM learning how to sort things and compiling an implementation of an algorithm like quicksort is not terribly large, from a certain perspective. I suppose something I'm interested in is whether an LLM that can't sort numbers could be instructed how as a prompt and then do so. There are some examples of similar phenomenon (the one with some kids made up language was interesting) which suggests the LLMs have a lot of space dedicated towards dynamic pattern selection in their context windows (somewhat tautological) in order to have prompts tune the selection for other layers. And, of course, lack of plasticity is really interesting.
  - jbay808 3y ago
    I just can't imagine how a stochastic parrot could repeat back a correctly-sorted list that it hasn't seen in training, without actually implementing a sorting algorithm in the process. It seems (and, by calculation, is) phenomenally unlikely that it would just stochastically happen to pick every single number correctly. When that is combined with the fact that transformers provably can implement proper deterministic sorting algorithms, it seems that the benefit of the doubt should go to the transformer having learned a sorting algorithm? LLMs aren't plastic in the sense that they don't learn anything when they aren't being trained. But they can be trained to execute different programs depending on the contents of the context window, like if it contains "wrong, try again:" so maybe they can learn from their mistakes in that sense. But if you could teach an LLM to sort by explaining it in the context window, the network would already have necessarily learned and stored a sorting algorithm somewhere; the text "here is how sorting is done: [...]" would just be serving as the trigger for that function call.
    - eximius 3y ago
      Again, I think the disagreement is not whether it has learned to approximate a sorting algorithm, but whether that qualifies as reasoning and, if it does, in what sense.
      - jbay808 3y ago
        I won't take a hard stance on what counts as "reasoning", which I picked in the title for lack of a better summarizing word; I am open to alternatives. So if you think that making abstractions and implementing a sorting algorithm does not count as reasoning, I will not disagree with that position. Where I am going to take a hard stance is on what does a stochastic parrot cannot do. And a stochastic parrot, defined as "stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning", cannot sort lists of 127 characters.
        hnfong 3y ago

deleted 3y ago

[deleted]

jbay808 3y ago

I'm glad to see someone express this view, because I think this gets to the heart of the question. How does a stochastic parrot learn to sort lists? Embeddings are part of the compression-by-abstraction that I'm explaining in the first two parts, but the embeddings generated by an LLM go beyond the normal word2vec picture that most people have of embeddings, and I believe are closer to whatever "understanding" means if it could be formally defined. It would be quite a coincidence if GPT-4 happened to solve the riddle merely by virtue of "Moonling" and "cabbage" being closely-located vectors.

eximius 3y ago
Eh. I still consider them stochastic parrots. My concessions lie elsewhere, primarily in the vocabulary. We refer to algorithms like quicksort as 'reasoning' about the input. So it's fine to use the same sense of the word to apply to stochastic parrots. The difference between an LLM learning how to sort things and compiling an implementation of an algorithm like quicksort is not terribly large, from a certain perspective. I suppose something I'm interested in is whether an LLM that can't sort numbers could be instructed how as a prompt and then do so. There are some examples of similar phenomenon (the one with some kids made up language was interesting) which suggests the LLMs have a lot of space dedicated towards dynamic pattern selection in their context windows (somewhat tautological) in order to have prompts tune the selection for other layers. And, of course, lack of plasticity is really interesting.
- jbay808 3y ago
  I just can't imagine how a stochastic parrot could repeat back a correctly-sorted list that it hasn't seen in training, without actually implementing a sorting algorithm in the process. It seems (and, by calculation, is) phenomenally unlikely that it would just stochastically happen to pick every single number correctly. When that is combined with the fact that transformers provably can implement proper deterministic sorting algorithms, it seems that the benefit of the doubt should go to the transformer having learned a sorting algorithm? LLMs aren't plastic in the sense that they don't learn anything when they aren't being trained. But they can be trained to execute different programs depending on the contents of the context window, like if it contains "wrong, try again:" so maybe they can learn from their mistakes in that sense. But if you could teach an LLM to sort by explaining it in the context window, the network would already have necessarily learned and stored a sorting algorithm somewhere; the text "here is how sorting is done: [...]" would just be serving as the trigger for that function call.
  - eximius 3y ago
    Again, I think the disagreement is not whether it has learned to approximate a sorting algorithm, but whether that qualifies as reasoning and, if it does, in what sense.
    - jbay808 3y ago
      I won't take a hard stance on what counts as "reasoning", which I picked in the title for lack of a better summarizing word; I am open to alternatives. So if you think that making abstractions and implementing a sorting algorithm does not count as reasoning, I will not disagree with that position. Where I am going to take a hard stance is on what does a stochastic parrot cannot do. And a stochastic parrot, defined as "stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning", cannot sort lists of 127 characters.
      - hnfong 3y ago