4 ms·

> In the field of understanding text, this is a progress which had seemed largely impossible before. ... LLMs don't understand text. They encode statistical

> In the field of understanding text, this is a progress which had seemed largely impossible before. ... LLMs don't understand text. They encode statistical probabilities of relationships between strings of characters. Those are not the same thing. That's the entire damn point of this article! Honestly, for someone taking pot shots at "laymen", this is a profound thing to misunderstand.

xg15 4y ago
Ok, on the risk of sounding exactly like one of the "sentience" guys, but "statistical probabilities of relationships between strings of characters" is a non-answer that makes the whole thing sound simpler than it is without actually providing any explanation how it works. What kind of statistical probabilities exactly? Right now, research is still ongoing what kind of knowledge is actually captured inside the models, but there are some hints that higher-level, "semantic" knowledge might emerge during training: https://thegradient.pub/othello/ https://thegradient.pub/othello/
- deleted 4y ago
  [deleted]
- tysam_and 4y ago
  Both are happening, I believe. The LLM absorbs the manifold (high-dimensional shape) of the data. The manifold contains the underlying concepts of grammatical structure, abstract thought, and inductive reasoning, which the LLM (within reasonable capacity and structural capability) captures because it is a more efficient representation of the underlying data. Transformers are simple and contain the correct building blocks to efficiently capture these representations during training. This allows for out-of domain generalization. This is most, to my knowledge, of what happens in most deep learning processes and is likely 90% of the main content that one needs to know to understand neural networks at their core. I think it's pretty basic but it often gets buried in unconscionable mathematical symbols and fancy-speak to be of use to anyone reading.
  - xg15 4y ago
    Yeah, that's seems to be my impression as well. Of course it's true that in a sense all it does is capturing statistical relationships - because that's all that is there in the input data. However what kind of relationships it captures may still be arbitrarily complex (within the contraints of the model architecture). We know it's more complex than just "empirical probability of word x given the n words before", because that would result in a markov chain and we know those don't generate the kind of output we are seeing. We also see that it's able to map descriptions of tasks to its execution, even for unseen tasks. E.g., I can tell it "Write a limerick that contains the names of all living US presidents and format is as a JSON array inside a python script.". The result will probably contain some dead US presidents or some canadian prime ministers or whatever, and the text may be a haiku and not a limerick - but the output will usually be a python function with a JSON array with a poem with some names in it. I don't see how that could work without a more abstract representation of the concepts "president's names", "poem", "json" and "python", so it can combine them meaningfully into a single response.
    - quartzic 4y ago
      Tried this out and got this result: import json presidents = ["Biden", "Obama", "Bush", "Clinton", "Carter"] limerick = "There once were presidents five, " \ "Biden, Obama, Bush, Clinton, and Carter alive. " \ "They served our country with pride, " \ "And kept our democracy alive. " \ "May they continue to thrive!" print(json.dumps({"limerick": limerick, "presidents": presidents}))
    - tysam_and 4y ago
      Yes, and very good points. I really appreciate your response. Though with respect to the markov chain side of things I think you may be missing that a neural network is still fundamentally distilling a markov chain into its representations -- that is exactly what it is predicting (the transition matrix over n tokens). Presumably though, the generalization happens in that the raw n-context chain (for example, a 2048 token context chain) is completely infeasible to compress into the network directly, so the network takes shortcuts where it is least punished, which I believe invariably trims the combinatorial edge cases where certain things don't show up in the data. So it is estimating the markov chain, just in a way that is compressible and according to the inductive biases as we define them (i.e. what we nearly force the network towards with our architectural and otherwise decisions).

xg15 4y ago

Ok, on the risk of sounding exactly like one of the "sentience" guys, but "statistical probabilities of relationships between strings of characters" is a non-answer that makes the whole thing sound simpler than it is without actually providing any explanation how it works. What kind of statistical probabilities exactly? Right now, research is still ongoing what kind of knowledge is actually captured inside the models, but there are some hints that higher-level, "semantic" knowledge might emerge during training: https://thegradient.pub/othello/ https://thegradient.pub/othello/

deleted 4y ago
[deleted]
tysam_and 4y ago
Both are happening, I believe. The LLM absorbs the manifold (high-dimensional shape) of the data. The manifold contains the underlying concepts of grammatical structure, abstract thought, and inductive reasoning, which the LLM (within reasonable capacity and structural capability) captures because it is a more efficient representation of the underlying data. Transformers are simple and contain the correct building blocks to efficiently capture these representations during training. This allows for out-of domain generalization. This is most, to my knowledge, of what happens in most deep learning processes and is likely 90% of the main content that one needs to know to understand neural networks at their core. I think it's pretty basic but it often gets buried in unconscionable mathematical symbols and fancy-speak to be of use to anyone reading.
- xg15 4y ago
  Yeah, that's seems to be my impression as well. Of course it's true that in a sense all it does is capturing statistical relationships - because that's all that is there in the input data. However what kind of relationships it captures may still be arbitrarily complex (within the contraints of the model architecture). We know it's more complex than just "empirical probability of word x given the n words before", because that would result in a markov chain and we know those don't generate the kind of output we are seeing. We also see that it's able to map descriptions of tasks to its execution, even for unseen tasks. E.g., I can tell it "Write a limerick that contains the names of all living US presidents and format is as a JSON array inside a python script.". The result will probably contain some dead US presidents or some canadian prime ministers or whatever, and the text may be a haiku and not a limerick - but the output will usually be a python function with a JSON array with a poem with some names in it. I don't see how that could work without a more abstract representation of the concepts "president's names", "poem", "json" and "python", so it can combine them meaningfully into a single response.
  - quartzic 4y ago
    Tried this out and got this result: import json presidents = ["Biden", "Obama", "Bush", "Clinton", "Carter"] limerick = "There once were presidents five, " \ "Biden, Obama, Bush, Clinton, and Carter alive. " \ "They served our country with pride, " \ "And kept our democracy alive. " \ "May they continue to thrive!" print(json.dumps({"limerick": limerick, "presidents": presidents}))
  - tysam_and 4y ago
    Yes, and very good points. I really appreciate your response. Though with respect to the markov chain side of things I think you may be missing that a neural network is still fundamentally distilling a markov chain into its representations -- that is exactly what it is predicting (the transition matrix over n tokens). Presumably though, the generalization happens in that the raw n-context chain (for example, a 2048 token context chain) is completely infeasible to compress into the network directly, so the network takes shortcuts where it is least punished, which I believe invariably trims the combinatorial edge cases where certain things don't show up in the data. So it is estimating the markov chain, just in a way that is compressible and according to the inductive biases as we define them (i.e. what we nearly force the network towards with our architectural and otherwise decisions).