3 ms·

> Indeed, there is no function which generates text data, there's a very large number of independent generating processes that give rise to the distribution of

> Indeed, there is no function which generates text data, there's a very large number of independent generating processes that give rise to the distribution of text. The phrase "the war in ukraine" acquires a different semantics over 2010-2030 in a radically different way than, "I liked that film" does. But those semantics are revealed in the greater context of the phrase! That's why it's so important that transformers can attend to large context ranges; that's what lets them learn the greater semantic patterns to begin with. And at the limit, at a scale greater than phrases, I simply reject the idea that the same article - the same book - can have totally different meanings depending on context. Language isn't just shaped by context, it shapes context itself. Because language cannot be considered without context, it reveals information about that context, and in fact any compression of language ultimately requires modelling the person and even society that produced it. That's what the network learns. If your words don't have meaning beyond themselves, what are you even talking about?

mjburgess 2y ago
> If your words don't have meaning beyond themselves, what are you even talking about? ...err... of course? That's the whole point. "Context" here isnt other words. It's the world. The meaning of words is the world. When I say, "I like what you're wearing" i'm not summarising a history of prior texts; it has nothnig to do with any statistical operation over historical documents. It has entirely to do with what you're wearing. Langauge use is a side-effect of being embedded in reality, directly attentive to it, and so on. Words are mere symptoms of how we are situated in the world. LLMs merely replay these back to us. They are not in the world. They cannot, in principle, ever mean, "I like what you're wearing"
- FeepingCreature 2y ago
  I'd say exactly the opposite. It's because human speech is about the world, and LLM speech is about human speech, that LLM speech is about the world, by transitivity. "The ball fell to the <floor>", the LLM predicts, ultimately, not because of any feature of the human brain, or any feature of the English language, but because of gravity. That the causal arrow passes through a human brain does not make this any less true! Because human speech is inextricable from physical reality, language models likewise learn to model reality. They learn this poorly, incrementally, making missteps on the way - granted! But that they learn it at all shows that there's more going on than statistical modelling - or that statistical modelling of language is more than it sounds like.
  - mjburgess 2y ago
    Human speech isnt about reality. Speech, langauge, tokens, etc. are just scratchings on a wall. When we speak we infer the reality-orientation of people's intentions (and so on). The strings 010101010111100101, 000010101010111010101, 120192019092109209102, etc. are all "about" the exactly the same scenario whereby different communities of speakers adopt different conventions for the meaning of the terms (indeed, whether pairs/tripples/etc. are terms). This is not like a thermometer in a glass of water, the height of the mecury being determined by the temperature. Nothing about language is determined by reality. It is purely a highly lossly encoding scheme based on historically arbitary encoding conventions. It isn't a measure of reality in any sense. Modelling this encoding scheme doesn't model anything about the world. It only appears to if you generate text according to this scheme.. and only to those who can decode it. Hieroglyphs likewise, scratches on clay, and everything else. You are subject to the illusion of intentionality whereby your phenomenal experience of langauge is rendered immediately meaningful by your brain, ie., your decoding scheme is prior to the construction of your experience of the world. This is wholey absent from anything machines are processing.. whih are only the frequency relationships between tokenizations of syntax
    - FeepingCreature 2y ago
      > Nothing about language is determined by reality. This is just transparently silly. Or rather, you're equivocating between "language is not fully determined by reality" and "language is uncorrelated with reality." Language is partially caused by reality; thus, large language models, seeking compression, learn the arbitrary parts and the reality parts separately. That's why they can translate. The human sensory stream is a kind of language. The exact same logic goes for it! If LLMs could not learn about reality from language because language was not determined by reality, then human brains could not learn anything about reality either. Nothing about reality forces a certain neural state vector to correspond to a certain sense impression. E pur si muove.
      - mjburgess 2y ago
        Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever. Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society, etc. And alongside this all the while internalizing arbitary conventions by which we signify these things. A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc. The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to. You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. There is literally nothing at all in speed signs that has anything to do with their meaning. This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases. In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have. eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702c2 https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702... ..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task. Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.

mjburgess 2y ago

> If your words don't have meaning beyond themselves, what are you even talking about? ...err... of course? That's the whole point. "Context" here isnt other words. It's the world. The meaning of words is the world. When I say, "I like what you're wearing" i'm not summarising a history of prior texts; it has nothnig to do with any statistical operation over historical documents. It has entirely to do with what you're wearing. Langauge use is a side-effect of being embedded in reality, directly attentive to it, and so on. Words are mere symptoms of how we are situated in the world. LLMs merely replay these back to us. They are not in the world. They cannot, in principle, ever mean, "I like what you're wearing"

FeepingCreature 2y ago
I'd say exactly the opposite. It's because human speech is about the world, and LLM speech is about human speech, that LLM speech is about the world, by transitivity. "The ball fell to the <floor>", the LLM predicts, ultimately, not because of any feature of the human brain, or any feature of the English language, but because of gravity. That the causal arrow passes through a human brain does not make this any less true! Because human speech is inextricable from physical reality, language models likewise learn to model reality. They learn this poorly, incrementally, making missteps on the way - granted! But that they learn it at all shows that there's more going on than statistical modelling - or that statistical modelling of language is more than it sounds like.
- mjburgess 2y ago
  Human speech isnt about reality. Speech, langauge, tokens, etc. are just scratchings on a wall. When we speak we infer the reality-orientation of people's intentions (and so on). The strings 010101010111100101, 000010101010111010101, 120192019092109209102, etc. are all "about" the exactly the same scenario whereby different communities of speakers adopt different conventions for the meaning of the terms (indeed, whether pairs/tripples/etc. are terms). This is not like a thermometer in a glass of water, the height of the mecury being determined by the temperature. Nothing about language is determined by reality. It is purely a highly lossly encoding scheme based on historically arbitary encoding conventions. It isn't a measure of reality in any sense. Modelling this encoding scheme doesn't model anything about the world. It only appears to if you generate text according to this scheme.. and only to those who can decode it. Hieroglyphs likewise, scratches on clay, and everything else. You are subject to the illusion of intentionality whereby your phenomenal experience of langauge is rendered immediately meaningful by your brain, ie., your decoding scheme is prior to the construction of your experience of the world. This is wholey absent from anything machines are processing.. whih are only the frequency relationships between tokenizations of syntax
  - FeepingCreature 2y ago
    > Nothing about language is determined by reality. This is just transparently silly. Or rather, you're equivocating between "language is not fully determined by reality" and "language is uncorrelated with reality." Language is partially caused by reality; thus, large language models, seeking compression, learn the arbitrary parts and the reality parts separately. That's why they can translate. The human sensory stream is a kind of language. The exact same logic goes for it! If LLMs could not learn about reality from language because language was not determined by reality, then human brains could not learn anything about reality either. Nothing about reality forces a certain neural state vector to correspond to a certain sense impression. E pur si muove.
    - mjburgess 2y ago
      Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever. Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society, etc. And alongside this all the while internalizing arbitary conventions by which we signify these things. A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc. The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to. You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. There is literally nothing at all in speed signs that has anything to do with their meaning. This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases. In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have. eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702c2 https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702... ..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task. Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.