3 ms·

> Nothing about language is determined by reality. This is just transparently silly. Or rather, you're equivocating between "language is not fully determined b

> Nothing about language is determined by reality. This is just transparently silly. Or rather, you're equivocating between "language is not fully determined by reality" and "language is uncorrelated with reality." Language is partially caused by reality; thus, large language models, seeking compression, learn the arbitrary parts and the reality parts separately. That's why they can translate. The human sensory stream is a kind of language. The exact same logic goes for it! If LLMs could not learn about reality from language because language was not determined by reality, then human brains could not learn anything about reality either. Nothing about reality forces a certain neural state vector to correspond to a certain sense impression. E pur si muove.

mjburgess 2y ago
Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever. Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society, etc. And alongside this all the while internalizing arbitary conventions by which we signify these things. A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc. The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to. You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. There is literally nothing at all in speed signs that has anything to do with their meaning. This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases. In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have. eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702c2 https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702... ..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task. Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.
- FeepingCreature 2y ago
  > Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. I just don't think this is correct. Language is embedded in reality, is a product of reality, and is correlated with reality. > As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Again, simply wrong. 5km/h limit, 60km/h limit, 120km/h limit - even the physical width of the text has a correlation to the exponent of the speed limit! Which will correlate to the curvature of the road! These are not arbitrary, they're tightly interlinked on every level except the specific shape of the glyphs themselves. That's why your argument only holds up for the most basic features such as glyph frequency. As you zoom out, structural similarities begin to appear; not in the shape of the text but in the shape of features abstracted from the text. That's why LLMs attain novel capabilities as you increase depth and context size, as they can begin to condition on the deeper structures of reality that leak through the language. As language is abstracted from reality, at sufficient scale, a cognition based on language begins to necessarily model reality itself. > You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. Completely unclear if this is true at any rate. Humans don't first learn to interact with the world and then learn language; language and world-understanding co-develop.
  - mjburgess 2y ago
    OK. well, lets give it 5yrs. in 10, at least, this will be over
    - FeepingCreature 2y ago
      This, at least, I agree with. (Though not quite in the same sense.)

mjburgess 2y ago

Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever. Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society, etc. And alongside this all the while internalizing arbitary conventions by which we signify these things. A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc. The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to. You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. There is literally nothing at all in speed signs that has anything to do with their meaning. This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases. In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have. eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702c2 https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702... ..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task. Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.

FeepingCreature 2y ago
> Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. I just don't think this is correct. Language is embedded in reality, is a product of reality, and is correlated with reality. > As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Again, simply wrong. 5km/h limit, 60km/h limit, 120km/h limit - even the physical width of the text has a correlation to the exponent of the speed limit! Which will correlate to the curvature of the road! These are not arbitrary, they're tightly interlinked on every level except the specific shape of the glyphs themselves. That's why your argument only holds up for the most basic features such as glyph frequency. As you zoom out, structural similarities begin to appear; not in the shape of the text but in the shape of features abstracted from the text. That's why LLMs attain novel capabilities as you increase depth and context size, as they can begin to condition on the deeper structures of reality that leak through the language. As language is abstracted from reality, at sufficient scale, a cognition based on language begins to necessarily model reality itself. > You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. Completely unclear if this is true at any rate. Humans don't first learn to interact with the world and then learn language; language and world-understanding co-develop.
- mjburgess 2y ago
  OK. well, lets give it 5yrs. in 10, at least, this will be over
  - FeepingCreature 2y ago
    This, at least, I agree with. (Though not quite in the same sense.)