6 ms·

Sure, they're a reification of some aspect of meaning. The question is: which aspect(s), and which not. It is also the case that animals do not "reliably and u

Sure, they're a reification of some aspect of meaning. The question is: which aspect(s), and which not. It is also the case that animals do not "reliably and univerally" implement all aspects of all meanings they are acquainted with, so we aren't looking for 100% of capacities, 100% of the time. Nevertheless, LLMs are only implementing a limited aspect of meaning: mostly association and "some extension". And with this, plus everything ever written, they can narrowly appear to implement much more. Let's be clear though, when we say "implement" we mean that an answer arises from a prompt for a very specific reason: because the answer is meant by the system in the relevant way. In this sense, LLMs can mean any association, perhaps they can mean a few extensions, but they cannot mean anything else. Whenver an LLM appears to partake in more aspects of meaning it is only cheating: it is using familiarity with families of associations to overcome its disabilities. Like the idiot savant who appears to know all hollywood starlets, but is discovered eventually, not to realise they are all film stars. We routinely discover these disabilities in LLMs, when they attempt to engage in reasoning beyond these (formally,) narrow contexts of use. Agentic AI is a very good "on steroids" version of this. Just try to use windsurf, and the brittle edges of this trick appear quickly. It's "reasononing" whenver it seems to work, and "hallucination" when not -- but of course, it just never was reasoning.

TeMPOraL 1y ago
> LLMs are only implementing a limited aspect of meaning: mostly association and "some extension". > Whenver an LLM appears to partake in more aspects of meaning it is only cheating: it is using familiarity with families of associations to overcome its disabilities. I'm not convinced there's anything more to "meaning" - we seem to be defining concepts through relationship to other concepts, and ground that directly or indirectly with experiences. The richer that structure is, the more nuanced it gets. > Like the idiot savant who appears to know all hollywood starlets, but is discovered eventually, not to realise they are all film stars. We routinely discover these disabilities in LLMs, when they attempt to engage in reasoning beyond these (formally,) narrow contexts of use. I see those as limitations of degree, not kind. Less idiot savant, more like someone being hurried to answer questions on the spot. Some associations are stronger and come to mind immediately, some are less "fresh in memory", and then associations can bring false positives and it takes extra time/effort to notice and correct those. It's a common human experience, too. "Yes, those reserved words are 'void', 'var', 'volatile',... wait, 'var' is JS stuff, it's not reserved in C..." etc. Then, of course, humans are learning continuously, and - perhaps more importantly - even if they're not learning, they're reinforcing (or attenuating) existing associations through bringing them up and observing feedback. LLMs can't do that on-line, but that's an engineering limitation, not a theoretical one. I'm not claiming that LLMs are equivalent to humans in general sense. Just that they seem to be implementing the fundamental machinery behind "meaning" and "understanding" in general sense, and the theoretical structure behind it is quite pretty, and looks to me like a solution to a host of philosophical problems around meaning and language.
- mjburgess 1y ago
  Is this based on analogising LLMs to animal mental capacities, or based on a scientific study of these capacities? ie., is this confirmation bias, or science? One can always find a kind of confirmation bias analysis here, which "saves the appearances", ie., one can always say "take a measurement set of people's mental capacities, given in their linguistic behaviour" and find such behaviours apparent in LLMs. This will always be possible for the obvious reason that LLMs are trained on human linguistic practice. This makes "linguistic measurement" of LLMs especially deceptive. Consider the analogous case of measuring a video games by it's pixels: does it really have a "3d space" ? No. It only appears to. We know that pixel-space measurements of video games are necessarily deceptive, because we constructed them that way, so it is obvious that you cannot "walk into a tv". Yet we did not construct the mechanism of deception in LLMs, making seeing thru the failure of "linguistic measurement" apparently somewhat harder. But I imagine this is just a matter of time -- in particular, when LLM's mechanisms are fully traced, it will be more obvious that their outputs are not generated for the reasons we suppose. That the "reason to linguistic output" mapping we use on people is deceptive as applied to LLMs. Just as a screenshot of a video game is a deceptive measure, whereas a photograph isnt. For a photograph, the reason the mountain is small is because its far away; for a screenshot, it isnt: there is no mountain, it is not far away from the camera, there is no camera. In the case of LLMs we know they cannot mean what they say. We know that if an LLM offers a report on new york it cannot mean what a person who has travelled to new york means. The LLM is drawing on an arrangment, in token space, of tokens placed there by people who have been to new york. This arrangement is like the "rasterization" of a video game: it places pixels as-if there were 3d. You could say, then, that an LLM's response is a kind of rasterization of meaning. And just as with a video game, there are failures, eg., clipping through "solid" objects. LLMs do not genuinely compose concepts, because they have no concpets -- they can only act as if they are composing them, so long as a token-space measurement of composition is available in the weight-space of the model. (And so on...) The failures of LLMs to have these capacities will be apparent after awhile, at the moment we're on the hype rollercoaster, and its not yeet peaked. At the moment, people are still using the "reason-lingusitic" mapping theyve learned from human communication on LLMs, to impart the relevant menetal states they would with people. The boundaries of the failure of this mapping isnt yet clear to everyone. Users don't yet avoid "clipping thru" objects, beause they can't understand what clipping is -- at the moment, many seem to be desperate to say that if a video game object is clipped thru, it must be designed to be hollow. In any case, as i've said in many places in this thread (which you can see from my recent comment history) -- there are a large variety of mental capacities associated with apprehending meaning that LLMs lack. But the process is anti-inductive so it will take quite awhile: for all those who are finding the fragile boundaries ("clipping thru the terrian") new models come out with invisible walls.

TeMPOraL 1y ago

> LLMs are only implementing a limited aspect of meaning: mostly association and "some extension". > Whenver an LLM appears to partake in more aspects of meaning it is only cheating: it is using familiarity with families of associations to overcome its disabilities. I'm not convinced there's anything more to "meaning" - we seem to be defining concepts through relationship to other concepts, and ground that directly or indirectly with experiences. The richer that structure is, the more nuanced it gets. > Like the idiot savant who appears to know all hollywood starlets, but is discovered eventually, not to realise they are all film stars. We routinely discover these disabilities in LLMs, when they attempt to engage in reasoning beyond these (formally,) narrow contexts of use. I see those as limitations of degree, not kind. Less idiot savant, more like someone being hurried to answer questions on the spot. Some associations are stronger and come to mind immediately, some are less "fresh in memory", and then associations can bring false positives and it takes extra time/effort to notice and correct those. It's a common human experience, too. "Yes, those reserved words are 'void', 'var', 'volatile',... wait, 'var' is JS stuff, it's not reserved in C..." etc. Then, of course, humans are learning continuously, and - perhaps more importantly - even if they're not learning, they're reinforcing (or attenuating) existing associations through bringing them up and observing feedback. LLMs can't do that on-line, but that's an engineering limitation, not a theoretical one. I'm not claiming that LLMs are equivalent to humans in general sense. Just that they seem to be implementing the fundamental machinery behind "meaning" and "understanding" in general sense, and the theoretical structure behind it is quite pretty, and looks to me like a solution to a host of philosophical problems around meaning and language.

mjburgess 1y ago
Is this based on analogising LLMs to animal mental capacities, or based on a scientific study of these capacities? ie., is this confirmation bias, or science? One can always find a kind of confirmation bias analysis here, which "saves the appearances", ie., one can always say "take a measurement set of people's mental capacities, given in their linguistic behaviour" and find such behaviours apparent in LLMs. This will always be possible for the obvious reason that LLMs are trained on human linguistic practice. This makes "linguistic measurement" of LLMs especially deceptive. Consider the analogous case of measuring a video games by it's pixels: does it really have a "3d space" ? No. It only appears to. We know that pixel-space measurements of video games are necessarily deceptive, because we constructed them that way, so it is obvious that you cannot "walk into a tv". Yet we did not construct the mechanism of deception in LLMs, making seeing thru the failure of "linguistic measurement" apparently somewhat harder. But I imagine this is just a matter of time -- in particular, when LLM's mechanisms are fully traced, it will be more obvious that their outputs are not generated for the reasons we suppose. That the "reason to linguistic output" mapping we use on people is deceptive as applied to LLMs. Just as a screenshot of a video game is a deceptive measure, whereas a photograph isnt. For a photograph, the reason the mountain is small is because its far away; for a screenshot, it isnt: there is no mountain, it is not far away from the camera, there is no camera. In the case of LLMs we know they cannot mean what they say. We know that if an LLM offers a report on new york it cannot mean what a person who has travelled to new york means. The LLM is drawing on an arrangment, in token space, of tokens placed there by people who have been to new york. This arrangement is like the "rasterization" of a video game: it places pixels as-if there were 3d. You could say, then, that an LLM's response is a kind of rasterization of meaning. And just as with a video game, there are failures, eg., clipping through "solid" objects. LLMs do not genuinely compose concepts, because they have no concpets -- they can only act as if they are composing them, so long as a token-space measurement of composition is available in the weight-space of the model. (And so on...) The failures of LLMs to have these capacities will be apparent after awhile, at the moment we're on the hype rollercoaster, and its not yeet peaked. At the moment, people are still using the "reason-lingusitic" mapping theyve learned from human communication on LLMs, to impart the relevant menetal states they would with people. The boundaries of the failure of this mapping isnt yet clear to everyone. Users don't yet avoid "clipping thru" objects, beause they can't understand what clipping is -- at the moment, many seem to be desperate to say that if a video game object is clipped thru, it must be designed to be hollow. In any case, as i've said in many places in this thread (which you can see from my recent comment history) -- there are a large variety of mental capacities associated with apprehending meaning that LLMs lack. But the process is anti-inductive so it will take quite awhile: for all those who are finding the fragile boundaries ("clipping thru the terrian") new models come out with invisible walls.