5 ms·

The concept is always fully formed (or at least SOME concept is fully formed) before I start typing. Like I read your comment. I had thoughts during that proce

by dools 4y ago

The concept is always fully formed (or at least SOME concept is fully formed) before I start typing. Like I read your comment. I had thoughts during that process, but I read until the end, then I made a decision about what I thought, and that decision contained all the information that I want to convey in my response. Now: when I am constructing my response, I am of course modulating which words might come next that would best fit into my language model to convey that concept, but the concept is an instantaneous understanding. That instantaneous understanding may change during the process of articulation, but not fundamentally -- I think that any changes during the process of articulation could be considered "elaboration" rather than a reconceptualisation. All the language that I just typed was designed to convey a concept that was more or less instantaneously formed and fully complete prior to my typing anything. The way in which I construct the language to articulate that concept may be very similar to the way in which a LLM constructs its responses, but I think the core difference is that original and instantaneous conceptual understanding of what it is trying to articulate.

withinboredom 4y ago
I’m a mix between you two, I think. I know what thought I want to convey, before writing, but not the exact words I want to use. Even as I’m writing, I’m pausing for a few moments to figure out the best word for the thought, determining if it will lead to the next thought, how you might accept it, and then writing it down. And so on. Only moments before I write the word, do I know what that word will be, but I know the complete thought I want to convey. I don’t know if this is important, but my thoughts are not words, so I’m basically just describing them using words.
- dools 4y ago
  Yeah I think that's basically what I'm saying. The concept is an elephant. You look at the elephant and know immediately what it is and all it's attributes, but if you were to describe that elephant to someone you'd have to go through all sorts of decisions about what to say and how to say it and you'd change your mind, and you'd say in one instance that the ears were flat, then you'd say no not really flat in the sense that they're level but very thin and floppy. And the person would ask questions and you'd have to reply to them. The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression is like an instantaneous graphical image of an elephant, that one must then put into words. I think that a LLM doesn't have that. I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language. The concepts are all pre-existing in the same way that all the chess move possibilities pre-exist in the programming of a chess computer. EDIT: In fact even the way I described concepts as "pre-existing" there might be giving it too much credit. Like the concept only arises on the receiving end, from a pool of concepts that were pre-existing but about which the LLM has no "awareness" whatsoever.
  - TeMPOraL 4y ago
    > The concept is an elephant. You look at the elephant and know immediately what it is and all it's attributes, but if you were to describe that elephant to someone you'd have to go through all sorts of decisions (...) The question is, do you know its attributes before you describe them to yourself? To me, at least, an "elephant" feels like a pointer, a handle, a promise - I know it's a thing, but to know more than that, I have to dereference the pointer. And that usually means something that's getting close to stringing words together. Possible confounder: I suffer from aphantasia[0]. I have nearly zero ability to perceive images in my mind, except when dreaming. Maybe it's easier to "know immediately what it [the elephant] is and all it's attributes", if it just pops into your head as a complete image. I, unfortunately, don't experience that. > The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression (...) > I think that a LLM doesn't have that. Crazy idea: what if a concept is... a partial set of coordinates in the latent space? Enough to describe a pretty narrow region. > I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language. My tentative guess is that language will turn out to not be the fundamental part of what makes LLMs so powerful/spectacular. I think the magic is the latent space itself - the high-dimensional constructs where all kinds of associations end up represented as spatial distance. Language is just a serialization format[1] for latent space associations. By being fed lots and lots of text in training, LLMs build up the latent space, piece by piece, until there's enough dimensions to it that it is no longer a simple mapping between words learned from the training data. I think this is how understanding and concepts look like. Also, I think it takes so much input text for LLMs to get to their current performance, because LLMs learn like babies - they soak up everything they can, letting the inputs self-arrange by correlations. We learn to understand what an elephant is by hearing stories about it, seeing it alive, and/or being told facts about it in a structured way (i.e. school). LLMs learn what elephant is by soaking up so much text that (the token representing) the word "elephant" is, in the latent space, close to all the words people used to describe its attributes - which themselves are close to other words with whatever various meanings and aspects we relate to those attributes, etc. EDIT: And connecting the two parts of the comment: LLMs are AI's poster child of aphantasia. If they have concepts, how else could they work with them if not by... articulating them? And when they articulate a concept, what makes us so sure they're just parroting associations, and not truly having a concept? -- [0] - https://en.wikipedia.org/wiki/Aphantasia https://en.wikipedia.org/wiki/Aphantasia. [1] - Quite literally, as communicating via any language happens via ingesting or producing a series of some kind of tokens (letters, words, ideographs, etc.) over time.
TeMPOraL 4y ago
> Like I read your comment. I had thoughts during that process, but I read until the end, then I made a decision about what I thought, and that decision contained all the information that I want to convey in my response. I recognize that mode. I have it to. I had it now; this comment is aiming to transmit a thought that came to me fully formed while reading your response. That said, it's a bit suspect how it popped up as if my mind was doing an associative search. It's also interesting that, however fully formed the thought might have been, it's also small, as if a conceptual token, and I usually have to run further associative lookups to extend it before the comment (or train of thought) feels satisfactory to me. In the previous paragraph, the last two sentences were not part of expressing the fully-formed thought described earlier. I'm not lying to you, it's just I didn't expect to write the second half of the previous paragraph (and this paragraph) when I was writing the first one; they came from conceptual autocomplete. > That instantaneous understanding may change during the process of articulation, but not fundamentally For me, it quite often does. I rely on it happening often. Example cases: - On HN, I post maybe 1/3 of the comments I write. 2/3 of the time, the act of trying to express my thoughts makes me realize that a) I'm wrong, and/or b) it's a bad idea to continue (e.g. I'm just procrastinating, or posting something impolite, or not useful, etc.). Of the 1/3 that survive long enough to be posted, I still delete some - as sometimes the realization from the previous sentence hits me only a couple seconds after posting. - When facing difficult problems, I do a lot of internal monologue. In it, exploring whether certain phrases feel right, and kind of sculpting the chain of thought, is a critical part of the process. - Because I often feel I lack the working memory to keep enough thoughts in flight, I often "talk" with myself via a text file - externalizing my internal monologue. That process also involves sculpting sentences, paragraphs or bullet points until they feel right, so it's arguably still articulation. And the more I think about it, the more this looks to me like the kind of tricks you do to work around the token limit / window of GPT-3 / GPT-4... EDIT: Ultimately, I think we're both doing thinking in similar ways, perhaps with different proportions between modes, and different conception of it. One relevant conceptualization I have that I remembered just now: in conversational situations, I often feel like my subconsciousness is a Markov chain / LLM autocomplete, spitting out phrases, and my consciousness is busy judging them, and choosing whether or not to pass them to the actuators (speech organs, or fingers on the keyboard). The two "models" - the autocomplete and the censor - is something I can clearly see in my mind when I'm processing a problem in a calm manner, or participating in a slow-paced discussion. But I've noticed that the more excited I get (or intoxicated), the weaker the "censor" part is; in the limit (when I'm really excited, angry, or drunk), I feel as if the Markov chain bit is feeding phrases straight to my mouth/fingers, cutting out the middleman of consciousness. (Of course, what I wrote is all "how the algorithm feels from inside", so it's not very objective or reliable...)

by dools 4y ago

withinboredom 4y ago

I’m a mix between you two, I think. I know what thought I want to convey, before writing, but not the exact words I want to use. Even as I’m writing, I’m pausing for a few moments to figure out the best word for the thought, determining if it will lead to the next thought, how you might accept it, and then writing it down. And so on. Only moments before I write the word, do I know what that word will be, but I know the complete thought I want to convey. I don’t know if this is important, but my thoughts are not words, so I’m basically just describing them using words.

dools 4y ago
Yeah I think that's basically what I'm saying. The concept is an elephant. You look at the elephant and know immediately what it is and all it's attributes, but if you were to describe that elephant to someone you'd have to go through all sorts of decisions about what to say and how to say it and you'd change your mind, and you'd say in one instance that the ears were flat, then you'd say no not really flat in the sense that they're level but very thin and floppy. And the person would ask questions and you'd have to reply to them. The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression is like an instantaneous graphical image of an elephant, that one must then put into words. I think that a LLM doesn't have that. I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language. The concepts are all pre-existing in the same way that all the chess move possibilities pre-exist in the programming of a chess computer. EDIT: In fact even the way I described concepts as "pre-existing" there might be giving it too much credit. Like the concept only arises on the receiving end, from a pool of concepts that were pre-existing but about which the LLM has no "awareness" whatsoever.
- TeMPOraL 4y ago
  > The concept is an elephant. You look at the elephant and know immediately what it is and all it's attributes, but if you were to describe that elephant to someone you'd have to go through all sorts of decisions (...) The question is, do you know its attributes before you describe them to yourself? To me, at least, an "elephant" feels like a pointer, a handle, a promise - I know it's a thing, but to know more than that, I have to dereference the pointer. And that usually means something that's getting close to stringing words together. Possible confounder: I suffer from aphantasia[0]. I have nearly zero ability to perceive images in my mind, except when dreaming. Maybe it's easier to "know immediately what it [the elephant] is and all it's attributes", if it just pops into your head as a complete image. I, unfortunately, don't experience that. > The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression (...) > I think that a LLM doesn't have that. Crazy idea: what if a concept is... a partial set of coordinates in the latent space? Enough to describe a pretty narrow region. > I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language. My tentative guess is that language will turn out to not be the fundamental part of what makes LLMs so powerful/spectacular. I think the magic is the latent space itself - the high-dimensional constructs where all kinds of associations end up represented as spatial distance. Language is just a serialization format[1] for latent space associations. By being fed lots and lots of text in training, LLMs build up the latent space, piece by piece, until there's enough dimensions to it that it is no longer a simple mapping between words learned from the training data. I think this is how understanding and concepts look like. Also, I think it takes so much input text for LLMs to get to their current performance, because LLMs learn like babies - they soak up everything they can, letting the inputs self-arrange by correlations. We learn to understand what an elephant is by hearing stories about it, seeing it alive, and/or being told facts about it in a structured way (i.e. school). LLMs learn what elephant is by soaking up so much text that (the token representing) the word "elephant" is, in the latent space, close to all the words people used to describe its attributes - which themselves are close to other words with whatever various meanings and aspects we relate to those attributes, etc. EDIT: And connecting the two parts of the comment: LLMs are AI's poster child of aphantasia. If they have concepts, how else could they work with them if not by... articulating them? And when they articulate a concept, what makes us so sure they're just parroting associations, and not truly having a concept? -- [0] - https://en.wikipedia.org/wiki/Aphantasia https://en.wikipedia.org/wiki/Aphantasia. [1] - Quite literally, as communicating via any language happens via ingesting or producing a series of some kind of tokens (letters, words, ideographs, etc.) over time.

TeMPOraL 4y ago

> Like I read your comment. I had thoughts during that process, but I read until the end, then I made a decision about what I thought, and that decision contained all the information that I want to convey in my response. I recognize that mode. I have it to. I had it now; this comment is aiming to transmit a thought that came to me fully formed while reading your response. That said, it's a bit suspect how it popped up as if my mind was doing an associative search. It's also interesting that, however fully formed the thought might have been, it's also small, as if a conceptual token, and I usually have to run further associative lookups to extend it before the comment (or train of thought) feels satisfactory to me. In the previous paragraph, the last two sentences were not part of expressing the fully-formed thought described earlier. I'm not lying to you, it's just I didn't expect to write the second half of the previous paragraph (and this paragraph) when I was writing the first one; they came from conceptual autocomplete. > That instantaneous understanding may change during the process of articulation, but not fundamentally For me, it quite often does. I rely on it happening often. Example cases: - On HN, I post maybe 1/3 of the comments I write. 2/3 of the time, the act of trying to express my thoughts makes me realize that a) I'm wrong, and/or b) it's a bad idea to continue (e.g. I'm just procrastinating, or posting something impolite, or not useful, etc.). Of the 1/3 that survive long enough to be posted, I still delete some - as sometimes the realization from the previous sentence hits me only a couple seconds after posting. - When facing difficult problems, I do a lot of internal monologue. In it, exploring whether certain phrases feel right, and kind of sculpting the chain of thought, is a critical part of the process. - Because I often feel I lack the working memory to keep enough thoughts in flight, I often "talk" with myself via a text file - externalizing my internal monologue. That process also involves sculpting sentences, paragraphs or bullet points until they feel right, so it's arguably still articulation. And the more I think about it, the more this looks to me like the kind of tricks you do to work around the token limit / window of GPT-3 / GPT-4... EDIT: Ultimately, I think we're both doing thinking in similar ways, perhaps with different proportions between modes, and different conception of it. One relevant conceptualization I have that I remembered just now: in conversational situations, I often feel like my subconsciousness is a Markov chain / LLM autocomplete, spitting out phrases, and my consciousness is busy judging them, and choosing whether or not to pass them to the actuators (speech organs, or fingers on the keyboard). The two "models" - the autocomplete and the censor - is something I can clearly see in my mind when I'm processing a problem in a calm manner, or participating in a slow-paced discussion. But I've noticed that the more excited I get (or intoxicated), the weaker the "censor" part is; in the limit (when I'm really excited, angry, or drunk), I feel as if the Markov chain bit is feeding phrases straight to my mouth/fingers, cutting out the middleman of consciousness. (Of course, what I wrote is all "how the algorithm feels from inside", so it's not very objective or reliable...)