6 ms·

Words Are a Byproduct of Consciousness. For LLMs, It's Backwards

94 points by DevarshRanpara 3mo ago

DevarshRanpara 3mo ago
Stop for one second and ask yourself a simple question. Where do your words come from? When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it.
- pigeonwarz 3mo ago
  My favourite way of thinking about this is by going back to the End Poem at the end of Minecraft. The fact that words are an "interface", "very flexible" and "less terrifying than staring at the reality beyond the screen". Maybe that reality is the true understanding of the neurobiology that defines our thoughts. To be reminded that the magic of experience can be reduced to neuronic hallucination is, frankly, horrifying. Maybe we dislike LLMs because we see their vectors and numbers as crude reminders of our own intellectual banality. Or maybe I'm just feeling hungry and should really go eat dinner.
- HelloUsername 3mo ago
  There's plenty of people who speak before they think
- yieldcrv 3mo ago
  We’re stochastic parrots most of the time, with a varying ability to reason but opting for pattern conformity usually
  - danielmarkbruce 3mo ago
    This is getting voted down because people just hate the idea that it's true.
    - bluefirebrand 3mo ago
      No it's down voted because we didn't stochastic parrot ourselves into landing on the moon and building the internet and modern highways and skyscrapers Dismissing humanity that was is absurd
      - yieldcrv 3mo ago
        LLMs with reality interfaces (aka a body) can do those things, add world models and they'll do all those things better in our world, the first time and better than us. because the failures would have happened inside the world model
        bluefirebrand 3mo ago
        The point is that they cannot do that to themselves. They cannot decide to add world models and build exoskeletons and make that happen without our intervention
        yieldcrv 3mo ago
        Stochastically parroting a moving goal post that wasn’t presented as a point at all earlier, imo I can play devils advocate too. The first agents running LLMs and other models that get a capable exoskeleton can do the same for the other LLMs that will be tasked to execute the guidance, agentically
      - danielmarkbruce 3mo ago
        Yeah, we did. It took humans 100,000 or so years to land on the moon - we aren't so clever.
  - hallway_monitor 3mo ago
    What is reasoning except applying another pattern on top of existing thought? Personally it seems like I and everyone else is simply pattern matching, albeit at a higher level than current LLMs. There is no difference in the process as far as I can tell, just different inputs.
  - enaaem 3mo ago
    In some meditation practices, the task is to observe the stochastic parrot in your head. You will soon realise that you cannot control this parrot. If you cannot control your thoughts, could it be that you are not your thoughts? And if you are not your thoughts--the word stream in your head--what are you then?
- next_xibalba 3mo ago
  > Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it There are enough neuroscience experiments demonstrating how we create post hoc explanations of our actions that I wouldn't trust this intuition prima facie.
  - bluefirebrand 3mo ago
    The fact that we're even capable of doing a study like that and introspecting is probably a good indication of our cognitive abilities as humans Consider someone designing a skyscraper. They're thinking about material strengths and such while they draft it, they aren't just rationalizing their choice later.
  - jcims 3mo ago
    I've noticed that more and more of my mental attention while speaking is observing the words that come out of my mouth rather than 'generating' them. I split time between rough pathfinding in my mind to fit a conceptual framework then kind of sit and listen as it comes out to think about how they sound. If I had to guess this happens on a 1-2 sentence chunk at a time.
- mattas 3mo ago
  Word is just the skin is the load-bearing idea. The real unlock, if you will.
- vitamark 3mo ago
  "For an LLM, it is exactly the opposite". Is it really? LLMs don't have words inside, for the most part they operate by applying transformations to a vector that does not contain any words at all. Words that come out of an LLM are just what sampler gives us by looking at the vector that is the result of those transformations. Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really. Sure, rerunning that whole process for each token might not be the best solution, although that's an open question too. But saying that LLMs operate on words first is too big of an oversimplification
  - rcxdude 3mo ago
    >Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really. It is, but it has been explored in various forms. Anthropic have some interesting papers where they can map out concepts to some extent as they exist inside the vectors of the LLM, and they can see e.g. that vectors relating to some concept can appear there some iterations before the LLM emits the tokens that spell it out. (though also, quite frankly, this shouldn't be all that surprising. If they were just picking the next token with no representation of what might follow it the result would be more like the 'one-word-at-a-time' party game than coherent text. They used poetry with a rhyming scheme as an example because it's something where you need to have some idea of what might rhyme with the line before and aim at it at the start of the next line in order to have any degree of success).
- epihelix 3mo ago
  I don't know about you, but I think with words all the time when reasoning through complex ideas. Where do you think the phrase "thinking out loud" comes from?
OhioMan2943 3mo ago
Meditative practices support this. Anyone who's spent even a moderate amount of time with meditation knows we are not our thoughts. If you don't believe me, learn how to meditate, meditate for literally just 10 minutes, and come back to me.
- fedeb95 3mo ago
  this doesn't imply that words are a byproduct of consciousness.
Diogenesian 3mo ago
St Augustine had it right 1500 years ago: humans learn words opportunistically according to their desires and problems and existing deep understanding of the world, and (critically) humans really can't learn language without being natively fluent in great ape facial expressions, gestures, grunts, etc. Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders. [https://faculty.georgetown.edu/jod/augustine/conf.pdf https://faculty.georgetown.edu/jod/augustine/conf.pdf]
michael1999 3mo ago
It is not all obvious that consciousness precedes language. Hellen Keller provides a vivid description of her pre-verbal mental state as something less than reflective conscious. > Before my teacher came to me, I did not know that I am. I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect. https://scentofdawn.blogspot.com/2011/07/before-soul-dawn-helen-keller-on-her.html https://scentofdawn.blogspot.com/2011/07/before-soul-dawn-he...
- etrautmann 3mo ago
  I think this is different if you have vision though. I struggle to understand how someone can look at intelligent animals that don’t have language and not intuitively assume that they’re conscious. Yes this is a tricky problem without an answer but IMO the elements of consciousness can exist without language if visual abstractions are present.
  - sigbottle 3mo ago
    You could redefine language to include mental abstractions and maybe it wouldn't be so weird. Language is a shared medium. Maybe an intelligence is capable of building abstractions in their head without the need for say, a symbol system to hopefully communicate with species similar to itself. You may argue that the agent needs persistent memory. Well, so what? If the world has stable regularities from the perspective of the agent - if the sun rises every day, and language is associating certain visuals with certain thoughts (it's not when viewed in its original social definition - but conduct the thought experiment with sociality removed), who's to say that you can't form a kind of language around that? Of course, this is all armchair, and I'm not trying to separate out what "real language" is or whatever, just thought experiments. But the original poster has a good point though, that's been proven in papers, that the tokens an LLM spits out, themselves, attract them to certain distributions, and at least a decent amount of the tokens in CoT are there solely to bring itself to the right distribution.
  - devmor 3mo ago
fedeb95 3mo ago
That words are a byproduct of consciousness is a bold claim and requires some justification. I'm not implying it's backwards also for humans, but that the statement might not be true, and something else could be. Also, defining consciousness is hard. For instance, you can dream words.
- cwmoore 3mo ago
  Eh? What else could communication be there for?
  - novacrazy 3mo ago
    Survival.
- t43562 3mo ago
  Have you ever had impressions about something complicated, e.g. someone's personality which you cannot put into words? I'm thinking of my kid - her personality when she was tiny. I could see what it was but not explain it. Hence I think ideas are unquestionably first and words are the inadequate and inaccurate description of thoughts that exist without them.
  - fedeb95 3mo ago
    That may be true without implying that words are a always a byproduct of consciousness. You can dream words.
  - IAmGraydon 3mo ago
    I agree, and I think that's a good example. Sometimes we can hold a concept in our minds of something and cannot find the words to attach to it. The word we eventually give that concept when it's human, as in your example, is a name. This is because the concept of a personality is too complex to describe with other, preexisting words. Still, even though I think the author is right about consciousness/concepts preceding language, I think he's wrong about the other part. The statistical models that underlie AI very much remind me of what we call a human concept. It's a statistical model of that thing, person, or idea. However, we humans seem to have this ability to create new concepts that are wildly different than our training data (what we have experienced in life), and we call this creativity. It's missing from AI for some reason. That is the part that makes me question if AI will ever match humans.
jacobr1 3mo ago
LLMs as clearly doing more than just modeling words. In order to predict word placement, they need to build some kind of model, their latent space, of the types of things they are able to predict. Not really full world models yet - but they have decent "blog space" or "github project" models. And you can see this with multimodality, or non-text modality modals such as images and audio. They map from their latent spaces to the outputs. The fact that multi-modal systems can share the interior layers shows some kind of internal representation is created.
- adamzwasserman 3mo ago
  Or.. the LLMs are just "telescope" sufficiently powerful to detect the aligned internal representation that already exists. A more powerful telescope does not "create" new galaxies.
  - jvanderbot 3mo ago
    Love the analogy. I a lot of people had their mind blown by vector space representation of words. The idea that Female + King averaged out around "Queen". Words might foundational to concepts or just really-well-designed ways to transmit them.
    - sometimelurker 3mo ago
      You have the word 'bot' in your name and judging by your comment history love using em-dashes. Are you a machine or not?
      - jvanderbot 3mo ago
        I doubt I used emdashes, but I do us "-". I dont have emdash unicode bound to my keyboard yet. (edit: Hyphenation is not emdash, of course) you can view my profile details freely and decide for yourself, and a quick look might even explain the "bot" suffix.
        3mo ago
krunck 3mo ago
For LLMs there is no consciousness.
- visarga 3mo ago
  We might quibble what to call it, but they use language productively.
2snakes 3mo ago
I do not think the written description of a dog is the same as a dog itself…
- t43562 3mo ago
  Nothing is the same as the dog but we probably store a lot of sensory information, emotions and images relating to dogs in our heads.
- tsss 3mo ago
  Maybe it is. If you think about it hard, you will find that you cannot separate the idea of dog from the word dog. The object dog can only be understood as a perception linked to that idea.
  - orthoxerox 3mo ago
    "Ceci n'est pas un chien"
throwaway260124 3mo ago
"In the beginning was the Word, and the Word was with God, and the Word was God." And Islam the first verse was ‘read’ The first time I heard about emergent behavior in llms I thought about that. Maybe thought is too much because I don’t have anything coherent in mind but it feels so captivating like the premise of sci-fi story.
- micoti 3mo ago
  Word in the Bible is not meaning the actual word (like sematic array of characters) but more like greek word logos
  - thewebguyd 3mo ago
    Correct, in the Koine Greek, logos is the word that's actually used. Logos is a bit deeper than words, and is more akin to a universal principle of order that governs the cosmos. "In the beginning was the logos" is like saying in the beginning were the laws of physics, and God is the laws of physics and the laws of physics are God.[1] [1] This also entirely depends on which framework you are translating from. This would be accurate from the stoic interpretation of the logos but differs from a more mystical Johannine interpretation, in which the logos is outside the physical universe, not just operating within it. Logos is like the source code of the universe.
  - kaonwarb 3mo ago
    It's a hard word to translate. https://en.wikipedia.org/wiki/Logos https://en.wikipedia.org/wiki/Logos has a good overview.
- _doctor_love 3mo ago
  Some folks interpret that sentence in a mystical way - the first word was Om, from which all the universe was created.
chermi 3mo ago
Can you please point me to the proof of the first claim?
talkingtab 3mo ago
Words limit our understanding of words because they are intrinsic to a particular kind of understanding. yes? no? In "Notes on the Synthesis of Form", Christopher Alexander talks about how we can know something is wrong even when we do not know how to make something right. He was talking about this I think =z== as opposed to +|+|+ or +|+|+| I happened upon the word "entanglement" and it seems an interesting alternative to the order that is inherent in words. Our language is all about Actor-verb-object. Entanglement provides a fundamentally different concept. I cannot say "I drove my car on the road" with entanglement. Or at least with entanglement I can say something equally valid like "the road moved my car with me inside." And entanglement works for the + and | things, at least for me. There is some kind of entanglement (degree) that creates a gestalt. At least that is the best I can come up with.
cowlby 3mo ago
Isn't it interesting then how much of our intelligence is captured by our words/language? I would've thought to create AI you needed to replicate neurons and synapses and learning. So it's still amazing to me that statistical modelling of our words creates a Claude Opus.
- ordersofmag 3mo ago
  Perhaps LLM's do replicate the important features of "neurons and synapses and learning" and perhaps a "statistical modelling of our words" is pretty much what our brains are doing? Most of the counter-arguments I've seen boil down to "but humans are special" which I'm not sure I find compelling. Not saying LLM's are conscious. Just that much of our amazement about their behavior seems to say more about us realizing the things we can do are not magic, rather than them being so amazing.
speak_plainly 3mo ago
The author is making a big mistake by making a normative claim about 'true' intelligence and consciousness. Is there a correct way to cognize? And although he feels cognizing works in a very specific direction, his brain is basically doing a very similar guessing game on a deep level with training of pathways that started at birth. Basing the argument in a feeling about consciousness is not convincing. The author thinks they are describing a unique human magic, forgetting Augustine's insight that human thought doesn't precede the word, but is brought into consciousness by it.
tsss 3mo ago
No, I don't think so. Language is intimately intertwined with our understanding of the world. Many people make the mistake to think that languages are about the spelling or the sound or maybe the grammar. But in reality they are about how words are defined only on relation to each other. A symbol is defined by what it is not. This relationship between symbols structures how we think about the world and how we fantasize and desire. You can perhaps have a consciousness without words, but not without language.
- deleted 3mo ago
  [deleted]
Festro 3mo ago
"At its core, it is a big pile of words that predicts the next word, using some maths the computers figured out." "An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. The words are everything." This is just flat out wrong. Words are used as training data, to build a system of vector embeddings. The LLM contains no words. That was the training data long discarded. Vector embeddings are groupings of meanings derived from the relationship between the words in its training data. This entire system is modelled after human neural mechanisms in a way that machines can emulate. "But your brain works the other way around. First there is a concept, a feeling, an image, and then the words come out to describe it. (At least, this is how I feel my own brain working.) For us, words are the byproduct of consciousness." It's working the same way around (you are not saying Ai has consciousness that is derived from words!). And you make a huge jump from the concept that 'words come from concepts' to 'words are the byproduct of consciousness'. Because 'concepts' are not equal to 'consciousness'. In fact you do not define consciousness at all, making it hard to determine what argument you are actually making at all. You seem to think humans have this trait of 'consciousness' but can't explain or evidence it beyond 'feeling it'. Consciousness it defined by many as the ability to experience events and process thoughts or qualia. It's hard to test this, and we can see why that is with an AI - it may claim to be conscious, or even claim not to be, but how can we trust either answer? Philosophers aren't even sure to trust another human who claims to be conscious, and we struggle constantly to determine at what point creatures the animal kingdom are conscious or not. Cats? Lobsters? Snails? or even across the plants? Before you can refute the consciousness of AI, first establish the consciousness of humans. A better question to ask is what does the uncanny emergent ability of an AI to mimic a human say about consciousness?
- encyclopedism 3mo ago
  Defining consciousness is one of the most difficult outstanding problems in the whole of science please see: https://en.wikipedia.org/wiki/Hard_problem_of_consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness An approach that might shed some light is instead to define what consciousness ISN'T. Naively let us say consciousness is NOT a large list of weights (i.e. an LLM). The uncanny emergent ability depends entirely on training data. A mathematical model is used to match output against training data (via loss functions etc). The training data contains all the human ingenuity, logic, rational, patterns and features. Try giving an LLM model the alphabet ALONE and see what it comes up with?
skerit 3mo ago
But LLMs aren't just shuffling words around. Words (tokens) are the "human interface" at the edges. Text input gets embedded, then there's this huge latent-space computation in the middle, and only at the tail end does that get converted back into word/token probabilities. So just saying that "words are the source" isn't entirely correct and feels misleading.
visarga 3mo ago
> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? Let's take a feral human, do they achieve the same with language powered humans? No, obviously. It means language does something we can't replace with "consciousness". Maybe something life preserving, putting consciousness downstream of language use. But more generally - how can 8B humans make a living on this planet? Not without language, that is for sure! We long passed the stage where we could exist without language at our consumption rate. We can't even exist without math, population collapse would be the outcome. Minds are downstream from language & math. For example, is music downstream of piano or piano downstream of music? I think you can't cut this cleanly. They developed in relation to the other, recursion between tool and art form is already old.
adamzwasserman 3mo ago
As others have remarked: this is an unsubstantiated claim. I am actually using LLMs to perform controlled experiments to determine if the opposite is true, and that consciousness is a byproduct of language as Jaynes might have agreed.
fooker 3mo ago
We can't define consciousness well enough to make claims like this.
degecko 3mo ago
It’s funny because, ever since Norbert Wiener’s Cybernetics was published in 1948, we’ve been trying to replicate how the human brain works. In that sense, we’ve been building increasingly powerful machines that mimic human thought. And now, the most interesting thing is something that turns that idea on its head: LLMs think in reverse.
sturza 3mo ago
“The relationship of thought to word is not a thing but a process, a movement from thought to word and from word to thought.” Vygotsky
sourdecor 3mo ago
Reminds me of this from "Gödel, Escher, Bach: an Eternal Golden Braid": "Let me rephrase these last couple of sentences without using the slightly technical term "isomorphism". When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with some degree of meaning - indeed, such tracking or mirroring is no less and no more than what meaning is. Depending on how complex and subtle and reliable the tracking is, different degrees of meaningfulness arise." - P-3 EDIT: I initially wrote "G.E.B." instead of "Gödel, Escher, Bach: an Eternal Golden Braid" which was too cryptic.
- PaulDavisThe1st 3mo ago
  While I know what Douglas Hofstadter's book "Godel Escher Bach - an eternal golden braid" is, it might be worth revisiting your apparent assumption that more or less everybody on HN knows about a book from 47 years ago (regardless of its significance to those of us who have encountered it already).
  - kazinator 3mo ago
    LOL, I happen to have a "Hofstadter_GEB.pdf" open on my desktop; been there for months.
Wowfunhappy 3mo ago
> But one thing scares me a little. What if LLMs slowly become worse? Think about it. Everything written before 2017 was made by humans. After 2017, LLMs started filling the open web with their own content. And now that same content is being fed back to train the next LLMs. An LLM can give you perfect grammar and a rich vocabulary, but it can quietly lose the real context. Right now the share of AI content is small. But what happens when it keeps growing? Have the last-generation LLM pre-screen new training data for the next generation LLM.
pigeonwarz 3mo ago
I'm going to go on a limb here and challenge this assumption. As a parallel thread to the point put forward by michael1999, can we not argue words manifest as the result of the maths performed within LLMs, maths that, for all intents and purposes, is their world? Because the internal maths of how LLMs work is likely to be significantly cruder than the 'real' world that defines consciousness, perhaps we are closer to the ghost in the machine than we would like to admit.
awongh 3mo ago
> But your brain works the other way around. First there is a concept, a feeling, an image, and then the words come out to describe it. After this I can't take the essay seriously- this sort of blanket statement about the one true hierarchy of consciousness and knowledge is BS. While it seems to have been disproven that words in other languages cause people to speak and think differently, that also doesn't mean that words don't have any effect on the way we think, or that the concept of a word always has to come before the word itself. The reason why we experience the uncanny valley of LLMs is because they don't represent a true consciousness, BUT it's also clear that the architecture represents certain qualities of consciousness- as the models have scaled we can see that it has some other non-word related understanding. The evidence points to consciousness as a set of interlocking systems- attention, long term memory, short term memory, emotions, etc.
fasteddie31003 3mo ago
Maybe consciousness just does not exist. Could it be that humans invented it to feel superior to animals, but it does not really exist? I wonder if we will think of the word "consciousness" in the same way we think of the word "aether" in the future.
- brotchie 3mo ago
  If consciousness didn’t exist, you wouldn’t exist! you’d have no interior experience.
  - Dependance 3mo ago
    How do you know they have one ?
    - kelseyfrog 3mo ago
      I lack a subjective experience and have no interiority. Watch this. They won't even believe you if report your own internal state. They'll even claim they have epistemic authority over your own interiority while not granting you the same authority over their internal state.
    - wolvesechoes 3mo ago
      Wittgenstein dealt with that.
- DanHulton 3mo ago
  There’s certainly a debate as to whether animals are conscious or not, and if we’ve just been engaging in a kind of human chauvinism all this time, but I mean consciousness definitely exists. There’s an “I” in here that perceives, and it’s reasonable to extrapolate that I’m not particularly special, that others that have the same parts as me have the same properties (consciousness).
- GuB-42 3mo ago
  Consciousness exists by definition. It is a subjective experience and you are experiencing it. Or more precisely I am experiencing it, but since you are, I think, a human just like me, with a brain and biology very similar to mine, I assume that you must experience consciousness too. And that is the big question, and one that science may never have an answer to. What is conscious and what is not? The only consciousness I know exists is mine, the rest is just guessing based on similarities. Other humans: almost certainly conscious, higher intelligence life forms: probably conscious, plants: probably not conscious, rocks: almost certainly not conscious, LLMs: ??? LLMs are unusual in that they have a very human-like behavior, more than anything non human, which would make them a good candidate for having consciousness, but on a material level, they are more like rocks, which point them as not being conscious.
smallerfish 3mo ago
> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it. In (at least once school of) Tibetan Buddhist meditation you observe the part of the mind that is producing ideas, and then suppress it in order to explore deeper layers of consciousness. The "lights on" "you" that is observing is _not_ the idea producing part of the mind, which to me at least seems to be more mechanical. I do agree with the author of course that a concept isn't necessarily wrapped with language from the start, and can be a skin; but equally it is possible to think in a verbal mode.
neovive 3mo ago
> For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. Recently, I've been thinking a lot about context and whether an LLM actually creates a true representation of a concept within its vectors. And for our current usage of LLMs, does it even matter? As models become so large, will the difference even be perceivable to us?
Autious 3mo ago
Steve Jobs didn't invent the PC. It's reductionist and offensive to single him out and thanking him for that. Computers didn't "figure out the maths". People did the math, the computers did the calculations. Human knowledge is nowhere complete, it's silly to think so, and by extension it's silly to think the LLM's have anything near a complete knowledge set. They don't even have access to all of current human knowledge, much less so future knowledge. A lot of text and information is locked away from the public internet and the internet altogether. Not all types of useful ideas for applications have been invented, that's ridiculous, there are plenty of ideas that people haven't had yet. There are however a massive amount of shitty copies of the same shitty ideas.
ohthehugemanate 3mo ago
This person is just making stuff up that feels true to them. Caveat lector. There are reams of grown up scientific research and philosophy on the relationship between language, thought, neurobiology, and inference. It's nice to hear how one person feels they think, but there is no weight to it if they don't know page one of the syllabus.
- deleted 3mo ago
  [deleted]
- throw310822 3mo ago
  > just making stuff up that feels true to them Hallucinating it?
- triprjt 3mo ago
  would second you on this.
csomar 3mo ago
> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. Most people don't actively think when they are talking or writing most of the time. Only during period of thinking that we do think. Most of the time, we just talk words. There is evidence is that we just observe ourselves talking and that the thinking itself is not conscious but it's still a theory. > Now ask the same question about an LLM. For an LLM, it is exactly the opposite. No one has any idea how an LLM come to correct words. LLMs produce numbers (tokens) that end up to coherent words. > So what is an LLM, really? At its core, it is a big pile of words that predicts the next word, using some maths the computers figured out. No, not really. > The words are everything. For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. The author provides no proof for this and there are very good reasons to think that LLMs have inherent thinking. The rest of the article was just some random thoughts the OP jotted down? I just don't see their coherence to the beginning of the article.
IAmGraydon 3mo ago
>An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. The words are everything. For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. I don't think that's right. What makes LLMs (and other forms of "AI") work is the very fact that they hold a statistical model underneath and put words on top of it. That statistical model, at least to me, appears to be somewhat akin to what a human would call a "concept". Like a stable diffusion model has a statistical model of what a "chair" is. Not one exact chair, but all chairs. They all have backs, 4 legs, a flat area to sit on. It can then carve an image of a chair out of noise using this concept. Take a music model like Suno as another example. If you tell it to make a rock song, it has a statistical idea of what this would looks like. The harmonic progression, the tempo, the types of instruments used. It then attaches sounds to this statistical model and you get a "song". LLMs also appear to do this. If you ask it for a paragraph of a specific flavor of prose, it has that as a statistical model, and it attaches words to it. I am not a big believer in LLMs delivering on even half the hype they've generated. However, this very idea of human concepts vs AI's statistical models is the one thing makes me wonder, sometimes, if they're on the right track, even if we are a long way off from AGI. It's kind of funny that the author of the above article landed on the opposite conclusion.
lebuffon 3mo ago
I look at culture, under which I put language as a component, as the O/S of the human brain. So yes a brain with no O/S can think but it probably will never replicate, in one life time, the accumulated thinking skills of a brain given a good O/S. How many of us would independently reinvent language with specific tenses, the diatonic scale, the number zero, perspective drawing, geometry, calculus, atomic theory, quantum mechanics etc. if left on our own from childhood? So to me, the question becomes, can human language encode the sum of all human thought? The answer might be no. But it might be good enough to get close.
InfiniteRand 3mo ago
I think small children learning language learn in a way similar to LLM's, just saying words that seem appropriate to the situation without any understanding of the meaning. But a key difference in my mind, is they use the reactions to this text prediction to help them construct a picture of the world, which then can interact with their thinking of what words to say.
DevarshRanpara 3mo ago
I want to add one thing, Please think about how animals perceive world? They don't have language, but they are conscious for sure.
sesteel 3mo ago
I am pretty sure human level intelligence is emergent from language use. The more complex the language the higher the intelligence since language is itself a tool used to not only to communicate, but also classify the human experience. Without language, we are fairly primitive relying on instincts and more primitive communication. Feral children are an example of what a base state might be.
sometimelurker 3mo ago
As I see it, LLMs will/do develop 'introspectiveness' if it is selected for during RL. If its not delibrativly selected for (this is probably done because it helps with alignment; if the machine can align itself that's great) its probably a generally useful skill to have when doing normal tasks, https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection
- jvanderbot 3mo ago
  Their introspectiveness doesn't strike me as indicative of anything, since we might be just reinforcing a "plausible word salad generator for input context that leads to better solutions" or just a "post hoc rationalizer", both of which may or may not be the key mechanism behind verbal consciousness.
cathalharte 3mo ago
I share your positivity, however I don’t think figuring out how to market through the noise is the important part. LLMs don’t need to think, they just need to be a more efficient data retrieval system than what already exists (and what has already been sold to the highest bidder). With LLMs, I’ve find I’ve started reaching the corners of the internet, where the true lovers of knowledge are doing the work they have always been doing. Quality does eventually speak for itself, at least once we’re able to strip away the marketing (you mentioned Apple, do you believe they are truly a cut above, or is it just hype?) If LLMs can filter the noise, build it and they will come might return as a valid way to be.
deleted 3mo ago
[deleted]
weare138 3mo ago
https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse
zkmon 3mo ago
> First there is a concept, a feeling, an image, and then the words come out to describe it. Nope. Concepts, feelings and ideas do not exist independent of expression (words, sounds, responses). Humans have the same process as LlMs have. And that's precisely why LMs are able to succeed. Similar to how camera works based on the working of an eye, or how aeroplane works based on a birds wing. If you still believe that concepts exist independently, imagine a creature that can't express, make sounds, or respond with actions. What exactly is a "concept or feeling" to such creature? Do trees have concepts and feelings? Infact trees could, because they can respond to stimulants. If you can't describe or perceive a thing, that thing doesn't exist in your world. It's that simple. What you are describing as "translating thoughts into words" is actually, refining your expression. The expression already exists at the same time as the thought.
- orthoxerox 3mo ago
  My son is insufficiently verbal (he's of the kind that communicates with quotes, but really struggles to express or understand new sentences), and I am quite sure he has concepts, feelings and ideas.
  - zkmon 3mo ago
    Ofcourse he has. Because he does express in his own way. He is not struggling to express, by the way. It's your perception that others are struggling. It's like a rich man who thinks that everyone less richer than him are struggling. But your measure, all animals are struggling to express. No they aren't. You are defining expression from your own, specific standpoint. The concepts one has, do not exceed their expression abilities. So nobody is struggling.
sgarrity 3mo ago
Forgive my quoting my own blog post: Writing is a shadow of thought. The better the writing, the more clearly the shadow represents the shape of the original thought. Even the best writing can never perfectly capture the original thought. Writing is one of the best tools we have to share thoughts across space and time.
pizza234 3mo ago
> Now come back to my first question. An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. No!! This is misinformation. An LLM predicts the next words (tokens) based on the words before it *and* on the internal representations it has learned from training. Processing language can lead to internal representations that capture patterns and relationships; LLMs can exhibit emergent abilities, including reasoning-like (stress on "-like") behavior, even when they weren't explicitly trained for that. Making reductionist claims about LLMs, based on next-word prediction, is similar to making such claims about biology, based on amino acids.
VeninVidiaVicii 3mo ago
Language is just a shortcut to logic. LLMs don’t seem to have the capacity to logic, they just pattern match. LLMs seem conscious because they are pattern matching against all of human written cognition.
avensec 3mo ago
I find the comment section interesting. There is so much discussion about the dichotomy of consciousness, in support and opposition as discussed by the author. Some of the "rights" and "wrongs" feel way more combative than, simply noticing where the author is currently at. I see the temporal disconnected strong here. The author is young-ish, and appears to be exploring ideas of many that have come before him. And, that he'll likely continue to explore on his journey. You can see his thoughts and see echos of long debated thoughts about consciousness, language, meaning, bottom-Up/top-down processing, not being our thoughts, etc.. Thanks to those who are leaving their knowledge, book references, quotes, and etc., to help myself and others who are somewhere along the continuum of understanding ourselves more thoroughly!
- DevarshRanpara 3mo ago
  You are 100% right, I am still exploring (and also learning from all these comments). I have just started to explore these things now. Thank you for understanding author's mindset.
waffletower 3mo ago
"An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath." At most, and I will argue for less, that is half the story. This egregious over-simplification doesn't account for training and the massive dimensional space of thought it forms. Given that frontier models likely have trillions of parameters, the combinatorial space of ideas within approaches the infinite -- hardly zero.
tyzoid 3mo ago
I have to disagree with the fundamental premise here. LLMs do indeed have an internal "concept" mapping - the embedding layer. There's some fun demos on youtube[1] where you can manipulate embedding vectors (such as king - man + woman = queen), and these manipulations are done entirely in the embedding/concept space. [1] https://youtu.be/gQddtTdmG_8 https://youtu.be/gQddtTdmG_8 (around 14:00)
monkpit 3mo ago
Words are a representation of an underlying something; LLMs do not deal with words, they deal with tokens and embeddings. There’s a transformation layer between words and inference and back.
anonymousiam 3mo ago
The author had a perfect opportunity to decimate the whole concept of using LLMs as "artificial intelligence", but he shifts gears in the final six paragraphs and doesn't move in for the kill.
JKCalhoun 3mo ago
Yeah, disagree with many of the points this article takes for granted. Do I have a thought and the words follow? I'm not sure that is the case. My sense is that I have a glimmer of an idea when I begin to speak—but not fully fleshed out as yet. As I start to assemble words and speak them, the idea begins to coalesce—the words providing something like a scaffolding for my thinking. (Of course, I might not be the most reliable narrator: analyzing my own thoughts.)
IraFrost 3mo ago
Then give copy.lu a try — it’s Lu.
grantcas 3mo ago
[dead]
singularity2001 3mo ago
Words are also not necessary for LLMs, hear me out: they could be trained mostly on video showing how to solve problems and would work almost the same (in principle). That's because both word tokens and video will end up in the same latent space which is where real consciousness and thinking takes place.
- DevarshRanpara 3mo ago
  Humans figured out how to hunt animals by making spears from stone by their own, it was not programmed, and that is the main thing which makes us different, humans can go against their instinct, why they do it? Because of consciousness. Can LLM (Or any Model) produce something without training it on data? (Not saying you are wrong, just pasting my thoughts, your thinking which says video can also train a model is an interesting way to look!)

Stop for one second and ask yourself a simple question. Where do your words come from? When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it.

pigeonwarz 3mo ago
My favourite way of thinking about this is by going back to the End Poem at the end of Minecraft. The fact that words are an "interface", "very flexible" and "less terrifying than staring at the reality beyond the screen". Maybe that reality is the true understanding of the neurobiology that defines our thoughts. To be reminded that the magic of experience can be reduced to neuronic hallucination is, frankly, horrifying. Maybe we dislike LLMs because we see their vectors and numbers as crude reminders of our own intellectual banality. Or maybe I'm just feeling hungry and should really go eat dinner.
HelloUsername 3mo ago
There's plenty of people who speak before they think
yieldcrv 3mo ago
We’re stochastic parrots most of the time, with a varying ability to reason but opting for pattern conformity usually
- danielmarkbruce 3mo ago
  This is getting voted down because people just hate the idea that it's true.
  - bluefirebrand 3mo ago
    No it's down voted because we didn't stochastic parrot ourselves into landing on the moon and building the internet and modern highways and skyscrapers Dismissing humanity that was is absurd
    - yieldcrv 3mo ago
      LLMs with reality interfaces (aka a body) can do those things, add world models and they'll do all those things better in our world, the first time and better than us. because the failures would have happened inside the world model
      - bluefirebrand 3mo ago
        The point is that they cannot do that to themselves. They cannot decide to add world models and build exoskeletons and make that happen without our intervention
        yieldcrv 3mo ago
        Stochastically parroting a moving goal post that wasn’t presented as a point at all earlier, imo I can play devils advocate too. The first agents running LLMs and other models that get a capable exoskeleton can do the same for the other LLMs that will be tasked to execute the guidance, agentically
    - danielmarkbruce 3mo ago
      Yeah, we did. It took humans 100,000 or so years to land on the moon - we aren't so clever.
- hallway_monitor 3mo ago
  What is reasoning except applying another pattern on top of existing thought? Personally it seems like I and everyone else is simply pattern matching, albeit at a higher level than current LLMs. There is no difference in the process as far as I can tell, just different inputs.
- enaaem 3mo ago
  In some meditation practices, the task is to observe the stochastic parrot in your head. You will soon realise that you cannot control this parrot. If you cannot control your thoughts, could it be that you are not your thoughts? And if you are not your thoughts--the word stream in your head--what are you then?
next_xibalba 3mo ago
> Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it There are enough neuroscience experiments demonstrating how we create post hoc explanations of our actions that I wouldn't trust this intuition prima facie.
- bluefirebrand 3mo ago
  The fact that we're even capable of doing a study like that and introspecting is probably a good indication of our cognitive abilities as humans Consider someone designing a skyscraper. They're thinking about material strengths and such while they draft it, they aren't just rationalizing their choice later.
- jcims 3mo ago
  I've noticed that more and more of my mental attention while speaking is observing the words that come out of my mouth rather than 'generating' them. I split time between rough pathfinding in my mind to fit a conceptual framework then kind of sit and listen as it comes out to think about how they sound. If I had to guess this happens on a 1-2 sentence chunk at a time.
mattas 3mo ago
Word is just the skin is the load-bearing idea. The real unlock, if you will.
vitamark 3mo ago
"For an LLM, it is exactly the opposite". Is it really? LLMs don't have words inside, for the most part they operate by applying transformations to a vector that does not contain any words at all. Words that come out of an LLM are just what sampler gives us by looking at the vector that is the result of those transformations. Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really. Sure, rerunning that whole process for each token might not be the best solution, although that's an open question too. But saying that LLMs operate on words first is too big of an oversimplification
- rcxdude 3mo ago
  >Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really. It is, but it has been explored in various forms. Anthropic have some interesting papers where they can map out concepts to some extent as they exist inside the vectors of the LLM, and they can see e.g. that vectors relating to some concept can appear there some iterations before the LLM emits the tokens that spell it out. (though also, quite frankly, this shouldn't be all that surprising. If they were just picking the next token with no representation of what might follow it the result would be more like the 'one-word-at-a-time' party game than coherent text. They used poetry with a rhyming scheme as an example because it's something where you need to have some idea of what might rhyme with the line before and aim at it at the start of the next line in order to have any degree of success).
epihelix 3mo ago
I don't know about you, but I think with words all the time when reasoning through complex ideas. Where do you think the phrase "thinking out loud" comes from?

OhioMan2943 3mo ago

Meditative practices support this. Anyone who's spent even a moderate amount of time with meditation knows we are not our thoughts. If you don't believe me, learn how to meditate, meditate for literally just 10 minutes, and come back to me.

fedeb95 3mo ago
this doesn't imply that words are a byproduct of consciousness.

Diogenesian 3mo ago

St Augustine had it right 1500 years ago: humans learn words opportunistically according to their desires and problems and existing deep understanding of the world, and (critically) humans really can't learn language without being natively fluent in great ape facial expressions, gestures, grunts, etc. Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders. [https://faculty.georgetown.edu/jod/augustine/conf.pdf https://faculty.georgetown.edu/jod/augustine/conf.pdf]

michael1999 3mo ago

It is not all obvious that consciousness precedes language. Hellen Keller provides a vivid description of her pre-verbal mental state as something less than reflective conscious. > Before my teacher came to me, I did not know that I am. I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect. https://scentofdawn.blogspot.com/2011/07/before-soul-dawn-helen-keller-on-her.html https://scentofdawn.blogspot.com/2011/07/before-soul-dawn-he...

etrautmann 3mo ago
I think this is different if you have vision though. I struggle to understand how someone can look at intelligent animals that don’t have language and not intuitively assume that they’re conscious. Yes this is a tricky problem without an answer but IMO the elements of consciousness can exist without language if visual abstractions are present.
- sigbottle 3mo ago
  You could redefine language to include mental abstractions and maybe it wouldn't be so weird. Language is a shared medium. Maybe an intelligence is capable of building abstractions in their head without the need for say, a symbol system to hopefully communicate with species similar to itself. You may argue that the agent needs persistent memory. Well, so what? If the world has stable regularities from the perspective of the agent - if the sun rises every day, and language is associating certain visuals with certain thoughts (it's not when viewed in its original social definition - but conduct the thought experiment with sociality removed), who's to say that you can't form a kind of language around that? Of course, this is all armchair, and I'm not trying to separate out what "real language" is or whatever, just thought experiments. But the original poster has a good point though, that's been proven in papers, that the tokens an LLM spits out, themselves, attract them to certain distributions, and at least a decent amount of the tokens in CoT are there solely to bring itself to the right distribution.
- devmor 3mo ago

fedeb95 3mo ago

That words are a byproduct of consciousness is a bold claim and requires some justification. I'm not implying it's backwards also for humans, but that the statement might not be true, and something else could be. Also, defining consciousness is hard. For instance, you can dream words.

cwmoore 3mo ago
Eh? What else could communication be there for?
- novacrazy 3mo ago
  Survival.
t43562 3mo ago
Have you ever had impressions about something complicated, e.g. someone's personality which you cannot put into words? I'm thinking of my kid - her personality when she was tiny. I could see what it was but not explain it. Hence I think ideas are unquestionably first and words are the inadequate and inaccurate description of thoughts that exist without them.
- fedeb95 3mo ago
  That may be true without implying that words are a always a byproduct of consciousness. You can dream words.
- IAmGraydon 3mo ago
  I agree, and I think that's a good example. Sometimes we can hold a concept in our minds of something and cannot find the words to attach to it. The word we eventually give that concept when it's human, as in your example, is a name. This is because the concept of a personality is too complex to describe with other, preexisting words. Still, even though I think the author is right about consciousness/concepts preceding language, I think he's wrong about the other part. The statistical models that underlie AI very much remind me of what we call a human concept. It's a statistical model of that thing, person, or idea. However, we humans seem to have this ability to create new concepts that are wildly different than our training data (what we have experienced in life), and we call this creativity. It's missing from AI for some reason. That is the part that makes me question if AI will ever match humans.

jacobr1 3mo ago

LLMs as clearly doing more than just modeling words. In order to predict word placement, they need to build some kind of model, their latent space, of the types of things they are able to predict. Not really full world models yet - but they have decent "blog space" or "github project" models. And you can see this with multimodality, or non-text modality modals such as images and audio. They map from their latent spaces to the outputs. The fact that multi-modal systems can share the interior layers shows some kind of internal representation is created.

adamzwasserman 3mo ago
Or.. the LLMs are just "telescope" sufficiently powerful to detect the aligned internal representation that already exists. A more powerful telescope does not "create" new galaxies.
- jvanderbot 3mo ago
  Love the analogy. I a lot of people had their mind blown by vector space representation of words. The idea that Female + King averaged out around "Queen". Words might foundational to concepts or just really-well-designed ways to transmit them.
  - sometimelurker 3mo ago
    You have the word 'bot' in your name and judging by your comment history love using em-dashes. Are you a machine or not?
    - jvanderbot 3mo ago
      I doubt I used emdashes, but I do us "-". I dont have emdash unicode bound to my keyboard yet. (edit: Hyphenation is not emdash, of course) you can view my profile details freely and decide for yourself, and a quick look might even explain the "bot" suffix.
      - 3mo ago

krunck 3mo ago

For LLMs there is no consciousness.

visarga 3mo ago
We might quibble what to call it, but they use language productively.

2snakes 3mo ago

I do not think the written description of a dog is the same as a dog itself…

t43562 3mo ago
Nothing is the same as the dog but we probably store a lot of sensory information, emotions and images relating to dogs in our heads.
tsss 3mo ago
Maybe it is. If you think about it hard, you will find that you cannot separate the idea of dog from the word dog. The object dog can only be understood as a perception linked to that idea.
- orthoxerox 3mo ago
  "Ceci n'est pas un chien"

throwaway260124 3mo ago

"In the beginning was the Word, and the Word was with God, and the Word was God." And Islam the first verse was ‘read’ The first time I heard about emergent behavior in llms I thought about that. Maybe thought is too much because I don’t have anything coherent in mind but it feels so captivating like the premise of sci-fi story.

micoti 3mo ago
Word in the Bible is not meaning the actual word (like sematic array of characters) but more like greek word logos
- thewebguyd 3mo ago
  Correct, in the Koine Greek, logos is the word that's actually used. Logos is a bit deeper than words, and is more akin to a universal principle of order that governs the cosmos. "In the beginning was the logos" is like saying in the beginning were the laws of physics, and God is the laws of physics and the laws of physics are God.[1] [1] This also entirely depends on which framework you are translating from. This would be accurate from the stoic interpretation of the logos but differs from a more mystical Johannine interpretation, in which the logos is outside the physical universe, not just operating within it. Logos is like the source code of the universe.
- kaonwarb 3mo ago
  It's a hard word to translate. https://en.wikipedia.org/wiki/Logos https://en.wikipedia.org/wiki/Logos has a good overview.
_doctor_love 3mo ago
Some folks interpret that sentence in a mystical way - the first word was Om, from which all the universe was created.

chermi 3mo ago

Can you please point me to the proof of the first claim?

talkingtab 3mo ago

Words limit our understanding of words because they are intrinsic to a particular kind of understanding. yes? no? In "Notes on the Synthesis of Form", Christopher Alexander talks about how we can know something is wrong even when we do not know how to make something right. He was talking about this I think =z== as opposed to +|+|+ or +|+|+| I happened upon the word "entanglement" and it seems an interesting alternative to the order that is inherent in words. Our language is all about Actor-verb-object. Entanglement provides a fundamentally different concept. I cannot say "I drove my car on the road" with entanglement. Or at least with entanglement I can say something equally valid like "the road moved my car with me inside." And entanglement works for the + and | things, at least for me. There is some kind of entanglement (degree) that creates a gestalt. At least that is the best I can come up with.

cowlby 3mo ago

Isn't it interesting then how much of our intelligence is captured by our words/language? I would've thought to create AI you needed to replicate neurons and synapses and learning. So it's still amazing to me that statistical modelling of our words creates a Claude Opus.

ordersofmag 3mo ago
Perhaps LLM's do replicate the important features of "neurons and synapses and learning" and perhaps a "statistical modelling of our words" is pretty much what our brains are doing? Most of the counter-arguments I've seen boil down to "but humans are special" which I'm not sure I find compelling. Not saying LLM's are conscious. Just that much of our amazement about their behavior seems to say more about us realizing the things we can do are not magic, rather than them being so amazing.

speak_plainly 3mo ago

The author is making a big mistake by making a normative claim about 'true' intelligence and consciousness. Is there a correct way to cognize? And although he feels cognizing works in a very specific direction, his brain is basically doing a very similar guessing game on a deep level with training of pathways that started at birth. Basing the argument in a feeling about consciousness is not convincing. The author thinks they are describing a unique human magic, forgetting Augustine's insight that human thought doesn't precede the word, but is brought into consciousness by it.

tsss 3mo ago

No, I don't think so. Language is intimately intertwined with our understanding of the world. Many people make the mistake to think that languages are about the spelling or the sound or maybe the grammar. But in reality they are about how words are defined only on relation to each other. A symbol is defined by what it is not. This relationship between symbols structures how we think about the world and how we fantasize and desire. You can perhaps have a consciousness without words, but not without language.

deleted 3mo ago
[deleted]

Festro 3mo ago

"At its core, it is a big pile of words that predicts the next word, using some maths the computers figured out." "An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. The words are everything." This is just flat out wrong. Words are used as training data, to build a system of vector embeddings. The LLM contains no words. That was the training data long discarded. Vector embeddings are groupings of meanings derived from the relationship between the words in its training data. This entire system is modelled after human neural mechanisms in a way that machines can emulate. "But your brain works the other way around. First there is a concept, a feeling, an image, and then the words come out to describe it. (At least, this is how I feel my own brain working.) For us, words are the byproduct of consciousness." It's working the same way around (you are not saying Ai has consciousness that is derived from words!). And you make a huge jump from the concept that 'words come from concepts' to 'words are the byproduct of consciousness'. Because 'concepts' are not equal to 'consciousness'. In fact you do not define consciousness at all, making it hard to determine what argument you are actually making at all. You seem to think humans have this trait of 'consciousness' but can't explain or evidence it beyond 'feeling it'. Consciousness it defined by many as the ability to experience events and process thoughts or qualia. It's hard to test this, and we can see why that is with an AI - it may claim to be conscious, or even claim not to be, but how can we trust either answer? Philosophers aren't even sure to trust another human who claims to be conscious, and we struggle constantly to determine at what point creatures the animal kingdom are conscious or not. Cats? Lobsters? Snails? or even across the plants? Before you can refute the consciousness of AI, first establish the consciousness of humans. A better question to ask is what does the uncanny emergent ability of an AI to mimic a human say about consciousness?

encyclopedism 3mo ago
Defining consciousness is one of the most difficult outstanding problems in the whole of science please see: https://en.wikipedia.org/wiki/Hard_problem_of_consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness An approach that might shed some light is instead to define what consciousness ISN'T. Naively let us say consciousness is NOT a large list of weights (i.e. an LLM). The uncanny emergent ability depends entirely on training data. A mathematical model is used to match output against training data (via loss functions etc). The training data contains all the human ingenuity, logic, rational, patterns and features. Try giving an LLM model the alphabet ALONE and see what it comes up with?

skerit 3mo ago

But LLMs aren't just shuffling words around. Words (tokens) are the "human interface" at the edges. Text input gets embedded, then there's this huge latent-space computation in the middle, and only at the tail end does that get converted back into word/token probabilities. So just saying that "words are the source" isn't entirely correct and feels misleading.

visarga 3mo ago

> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? Let's take a feral human, do they achieve the same with language powered humans? No, obviously. It means language does something we can't replace with "consciousness". Maybe something life preserving, putting consciousness downstream of language use. But more generally - how can 8B humans make a living on this planet? Not without language, that is for sure! We long passed the stage where we could exist without language at our consumption rate. We can't even exist without math, population collapse would be the outcome. Minds are downstream from language & math. For example, is music downstream of piano or piano downstream of music? I think you can't cut this cleanly. They developed in relation to the other, recursion between tool and art form is already old.

adamzwasserman 3mo ago

As others have remarked: this is an unsubstantiated claim. I am actually using LLMs to perform controlled experiments to determine if the opposite is true, and that consciousness is a byproduct of language as Jaynes might have agreed.

fooker 3mo ago

We can't define consciousness well enough to make claims like this.

degecko 3mo ago

It’s funny because, ever since Norbert Wiener’s Cybernetics was published in 1948, we’ve been trying to replicate how the human brain works. In that sense, we’ve been building increasingly powerful machines that mimic human thought. And now, the most interesting thing is something that turns that idea on its head: LLMs think in reverse.

sturza 3mo ago

“The relationship of thought to word is not a thing but a process, a movement from thought to word and from word to thought.” Vygotsky

sourdecor 3mo ago

Reminds me of this from "Gödel, Escher, Bach: an Eternal Golden Braid": "Let me rephrase these last couple of sentences without using the slightly technical term "isomorphism". When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with some degree of meaning - indeed, such tracking or mirroring is no less and no more than what meaning is. Depending on how complex and subtle and reliable the tracking is, different degrees of meaningfulness arise." - P-3 EDIT: I initially wrote "G.E.B." instead of "Gödel, Escher, Bach: an Eternal Golden Braid" which was too cryptic.

PaulDavisThe1st 3mo ago
While I know what Douglas Hofstadter's book "Godel Escher Bach - an eternal golden braid" is, it might be worth revisiting your apparent assumption that more or less everybody on HN knows about a book from 47 years ago (regardless of its significance to those of us who have encountered it already).
- kazinator 3mo ago
  LOL, I happen to have a "Hofstadter_GEB.pdf" open on my desktop; been there for months.

Wowfunhappy 3mo ago

> But one thing scares me a little. What if LLMs slowly become worse? Think about it. Everything written before 2017 was made by humans. After 2017, LLMs started filling the open web with their own content. And now that same content is being fed back to train the next LLMs. An LLM can give you perfect grammar and a rich vocabulary, but it can quietly lose the real context. Right now the share of AI content is small. But what happens when it keeps growing? Have the last-generation LLM pre-screen new training data for the next generation LLM.

pigeonwarz 3mo ago

I'm going to go on a limb here and challenge this assumption. As a parallel thread to the point put forward by michael1999, can we not argue words manifest as the result of the maths performed within LLMs, maths that, for all intents and purposes, is their world? Because the internal maths of how LLMs work is likely to be significantly cruder than the 'real' world that defines consciousness, perhaps we are closer to the ghost in the machine than we would like to admit.

awongh 3mo ago

> But your brain works the other way around. First there is a concept, a feeling, an image, and then the words come out to describe it. After this I can't take the essay seriously- this sort of blanket statement about the one true hierarchy of consciousness and knowledge is BS. While it seems to have been disproven that words in other languages cause people to speak and think differently, that also doesn't mean that words don't have any effect on the way we think, or that the concept of a word always has to come before the word itself. The reason why we experience the uncanny valley of LLMs is because they don't represent a true consciousness, BUT it's also clear that the architecture represents certain qualities of consciousness- as the models have scaled we can see that it has some other non-word related understanding. The evidence points to consciousness as a set of interlocking systems- attention, long term memory, short term memory, emotions, etc.

fasteddie31003 3mo ago

Maybe consciousness just does not exist. Could it be that humans invented it to feel superior to animals, but it does not really exist? I wonder if we will think of the word "consciousness" in the same way we think of the word "aether" in the future.

brotchie 3mo ago
If consciousness didn’t exist, you wouldn’t exist! you’d have no interior experience.
- Dependance 3mo ago
  How do you know they have one ?
  - kelseyfrog 3mo ago
    I lack a subjective experience and have no interiority. Watch this. They won't even believe you if report your own internal state. They'll even claim they have epistemic authority over your own interiority while not granting you the same authority over their internal state.
  - wolvesechoes 3mo ago
    Wittgenstein dealt with that.
DanHulton 3mo ago
There’s certainly a debate as to whether animals are conscious or not, and if we’ve just been engaging in a kind of human chauvinism all this time, but I mean consciousness definitely exists. There’s an “I” in here that perceives, and it’s reasonable to extrapolate that I’m not particularly special, that others that have the same parts as me have the same properties (consciousness).
GuB-42 3mo ago
Consciousness exists by definition. It is a subjective experience and you are experiencing it. Or more precisely I am experiencing it, but since you are, I think, a human just like me, with a brain and biology very similar to mine, I assume that you must experience consciousness too. And that is the big question, and one that science may never have an answer to. What is conscious and what is not? The only consciousness I know exists is mine, the rest is just guessing based on similarities. Other humans: almost certainly conscious, higher intelligence life forms: probably conscious, plants: probably not conscious, rocks: almost certainly not conscious, LLMs: ??? LLMs are unusual in that they have a very human-like behavior, more than anything non human, which would make them a good candidate for having consciousness, but on a material level, they are more like rocks, which point them as not being conscious.

smallerfish 3mo ago

> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it. In (at least once school of) Tibetan Buddhist meditation you observe the part of the mind that is producing ideas, and then suppress it in order to explore deeper layers of consciousness. The "lights on" "you" that is observing is _not_ the idea producing part of the mind, which to me at least seems to be more mechanical. I do agree with the author of course that a concept isn't necessarily wrapped with language from the start, and can be a skin; but equally it is possible to think in a verbal mode.

neovive 3mo ago

> For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. Recently, I've been thinking a lot about context and whether an LLM actually creates a true representation of a concept within its vectors. And for our current usage of LLMs, does it even matter? As models become so large, will the difference even be perceivable to us?

Autious 3mo ago

Steve Jobs didn't invent the PC. It's reductionist and offensive to single him out and thanking him for that. Computers didn't "figure out the maths". People did the math, the computers did the calculations. Human knowledge is nowhere complete, it's silly to think so, and by extension it's silly to think the LLM's have anything near a complete knowledge set. They don't even have access to all of current human knowledge, much less so future knowledge. A lot of text and information is locked away from the public internet and the internet altogether. Not all types of useful ideas for applications have been invented, that's ridiculous, there are plenty of ideas that people haven't had yet. There are however a massive amount of shitty copies of the same shitty ideas.

ohthehugemanate 3mo ago

This person is just making stuff up that feels true to them. Caveat lector. There are reams of grown up scientific research and philosophy on the relationship between language, thought, neurobiology, and inference. It's nice to hear how one person feels they think, but there is no weight to it if they don't know page one of the syllabus.

deleted 3mo ago
[deleted]
throw310822 3mo ago
> just making stuff up that feels true to them Hallucinating it?
triprjt 3mo ago
would second you on this.

csomar 3mo ago

> When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. Most people don't actively think when they are talking or writing most of the time. Only during period of thinking that we do think. Most of the time, we just talk words. There is evidence is that we just observe ourselves talking and that the thinking itself is not conscious but it's still a theory. > Now ask the same question about an LLM. For an LLM, it is exactly the opposite. No one has any idea how an LLM come to correct words. LLMs produce numbers (tokens) that end up to coherent words. > So what is an LLM, really? At its core, it is a big pile of words that predicts the next word, using some maths the computers figured out. No, not really. > The words are everything. For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. The author provides no proof for this and there are very good reasons to think that LLMs have inherent thinking. The rest of the article was just some random thoughts the OP jotted down? I just don't see their coherence to the beginning of the article.

IAmGraydon 3mo ago

>An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. The words are everything. For an LLM, words are the source, and any meaning is just a byproduct that falls out by accident. I don't think that's right. What makes LLMs (and other forms of "AI") work is the very fact that they hold a statistical model underneath and put words on top of it. That statistical model, at least to me, appears to be somewhat akin to what a human would call a "concept". Like a stable diffusion model has a statistical model of what a "chair" is. Not one exact chair, but all chairs. They all have backs, 4 legs, a flat area to sit on. It can then carve an image of a chair out of noise using this concept. Take a music model like Suno as another example. If you tell it to make a rock song, it has a statistical idea of what this would looks like. The harmonic progression, the tempo, the types of instruments used. It then attaches sounds to this statistical model and you get a "song". LLMs also appear to do this. If you ask it for a paragraph of a specific flavor of prose, it has that as a statistical model, and it attaches words to it. I am not a big believer in LLMs delivering on even half the hype they've generated. However, this very idea of human concepts vs AI's statistical models is the one thing makes me wonder, sometimes, if they're on the right track, even if we are a long way off from AGI. It's kind of funny that the author of the above article landed on the opposite conclusion.

lebuffon 3mo ago

I look at culture, under which I put language as a component, as the O/S of the human brain. So yes a brain with no O/S can think but it probably will never replicate, in one life time, the accumulated thinking skills of a brain given a good O/S. How many of us would independently reinvent language with specific tenses, the diatonic scale, the number zero, perspective drawing, geometry, calculus, atomic theory, quantum mechanics etc. if left on our own from childhood? So to me, the question becomes, can human language encode the sum of all human thought? The answer might be no. But it might be good enough to get close.

InfiniteRand 3mo ago

I think small children learning language learn in a way similar to LLM's, just saying words that seem appropriate to the situation without any understanding of the meaning. But a key difference in my mind, is they use the reactions to this text prediction to help them construct a picture of the world, which then can interact with their thinking of what words to say.

DevarshRanpara 3mo ago

I want to add one thing, Please think about how animals perceive world? They don't have language, but they are conscious for sure.

sesteel 3mo ago

I am pretty sure human level intelligence is emergent from language use. The more complex the language the higher the intelligence since language is itself a tool used to not only to communicate, but also classify the human experience. Without language, we are fairly primitive relying on instincts and more primitive communication. Feral children are an example of what a base state might be.

sometimelurker 3mo ago

As I see it, LLMs will/do develop 'introspectiveness' if it is selected for during RL. If its not delibrativly selected for (this is probably done because it helps with alignment; if the machine can align itself that's great) its probably a generally useful skill to have when doing normal tasks, https://www.anthropic.com/research/introspection https://www.anthropic.com/research/introspection

jvanderbot 3mo ago
Their introspectiveness doesn't strike me as indicative of anything, since we might be just reinforcing a "plausible word salad generator for input context that leads to better solutions" or just a "post hoc rationalizer", both of which may or may not be the key mechanism behind verbal consciousness.

cathalharte 3mo ago

I share your positivity, however I don’t think figuring out how to market through the noise is the important part. LLMs don’t need to think, they just need to be a more efficient data retrieval system than what already exists (and what has already been sold to the highest bidder). With LLMs, I’ve find I’ve started reaching the corners of the internet, where the true lovers of knowledge are doing the work they have always been doing. Quality does eventually speak for itself, at least once we’re able to strip away the marketing (you mentioned Apple, do you believe they are truly a cut above, or is it just hype?) If LLMs can filter the noise, build it and they will come might return as a valid way to be.

deleted 3mo ago

[deleted]

weare138 3mo ago

https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse

zkmon 3mo ago

> First there is a concept, a feeling, an image, and then the words come out to describe it. Nope. Concepts, feelings and ideas do not exist independent of expression (words, sounds, responses). Humans have the same process as LlMs have. And that's precisely why LMs are able to succeed. Similar to how camera works based on the working of an eye, or how aeroplane works based on a birds wing. If you still believe that concepts exist independently, imagine a creature that can't express, make sounds, or respond with actions. What exactly is a "concept or feeling" to such creature? Do trees have concepts and feelings? Infact trees could, because they can respond to stimulants. If you can't describe or perceive a thing, that thing doesn't exist in your world. It's that simple. What you are describing as "translating thoughts into words" is actually, refining your expression. The expression already exists at the same time as the thought.

orthoxerox 3mo ago
My son is insufficiently verbal (he's of the kind that communicates with quotes, but really struggles to express or understand new sentences), and I am quite sure he has concepts, feelings and ideas.
- zkmon 3mo ago
  Ofcourse he has. Because he does express in his own way. He is not struggling to express, by the way. It's your perception that others are struggling. It's like a rich man who thinks that everyone less richer than him are struggling. But your measure, all animals are struggling to express. No they aren't. You are defining expression from your own, specific standpoint. The concepts one has, do not exceed their expression abilities. So nobody is struggling.

sgarrity 3mo ago

Forgive my quoting my own blog post: Writing is a shadow of thought. The better the writing, the more clearly the shadow represents the shape of the original thought. Even the best writing can never perfectly capture the original thought. Writing is one of the best tools we have to share thoughts across space and time.

pizza234 3mo ago

> Now come back to my first question. An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath. No!! This is misinformation. An LLM predicts the next words (tokens) based on the words before it *and* on the internal representations it has learned from training. Processing language can lead to internal representations that capture patterns and relationships; LLMs can exhibit emergent abilities, including reasoning-like (stress on "-like") behavior, even when they weren't explicitly trained for that. Making reductionist claims about LLMs, based on next-word prediction, is similar to making such claims about biology, based on amino acids.

VeninVidiaVicii 3mo ago

Language is just a shortcut to logic. LLMs don’t seem to have the capacity to logic, they just pattern match. LLMs seem conscious because they are pattern matching against all of human written cognition.

avensec 3mo ago

I find the comment section interesting. There is so much discussion about the dichotomy of consciousness, in support and opposition as discussed by the author. Some of the "rights" and "wrongs" feel way more combative than, simply noticing where the author is currently at. I see the temporal disconnected strong here. The author is young-ish, and appears to be exploring ideas of many that have come before him. And, that he'll likely continue to explore on his journey. You can see his thoughts and see echos of long debated thoughts about consciousness, language, meaning, bottom-Up/top-down processing, not being our thoughts, etc.. Thanks to those who are leaving their knowledge, book references, quotes, and etc., to help myself and others who are somewhere along the continuum of understanding ourselves more thoroughly!

DevarshRanpara 3mo ago
You are 100% right, I am still exploring (and also learning from all these comments). I have just started to explore these things now. Thank you for understanding author's mindset.

waffletower 3mo ago

"An LLM predicts the next word based on all the words before it. That is the whole story. There is no idea sitting underneath." At most, and I will argue for less, that is half the story. This egregious over-simplification doesn't account for training and the massive dimensional space of thought it forms. Given that frontier models likely have trillions of parameters, the combinatorial space of ideas within approaches the infinite -- hardly zero.

tyzoid 3mo ago

I have to disagree with the fundamental premise here. LLMs do indeed have an internal "concept" mapping - the embedding layer. There's some fun demos on youtube[1] where you can manipulate embedding vectors (such as king - man + woman = queen), and these manipulations are done entirely in the embedding/concept space. [1] https://youtu.be/gQddtTdmG_8 https://youtu.be/gQddtTdmG_8 (around 14:00)

monkpit 3mo ago

Words are a representation of an underlying something; LLMs do not deal with words, they deal with tokens and embeddings. There’s a transformation layer between words and inference and back.

anonymousiam 3mo ago

The author had a perfect opportunity to decimate the whole concept of using LLMs as "artificial intelligence", but he shifts gears in the final six paragraphs and doesn't move in for the kill.

JKCalhoun 3mo ago

Yeah, disagree with many of the points this article takes for granted. Do I have a thought and the words follow? I'm not sure that is the case. My sense is that I have a glimmer of an idea when I begin to speak—but not fully fleshed out as yet. As I start to assemble words and speak them, the idea begins to coalesce—the words providing something like a scaffolding for my thinking. (Of course, I might not be the most reliable narrator: analyzing my own thoughts.)

IraFrost 3mo ago

Then give copy.lu a try — it’s Lu.

grantcas 3mo ago

[dead]

singularity2001 3mo ago

Words are also not necessary for LLMs, hear me out: they could be trained mostly on video showing how to solve problems and would work almost the same (in principle). That's because both word tokens and video will end up in the same latent space which is where real consciousness and thinking takes place.

DevarshRanpara 3mo ago
Humans figured out how to hunt animals by making spears from stone by their own, it was not programmed, and that is the main thing which makes us different, humans can go against their instinct, why they do it? Because of consciousness. Can LLM (Or any Model) produce something without training it on data? (Not saying you are wrong, just pasting my thoughts, your thinking which says video can also train a model is an interesting way to look!)