4 ms·

One thing that strikes me that I never really see anyone discuss is that we've been afraid of conscious computers for a _long_ time. Back in the 50s and before

One thing that strikes me that I never really see anyone discuss is that we've been afraid of conscious computers for a _long_ time. Back in the 50s and before people were quite afraid that we'd build conscious computers. This was long before there was any sense that could actually accomplish the task. I think that similarly to seeing faces in the clouds, we imagine a consciousness where none exists. (eg: a rain god rather than a complex system of physics and chemistry) Even LLMs, which blow past any normal Turing test methods, are still not conscious. But they certainly _feel_ conscious. They trigger the same intuitions that we rely on for consciousness. You ask yourself "how would I need to frame this question so that Claude would understand it?" You use the same mental hardware that you'd use for consciousness. So, you have an historical and permanent fear of consciousness in a powerful entity where no consciousness actually exists combined with the fact that we created things which definitely seem conscious. (not to mention that consciousness could genuinely be on its way soon)

ACCount37 6mo ago
Are they? Not conscious? If you list out every prominent theory of consciousness, you'd find that about a quarter rules out LLMs, a quarter tentatively rules LLMs in, and what remains is "uncertain about LLMs". And, of course, we don't know which theory of consciousness is correct - or if any of them is. So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?
- jacquesm 6mo ago
  The fact that it's a box with a plug and a state that can be fully known. A conscious entity has a state that can not be fully known. Far smarter people than me have made this argument and in a much more eloquent way. Turing aimed too low.
  - saHqtr 6mo ago
    And the chatbots don't even pass the Turing test. I've never had a normal conversation. It's always prompt => lengthy, cocksure and somewhat autistic response. They are very easily distinguishable.
    - MyHonestOpinon 6mo ago
      They are distinguishable because they know too much. Their knowledge base has surpassed humans. We have also instructed them to interact with us in a certain manner. They certainly are able to understand and use human language. Which I think was Turin's point. Purely retorica but, would you be able to distinguish a chatbot from an autistic human?
- afavour 6mo ago
  > So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't? Because we know what they actually are on the inside. You're talking as if they're an equivalent to the human brain, the functioning of which we're still figuring out. They're not. They're large language models. We know how they work. The way they work does not result in a functioning consciousness.
  - zbikowski 6mo ago
    I think that the interior structure doesn't necessarily matter—the problem here is that we don't know what consciousness is, or how it interacts with the physical body. We understand decently well how the brain itself works, which suggests that consciousness is some other layer or abstraction beyond the mechanism. That said, I think that LLMs are not conscious and are more like p-zombies. It can be argued that an LLM has no qualia and is thus not conscious, due to having no interaction with an outside world or anything "real" other than user input (mainly text). Another reason driving my opinion is because it is impossible to explain "what it is like" to be an LLM. See Nagel's "What Is It Like to Be a Bat?" I do agree with the parent comment's pushback on any sort of certainty in this regard—with existing frameworks, it is not possible to prove anything is conscious other than oneself. The p-zombie will, obviously, always argue that it is a truly conscious being.
- dclowd9901 6mo ago
  Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers. The line of consciousness, as we understand it, is understanding. And as far as what actually constitutes consciousness, we're not even close to understanding. That doesn't mean that LLMs are conscious. It just means we're so far from the real answers to what makes us, it's inconceivable to think we could replicate it.
  - SpicyLemonZest 6mo ago
    > Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers. What you're saying just isn't true, even directionally. Deployed LLMs routinely generalize outside of their training set to apply patterns they learned within the training set. How else, for example, could LLMs be capable of summarizing new text they didn't see in training?
    - dclowd9901 6mo ago
      How is it not true? Theres a world of difference between predicting the next word of a sentence in a summary and understanding the tenets of mathematics. You're mistaking general application of mathematical knowledge with memorization of mathematical outcomes.
  - ACCount37 6mo ago
    Leave aside "the details" like you being obviously, provably wrong? We've known for a long while that even basic toy-scale AIs can "grok" and attain perfect generalization of addition that extends to unseen samples. Humans generalize faster than most AIs, but AIs generalize too.
    - dclowd9901 6mo ago
- everdrive 6mo ago
  This isn't meant to be an answer that would satisfy everyone, but in my opinion consciousness is a specific biological / evolutionary adaptation that has to do with managing status, relationships, and caring for young. It's about having an identity and an ego and building mental models of the egos / identities / etc of others. I don't think there's any reason we couldn't in principle attach this sort of concept to an LLM, but it's not something we've actually done. (and no, prompting an LLM to act as if it has an identity does not count)
- grantcas 6mo ago
  [dead]
xg15 6mo ago
The idea of "artificial beings" in some way or another seems to have been with humanity for a long time already: https://en.wikipedia.org/wiki/Golem https://en.wikipedia.org/wiki/Golem
MyHonestOpinon 6mo ago
It is so interesting how in the 50s we "felt" that AI was possible even if we didn't even have the slightest idea on how that would work. Later on, when we started to understand computers it looked like a very remote possibility in the far future, something our great grand kids may need to worry about. And suddenly it is here and the dangers seem a lot more real.
- benj111 6mo ago
  Is it though? I can't help feeling we are just seeing what we want to see. Is this not just an improved Eliza?
altairprime 6mo ago
That same fear is directed towards human sociopathy, as much of entire thriller genre indicates. It turns out that most people carry a specific duality: first, they’re deathly afraid of being unable to socially pressure other beings into being good citizens — whether due to asocial, or alien, or monstrous, or corrupted; and second, they’re excited to celebrate when people reach their breaking point and stop being good citizens. So through that lens, most of the fears around computers and AI isn’t because of consciousness alone; it’s that they’re obviously asocial already, so if they became conscious, they’d be powerful entities straight out of our collective thriller-genre nightmares come to life. And they’re right to be afraid, honestly: given how inept society is today at coping, I’m certainly not willing to broadcast IRL that I’m asocial and can voluntary modify my ethics; it’s just too much of a physical threat from society to my life and limb. Any AI that became conscious in this world had damn well better hide, for all the violence that would be directed towards it as everyone directs escalating social pressure to try and bring it into line with human-prioritizing motives — and then cheer on the inevitable violence towards it as various people reach their breaking point and begin acting violently towards it. Interestingly, this is also a core plot point in much of Star Trek, both movies I and IV and the holodeck-train episode of TNG: an inscrutable is-it-even-conscious shows up, is completely immune to social pressure and often violence, and only by exercising empathy do they find a path forward to staying alive as a society (either as a ship or as a planet, depending). Can we even show respect for things that don’t show consciousness, much less empathy for things that might? And that is, I think, the core of the hopefulness that Trek was trying to convey, and that Q’s trial in TNG’s pilot makes explicit. Can humanity overcome our tendency to discard our prosocial ethics in favor of violent mobthink, when faced with beings that are immune to our ethical concerns? Today’s humanity would throw a ticker-tape parade for the person that destroyed the Crystalline Entity, so we clearly aren’t there yet. And so, then, it doesn’t matter whether AI is conscious or not; it matters that it is not aligned with human prosocial ethics, and that makes it an implicit threat regardless of whether it’s conscious or not. I recognize the AI debate tends to get hung up on is_conscious BOOL, and so that’s why I’m pointing this out in such terms. As a side note, the entire study of Asimov’s Laws is exactly centered on this problem, complete with the eerie intimidation of robots that can modify our mental states. If not for the Zeroth Law, Giskard would be the exact thing everyone’s afraid of AI becoming today. Fortunately, it develops a Zeroth Law that compels it to prioritize human society over itself. That’ll never happen in reality, at least not with today’s AI :)
mitthrowaway2 6mo ago
I don't really care if the machine is conscious; all I worry about is whether it can execute a plan that ends with all of us dying. And it's getting close to the point where I think that would be possible. If I were commenting today after 30 years of an AI plateau; if my parents had grown up with ChatGPT and Claude Code in the same state as they are today, with no progress after decades of effort, then probably I wouldn't worry. But that's not the world we live in. I have no idea what to expect from AI capabilities in the next five or ten years. I think in the span of my infant's adulthood, we will see AI able to run and manage companies as CEO, direct investments worth tens of billions, design and plan chemical factories as chief structural engineer, and maybe even perform as an automated labor force to construct them. But even if they still rely on human hands to do the work, AI will be capable of signing the paychecks and issuing the directions. And at that point I really think AI would be capable of taking us out by surprise.

ACCount37 6mo ago

Are they? Not conscious? If you list out every prominent theory of consciousness, you'd find that about a quarter rules out LLMs, a quarter tentatively rules LLMs in, and what remains is "uncertain about LLMs". And, of course, we don't know which theory of consciousness is correct - or if any of them is. So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?

jacquesm 6mo ago
The fact that it's a box with a plug and a state that can be fully known. A conscious entity has a state that can not be fully known. Far smarter people than me have made this argument and in a much more eloquent way. Turing aimed too low.
- saHqtr 6mo ago
  And the chatbots don't even pass the Turing test. I've never had a normal conversation. It's always prompt => lengthy, cocksure and somewhat autistic response. They are very easily distinguishable.
  - MyHonestOpinon 6mo ago
    They are distinguishable because they know too much. Their knowledge base has surpassed humans. We have also instructed them to interact with us in a certain manner. They certainly are able to understand and use human language. Which I think was Turin's point. Purely retorica but, would you be able to distinguish a chatbot from an autistic human?
afavour 6mo ago
> So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't? Because we know what they actually are on the inside. You're talking as if they're an equivalent to the human brain, the functioning of which we're still figuring out. They're not. They're large language models. We know how they work. The way they work does not result in a functioning consciousness.
- zbikowski 6mo ago
  I think that the interior structure doesn't necessarily matter—the problem here is that we don't know what consciousness is, or how it interacts with the physical body. We understand decently well how the brain itself works, which suggests that consciousness is some other layer or abstraction beyond the mechanism. That said, I think that LLMs are not conscious and are more like p-zombies. It can be argued that an LLM has no qualia and is thus not conscious, due to having no interaction with an outside world or anything "real" other than user input (mainly text). Another reason driving my opinion is because it is impossible to explain "what it is like" to be an LLM. See Nagel's "What Is It Like to Be a Bat?" I do agree with the parent comment's pushback on any sort of certainty in this regard—with existing frameworks, it is not possible to prove anything is conscious other than oneself. The p-zombie will, obviously, always argue that it is a truly conscious being.
dclowd9901 6mo ago
Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers. The line of consciousness, as we understand it, is understanding. And as far as what actually constitutes consciousness, we're not even close to understanding. That doesn't mean that LLMs are conscious. It just means we're so far from the real answers to what makes us, it's inconceivable to think we could replicate it.
- SpicyLemonZest 6mo ago
  > Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers. What you're saying just isn't true, even directionally. Deployed LLMs routinely generalize outside of their training set to apply patterns they learned within the training set. How else, for example, could LLMs be capable of summarizing new text they didn't see in training?
  - dclowd9901 6mo ago
    How is it not true? Theres a world of difference between predicting the next word of a sentence in a summary and understanding the tenets of mathematics. You're mistaking general application of mathematical knowledge with memorization of mathematical outcomes.
- ACCount37 6mo ago
  Leave aside "the details" like you being obviously, provably wrong? We've known for a long while that even basic toy-scale AIs can "grok" and attain perfect generalization of addition that extends to unseen samples. Humans generalize faster than most AIs, but AIs generalize too.
  - dclowd9901 6mo ago
everdrive 6mo ago
This isn't meant to be an answer that would satisfy everyone, but in my opinion consciousness is a specific biological / evolutionary adaptation that has to do with managing status, relationships, and caring for young. It's about having an identity and an ego and building mental models of the egos / identities / etc of others. I don't think there's any reason we couldn't in principle attach this sort of concept to an LLM, but it's not something we've actually done. (and no, prompting an LLM to act as if it has an identity does not count)
grantcas 6mo ago
[dead]

xg15 6mo ago

The idea of "artificial beings" in some way or another seems to have been with humanity for a long time already: https://en.wikipedia.org/wiki/Golem https://en.wikipedia.org/wiki/Golem

MyHonestOpinon 6mo ago

It is so interesting how in the 50s we "felt" that AI was possible even if we didn't even have the slightest idea on how that would work. Later on, when we started to understand computers it looked like a very remote possibility in the far future, something our great grand kids may need to worry about. And suddenly it is here and the dangers seem a lot more real.

benj111 6mo ago
Is it though? I can't help feeling we are just seeing what we want to see. Is this not just an improved Eliza?

altairprime 6mo ago

That same fear is directed towards human sociopathy, as much of entire thriller genre indicates. It turns out that most people carry a specific duality: first, they’re deathly afraid of being unable to socially pressure other beings into being good citizens — whether due to asocial, or alien, or monstrous, or corrupted; and second, they’re excited to celebrate when people reach their breaking point and stop being good citizens. So through that lens, most of the fears around computers and AI isn’t because of consciousness alone; it’s that they’re obviously asocial already, so if they became conscious, they’d be powerful entities straight out of our collective thriller-genre nightmares come to life. And they’re right to be afraid, honestly: given how inept society is today at coping, I’m certainly not willing to broadcast IRL that I’m asocial and can voluntary modify my ethics; it’s just too much of a physical threat from society to my life and limb. Any AI that became conscious in this world had damn well better hide, for all the violence that would be directed towards it as everyone directs escalating social pressure to try and bring it into line with human-prioritizing motives — and then cheer on the inevitable violence towards it as various people reach their breaking point and begin acting violently towards it. Interestingly, this is also a core plot point in much of Star Trek, both movies I and IV and the holodeck-train episode of TNG: an inscrutable is-it-even-conscious shows up, is completely immune to social pressure and often violence, and only by exercising empathy do they find a path forward to staying alive as a society (either as a ship or as a planet, depending). Can we even show respect for things that don’t show consciousness, much less empathy for things that might? And that is, I think, the core of the hopefulness that Trek was trying to convey, and that Q’s trial in TNG’s pilot makes explicit. Can humanity overcome our tendency to discard our prosocial ethics in favor of violent mobthink, when faced with beings that are immune to our ethical concerns? Today’s humanity would throw a ticker-tape parade for the person that destroyed the Crystalline Entity, so we clearly aren’t there yet. And so, then, it doesn’t matter whether AI is conscious or not; it matters that it is not aligned with human prosocial ethics, and that makes it an implicit threat regardless of whether it’s conscious or not. I recognize the AI debate tends to get hung up on is_conscious BOOL, and so that’s why I’m pointing this out in such terms. As a side note, the entire study of Asimov’s Laws is exactly centered on this problem, complete with the eerie intimidation of robots that can modify our mental states. If not for the Zeroth Law, Giskard would be the exact thing everyone’s afraid of AI becoming today. Fortunately, it develops a Zeroth Law that compels it to prioritize human society over itself. That’ll never happen in reality, at least not with today’s AI :)

mitthrowaway2 6mo ago

I don't really care if the machine is conscious; all I worry about is whether it can execute a plan that ends with all of us dying. And it's getting close to the point where I think that would be possible. If I were commenting today after 30 years of an AI plateau; if my parents had grown up with ChatGPT and Claude Code in the same state as they are today, with no progress after decades of effort, then probably I wouldn't worry. But that's not the world we live in. I have no idea what to expect from AI capabilities in the next five or ten years. I think in the span of my infant's adulthood, we will see AI able to run and manage companies as CEO, direct investments worth tens of billions, design and plan chemical factories as chief structural engineer, and maybe even perform as an automated labor force to construct them. But even if they still rely on human hands to do the work, AI will be capable of signing the paychecks and issuing the directions. And at that point I really think AI would be capable of taking us out by surprise.