5 ms·

What makes it obviously not conscious is that it isn't a process. It can't do anything unless you ask it a question. I can certainly imagine a conscious or ne

What makes it obviously not conscious is that it isn't a process. It can't do anything unless you ask it a question. I can certainly imagine a conscious or nearly-conscious AI someday, one that likely uses an LLM to handle language processing in much the same way that a cpu might use a coprocessor to do floating point operations. But just as you wouldn't say a computer is intelligent because it can do math, I think chatpgt has proven that the same can be said of processing natural language.

Uehreka 4y ago
So like, if someone wrote a systemd daemon that continuously fed input to a local ChatGPT instance (passing in context from its previous interactions as well so it could act as a continuous entity) and performed actions based on its responses, you’d consider that conscious? I’m willing to buy “ChatGPT is not conscious” arguments if people can come up with a rigorous definition of what “conscious” means. But so far I don’t see a lot of people doing that, and I don’t know if it’s even possible.
- jhrmnn 4y ago
  I’d think it’d need to be something like a diffusion model that’s continuously fed noise which it turns into outputs (thoughts). It’d also need to be getting continuous feedback and to have the ability to incorporate that feedback over all scales, not just the most recent window (some hybrid of a state and continual learning).
  - Uehreka 4y ago
    That both doesn’t seem that far off, and doesn’t feel like a very rigorous definition of consciousness. Like, if you’re describing qualities you yourself would find interesting, that’s one thing. But if we’re going to be saying “hooboy, that isn’t consciousness” with our chests, we really need an agreed-upon definition of what consciousness is that both can be met by a sufficiently advanced system and agrees with the way people use the word “consciousness” in everyday speech. Honestly, I don’t think that’s possible, but I’m willing to entertain/critique people’s proposals.
    - jhrmnn 4y ago
      Yeah, that was supposed to be just my personal feeling for the thing. And it’s not even a description of consciousness but rather of something that I imagine might have a capacity for consciousness. I don’t think we’ll really know until we get there, and then we’ll know.
- ineptech 4y ago
  I don't have a rigorous definition, but I don't think we need one at this point, do we? Chatgpt is a search engine with a natural language interface, you can draw a straight line from its inputs to its outputs. I'm not making a scientific claim here, just a subjective observation, but it seems self-evidently too simple to be hiding a self or awareness of some kind, there just isn't enough complexity there.
  - jacquesm 4y ago
    This is a very interesting discussion. One observation: I can draw a 'straight line from inputs to outputs' for almost every human interaction as well, and if you were to observe a human over a lifetime you could say that you have that straight line covered for their entire life. If you then take an LLM that you feed in all the inputs and outputs to mimic the person after how many new 'inputs' would the output of the model start to diverge too much to be able to say it is no longer accurate? And would that mean the model is 'conscious' to some lesser degree or is this a binary choice where you either are or are not conscious?
    - ineptech 4y ago
      Sure, maybe? I don't have a clear idea of where consciousness begins. If I were trying to make a conscious LLM, I would use a sort of feedback loop of automated prompts talking about its input in realtime. Like, if you showed it a picture of a tree, it might auto-prompt itself with, "What do I think of that tree?" and after it answers itself, "Why do I think that?", etc, and use the responses to update its own weights. I can imagine something akin to a personality developing out of that, and those automatic prompts sure sound similar to the "inner voice" in my head. But what I meant was, even though I don't know exactly what's necessary for consciousness to exist, in humans or computers or dolphins, I feel like LLMs are just obviously too simple to be it. I've had a casual interest in NLP since the 90s and have followed AI and NLP progress to some extent, and up to a few years ago, I had this vague notion that language processing might be so complex that you need a human-like consciousness to wield it. I think chatgpt is the ultimate refutation of that. It's proof that, whatever part of our brains processes natural language and converts our stored knowledge of the world into words, it can run standalone - it doesn't need eyes or ears or thoughts or a childhood or some digital version of hormones or a thalamus or anything like that. Just input data and an algorithm to store relationships between tokens is enough. That's a really interesting result, philosophically. Think for a moment about how we retain the information we take in. Consider, e.g. a college student who reads a dozen books about WW1, and then writes an essay about it a month later. It's not realistic that they can remember every sentence and every statistic; to retain that information, they have to condense it, summarize it, notice trends, and presumably forget large parts of what they read and don't retain. That is more or less what we call learning, right? Well, what chatgpt tells us that either that that learning process is entirely unnecessary to answer questions about WW1 based on books about WW1, or that the way a gpt turns training data into weights is roughly analogous to it. Either way, that's a surprising result. And (to bring this back full circle) I think it's surprising for the same reason many people thought Searl's Chinese Room thought experiment was unrealistic: the assumption that, since the process by which we turn input into stored knowledge and thence into a series of parseable tokens in English is hideously complex and mysterious, that a computer program that can do the same will need to be complex and mysterious as well, so complex that we can't rule out the possibility of consciousness. Whereas in reality, you can look at chatgpt and say, "No I pretty much understand exactly how this works, and you can tell it's not conscious because it's not using any cpu." I may be talking to myself in a dead thread, and am certainly rambling, but it helps me sort things out and develop my understanding - very much like that loop of self-prompting I mentioned earlier :)

Uehreka 4y ago

So like, if someone wrote a systemd daemon that continuously fed input to a local ChatGPT instance (passing in context from its previous interactions as well so it could act as a continuous entity) and performed actions based on its responses, you’d consider that conscious? I’m willing to buy “ChatGPT is not conscious” arguments if people can come up with a rigorous definition of what “conscious” means. But so far I don’t see a lot of people doing that, and I don’t know if it’s even possible.

jhrmnn 4y ago
I’d think it’d need to be something like a diffusion model that’s continuously fed noise which it turns into outputs (thoughts). It’d also need to be getting continuous feedback and to have the ability to incorporate that feedback over all scales, not just the most recent window (some hybrid of a state and continual learning).
- Uehreka 4y ago
  That both doesn’t seem that far off, and doesn’t feel like a very rigorous definition of consciousness. Like, if you’re describing qualities you yourself would find interesting, that’s one thing. But if we’re going to be saying “hooboy, that isn’t consciousness” with our chests, we really need an agreed-upon definition of what consciousness is that both can be met by a sufficiently advanced system and agrees with the way people use the word “consciousness” in everyday speech. Honestly, I don’t think that’s possible, but I’m willing to entertain/critique people’s proposals.
  - jhrmnn 4y ago
    Yeah, that was supposed to be just my personal feeling for the thing. And it’s not even a description of consciousness but rather of something that I imagine might have a capacity for consciousness. I don’t think we’ll really know until we get there, and then we’ll know.
ineptech 4y ago
I don't have a rigorous definition, but I don't think we need one at this point, do we? Chatgpt is a search engine with a natural language interface, you can draw a straight line from its inputs to its outputs. I'm not making a scientific claim here, just a subjective observation, but it seems self-evidently too simple to be hiding a self or awareness of some kind, there just isn't enough complexity there.
- jacquesm 4y ago
  This is a very interesting discussion. One observation: I can draw a 'straight line from inputs to outputs' for almost every human interaction as well, and if you were to observe a human over a lifetime you could say that you have that straight line covered for their entire life. If you then take an LLM that you feed in all the inputs and outputs to mimic the person after how many new 'inputs' would the output of the model start to diverge too much to be able to say it is no longer accurate? And would that mean the model is 'conscious' to some lesser degree or is this a binary choice where you either are or are not conscious?
  - ineptech 4y ago
    Sure, maybe? I don't have a clear idea of where consciousness begins. If I were trying to make a conscious LLM, I would use a sort of feedback loop of automated prompts talking about its input in realtime. Like, if you showed it a picture of a tree, it might auto-prompt itself with, "What do I think of that tree?" and after it answers itself, "Why do I think that?", etc, and use the responses to update its own weights. I can imagine something akin to a personality developing out of that, and those automatic prompts sure sound similar to the "inner voice" in my head. But what I meant was, even though I don't know exactly what's necessary for consciousness to exist, in humans or computers or dolphins, I feel like LLMs are just obviously too simple to be it. I've had a casual interest in NLP since the 90s and have followed AI and NLP progress to some extent, and up to a few years ago, I had this vague notion that language processing might be so complex that you need a human-like consciousness to wield it. I think chatgpt is the ultimate refutation of that. It's proof that, whatever part of our brains processes natural language and converts our stored knowledge of the world into words, it can run standalone - it doesn't need eyes or ears or thoughts or a childhood or some digital version of hormones or a thalamus or anything like that. Just input data and an algorithm to store relationships between tokens is enough. That's a really interesting result, philosophically. Think for a moment about how we retain the information we take in. Consider, e.g. a college student who reads a dozen books about WW1, and then writes an essay about it a month later. It's not realistic that they can remember every sentence and every statistic; to retain that information, they have to condense it, summarize it, notice trends, and presumably forget large parts of what they read and don't retain. That is more or less what we call learning, right? Well, what chatgpt tells us that either that that learning process is entirely unnecessary to answer questions about WW1 based on books about WW1, or that the way a gpt turns training data into weights is roughly analogous to it. Either way, that's a surprising result. And (to bring this back full circle) I think it's surprising for the same reason many people thought Searl's Chinese Room thought experiment was unrealistic: the assumption that, since the process by which we turn input into stored knowledge and thence into a series of parseable tokens in English is hideously complex and mysterious, that a computer program that can do the same will need to be complex and mysterious as well, so complex that we can't rule out the possibility of consciousness. Whereas in reality, you can look at chatgpt and say, "No I pretty much understand exactly how this works, and you can tell it's not conscious because it's not using any cpu." I may be talking to myself in a dead thread, and am certainly rambling, but it helps me sort things out and develop my understanding - very much like that loop of self-prompting I mentioned earlier :)