5 ms·

I don't have a rigorous definition, but I don't think we need one at this point, do we? Chatgpt is a search engine with a natural language interface, you can d

I don't have a rigorous definition, but I don't think we need one at this point, do we? Chatgpt is a search engine with a natural language interface, you can draw a straight line from its inputs to its outputs. I'm not making a scientific claim here, just a subjective observation, but it seems self-evidently too simple to be hiding a self or awareness of some kind, there just isn't enough complexity there.

jacquesm 3y ago
This is a very interesting discussion. One observation: I can draw a 'straight line from inputs to outputs' for almost every human interaction as well, and if you were to observe a human over a lifetime you could say that you have that straight line covered for their entire life. If you then take an LLM that you feed in all the inputs and outputs to mimic the person after how many new 'inputs' would the output of the model start to diverge too much to be able to say it is no longer accurate? And would that mean the model is 'conscious' to some lesser degree or is this a binary choice where you either are or are not conscious?
- ineptech 3y ago
  Sure, maybe? I don't have a clear idea of where consciousness begins. If I were trying to make a conscious LLM, I would use a sort of feedback loop of automated prompts talking about its input in realtime. Like, if you showed it a picture of a tree, it might auto-prompt itself with, "What do I think of that tree?" and after it answers itself, "Why do I think that?", etc, and use the responses to update its own weights. I can imagine something akin to a personality developing out of that, and those automatic prompts sure sound similar to the "inner voice" in my head. But what I meant was, even though I don't know exactly what's necessary for consciousness to exist, in humans or computers or dolphins, I feel like LLMs are just obviously too simple to be it. I've had a casual interest in NLP since the 90s and have followed AI and NLP progress to some extent, and up to a few years ago, I had this vague notion that language processing might be so complex that you need a human-like consciousness to wield it. I think chatgpt is the ultimate refutation of that. It's proof that, whatever part of our brains processes natural language and converts our stored knowledge of the world into words, it can run standalone - it doesn't need eyes or ears or thoughts or a childhood or some digital version of hormones or a thalamus or anything like that. Just input data and an algorithm to store relationships between tokens is enough. That's a really interesting result, philosophically. Think for a moment about how we retain the information we take in. Consider, e.g. a college student who reads a dozen books about WW1, and then writes an essay about it a month later. It's not realistic that they can remember every sentence and every statistic; to retain that information, they have to condense it, summarize it, notice trends, and presumably forget large parts of what they read and don't retain. That is more or less what we call learning, right? Well, what chatgpt tells us that either that that learning process is entirely unnecessary to answer questions about WW1 based on books about WW1, or that the way a gpt turns training data into weights is roughly analogous to it. Either way, that's a surprising result. And (to bring this back full circle) I think it's surprising for the same reason many people thought Searl's Chinese Room thought experiment was unrealistic: the assumption that, since the process by which we turn input into stored knowledge and thence into a series of parseable tokens in English is hideously complex and mysterious, that a computer program that can do the same will need to be complex and mysterious as well, so complex that we can't rule out the possibility of consciousness. Whereas in reality, you can look at chatgpt and say, "No I pretty much understand exactly how this works, and you can tell it's not conscious because it's not using any cpu." I may be talking to myself in a dead thread, and am certainly rambling, but it helps me sort things out and develop my understanding - very much like that loop of self-prompting I mentioned earlier :)
  - jacquesm 3y ago
    Certainly not a dead thread and thank you for your long and well reasoned comment. It reflects a chunk of a conversation I had with some colleagues a few days ago. And the 'Chinese Room' turned out to be very feasible, much more feasible than people ever thought it would, probably including Searle (who is still alive, I really wonder if there is any text from him post ChatGPT that gives his views).

jacquesm 3y ago

This is a very interesting discussion. One observation: I can draw a 'straight line from inputs to outputs' for almost every human interaction as well, and if you were to observe a human over a lifetime you could say that you have that straight line covered for their entire life. If you then take an LLM that you feed in all the inputs and outputs to mimic the person after how many new 'inputs' would the output of the model start to diverge too much to be able to say it is no longer accurate? And would that mean the model is 'conscious' to some lesser degree or is this a binary choice where you either are or are not conscious?

ineptech 3y ago
Sure, maybe? I don't have a clear idea of where consciousness begins. If I were trying to make a conscious LLM, I would use a sort of feedback loop of automated prompts talking about its input in realtime. Like, if you showed it a picture of a tree, it might auto-prompt itself with, "What do I think of that tree?" and after it answers itself, "Why do I think that?", etc, and use the responses to update its own weights. I can imagine something akin to a personality developing out of that, and those automatic prompts sure sound similar to the "inner voice" in my head. But what I meant was, even though I don't know exactly what's necessary for consciousness to exist, in humans or computers or dolphins, I feel like LLMs are just obviously too simple to be it. I've had a casual interest in NLP since the 90s and have followed AI and NLP progress to some extent, and up to a few years ago, I had this vague notion that language processing might be so complex that you need a human-like consciousness to wield it. I think chatgpt is the ultimate refutation of that. It's proof that, whatever part of our brains processes natural language and converts our stored knowledge of the world into words, it can run standalone - it doesn't need eyes or ears or thoughts or a childhood or some digital version of hormones or a thalamus or anything like that. Just input data and an algorithm to store relationships between tokens is enough. That's a really interesting result, philosophically. Think for a moment about how we retain the information we take in. Consider, e.g. a college student who reads a dozen books about WW1, and then writes an essay about it a month later. It's not realistic that they can remember every sentence and every statistic; to retain that information, they have to condense it, summarize it, notice trends, and presumably forget large parts of what they read and don't retain. That is more or less what we call learning, right? Well, what chatgpt tells us that either that that learning process is entirely unnecessary to answer questions about WW1 based on books about WW1, or that the way a gpt turns training data into weights is roughly analogous to it. Either way, that's a surprising result. And (to bring this back full circle) I think it's surprising for the same reason many people thought Searl's Chinese Room thought experiment was unrealistic: the assumption that, since the process by which we turn input into stored knowledge and thence into a series of parseable tokens in English is hideously complex and mysterious, that a computer program that can do the same will need to be complex and mysterious as well, so complex that we can't rule out the possibility of consciousness. Whereas in reality, you can look at chatgpt and say, "No I pretty much understand exactly how this works, and you can tell it's not conscious because it's not using any cpu." I may be talking to myself in a dead thread, and am certainly rambling, but it helps me sort things out and develop my understanding - very much like that loop of self-prompting I mentioned earlier :)
- jacquesm 3y ago
  Certainly not a dead thread and thank you for your long and well reasoned comment. It reflects a chunk of a conversation I had with some colleagues a few days ago. And the 'Chinese Room' turned out to be very feasible, much more feasible than people ever thought it would, probably including Searle (who is still alive, I really wonder if there is any text from him post ChatGPT that gives his views).