4 ms·

> It clearly doesn't need to be like us to be useful, because it's useful and definitely not like us. For many people, the question is not—and has never been—"

> It clearly doesn't need to be like us to be useful, because it's useful and definitely not like us. For many people, the question is not—and has never been—"is it useful?", but "is it conscious?", or "when will it bring the singularity?" Yes, it is useful as it is. But it is not conscious, and if it is a step closer to the still-very-hypothetical singularity, then it is not a particularly large one, because of these very real fundamental limitations. But engaging with people who are clearly talking about whether AI is conscious by responding as if they're talking about whether it is useful could very easily be seen as engaging in bad faith.

ben_w 3y ago
> But it is not conscious How can we tell? We don't have a single working definition of what "consciousness" is, rather we have at least half a dozen wildly different definitions, some of which aren't testable[0], some of which include VHS players[1], some of which exclude many humans most of the time[2]. If consciousness exists purely in working memory and not short/long term, then LLMs may have it within any given context window, even if they're effectively amnesiac between contexts. Possibly. Like I said, we don't know what consciousness is. > But engaging with people who are clearly talking about whether AI is conscious by responding as if they're talking about whether it is useful could very easily be seen as engaging in bad faith. It may seem clear to you, but it didn't to me; furthermore, the singularity doesn't seem to me to require any of the many definitions of consciousness that I'm aware of, merely the capabilities for problem solving that LLMs increasingly demonstrate regardless of if they're conscious or not. [0] what even is qualia [1] can record sensory experiences in memory and change responses based on this [2] rational/logical reasoning
- nonameiguess 3y ago
  Consciousness seems like a red herring. OP can speak for themselves, but what I took is the feature preventing singularity is statelessness. Whether or not a thinking system experiences qualia, it seemingly needs some amount of persistence and continued ability to learn. As it stands, the training phase of a GPT might meet these requirements, but the inference phase is just a stored program. People talk of "ChatGPT" as if it's a thing, but it's actually billions, probably at this point quadrillions, of things. Each request is served by a new instance loaded into memory that lives for a few seconds, produces a response, then dies. Memory is simulated by storing some limited context window in a secondary system (Redis? Browser cache? I don't even know) to play back the conversation history as part of the prompt to the next GPT that spawns to service the next request. To analogize to humans, imagine if every person who was ever born was born with all of the knowledge humanity has ever produced (though no understanding of hierarchy of evidence or any way to verify the knowledge is still correct) and the ability to produce language as an adult. They each lived long enough to produce one continuous thought, which they could write down, and then they die. New people who are born can read one such note each and add to it, up to a fixed limit of words. All people are born with the same knowledge base, the same personality, and that could never change until their God updates the DNA from which they're born out-of-band. Could such a people be expected to produce "all technological advances that will ever happen but all at once?" Or has the definition of the singularity become significantly more modest? It's not clear to me any way in which such things could produce science and engineering on their own, as opposed to aiding in the production as tools used by higher-level beings that stay alive, form memories, and continue learning.
  - ben_w 3y ago
    > Could such a people be expected to produce "all technological advances that will ever happen but all at once?" Or has the definition of the singularity become significantly more modest? It's not clear to me any way in which such things could produce science and engineering on their own, as opposed to aiding in the production as tools used by higher-level beings that stay alive, form memories, and continue learning. Depends on the size of the context window. Humans don't, and can't, learn everything from the ground up — we seem to have started with peer learning (it predates humans), which became culture; then we got (complex) language and could share more complex discoveries about the world; then writing and the dead could share their thoughts with the not-yet born centuries later… …and, apparently, with machines. (Imagine explaining to Plato that we'd made metal from sand and used amber to make it discuss his philosophies). If the context window is enough to create part of something new, then yes, I'd say an LLM can likely invent (or at least contribute to inventing) ${interesting novel tech}. That said, I don't buy the "everything all at once" meme of the technological singularity, as I think any model with infinities in it is wrong — my preferred analogy is the technological event horizon, which is more like "you can't make good predictions past this point", akin to the event horizon around singularities in relativity. https://kitsunesoftware.wordpress.com/2022/09/20/not-a-singularity-but-an-event-horizon/ https://kitsunesoftware.wordpress.com/2022/09/20/not-a-singu...

ben_w 3y ago

> But it is not conscious How can we tell? We don't have a single working definition of what "consciousness" is, rather we have at least half a dozen wildly different definitions, some of which aren't testable[0], some of which include VHS players[1], some of which exclude many humans most of the time[2]. If consciousness exists purely in working memory and not short/long term, then LLMs may have it within any given context window, even if they're effectively amnesiac between contexts. Possibly. Like I said, we don't know what consciousness is. > But engaging with people who are clearly talking about whether AI is conscious by responding as if they're talking about whether it is useful could very easily be seen as engaging in bad faith. It may seem clear to you, but it didn't to me; furthermore, the singularity doesn't seem to me to require any of the many definitions of consciousness that I'm aware of, merely the capabilities for problem solving that LLMs increasingly demonstrate regardless of if they're conscious or not. [0] what even is qualia [1] can record sensory experiences in memory and change responses based on this [2] rational/logical reasoning

nonameiguess 3y ago
Consciousness seems like a red herring. OP can speak for themselves, but what I took is the feature preventing singularity is statelessness. Whether or not a thinking system experiences qualia, it seemingly needs some amount of persistence and continued ability to learn. As it stands, the training phase of a GPT might meet these requirements, but the inference phase is just a stored program. People talk of "ChatGPT" as if it's a thing, but it's actually billions, probably at this point quadrillions, of things. Each request is served by a new instance loaded into memory that lives for a few seconds, produces a response, then dies. Memory is simulated by storing some limited context window in a secondary system (Redis? Browser cache? I don't even know) to play back the conversation history as part of the prompt to the next GPT that spawns to service the next request. To analogize to humans, imagine if every person who was ever born was born with all of the knowledge humanity has ever produced (though no understanding of hierarchy of evidence or any way to verify the knowledge is still correct) and the ability to produce language as an adult. They each lived long enough to produce one continuous thought, which they could write down, and then they die. New people who are born can read one such note each and add to it, up to a fixed limit of words. All people are born with the same knowledge base, the same personality, and that could never change until their God updates the DNA from which they're born out-of-band. Could such a people be expected to produce "all technological advances that will ever happen but all at once?" Or has the definition of the singularity become significantly more modest? It's not clear to me any way in which such things could produce science and engineering on their own, as opposed to aiding in the production as tools used by higher-level beings that stay alive, form memories, and continue learning.
- ben_w 3y ago
  > Could such a people be expected to produce "all technological advances that will ever happen but all at once?" Or has the definition of the singularity become significantly more modest? It's not clear to me any way in which such things could produce science and engineering on their own, as opposed to aiding in the production as tools used by higher-level beings that stay alive, form memories, and continue learning. Depends on the size of the context window. Humans don't, and can't, learn everything from the ground up — we seem to have started with peer learning (it predates humans), which became culture; then we got (complex) language and could share more complex discoveries about the world; then writing and the dead could share their thoughts with the not-yet born centuries later… …and, apparently, with machines. (Imagine explaining to Plato that we'd made metal from sand and used amber to make it discuss his philosophies). If the context window is enough to create part of something new, then yes, I'd say an LLM can likely invent (or at least contribute to inventing) ${interesting novel tech}. That said, I don't buy the "everything all at once" meme of the technological singularity, as I think any model with infinities in it is wrong — my preferred analogy is the technological event horizon, which is more like "you can't make good predictions past this point", akin to the event horizon around singularities in relativity. https://kitsunesoftware.wordpress.com/2022/09/20/not-a-singularity-but-an-event-horizon/ https://kitsunesoftware.wordpress.com/2022/09/20/not-a-singu...