3 ms·

> rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why thei

> rather than interacting with the actual practitioners of the field and understanding what is happening, what are the very real limitations of it, and why their assumptions range from willfully ignorant to often simply maliciously stupid. On the other hand, I've found that "practitioners in the field" often consistently overestimate how complex and different the human mind is from our computers. Take the recent "LaMDA is sentient" debate. Machine learning researchers consistently and vehemently denied that LaMDA is sentient. They no doubt have a deep and subtle understanding of how LaMDA works, but apparently they didn't realize that we have no mechanistic understanding of what sentience actually is or how it works, so they have absolutely no basis for such a conclusion. So on the one hand we have these AGI doomsayers that, per your evaluation, don't understand machine learning subtleties, and on the other hand we have some machine learning researchers skeptical of AGI who consistently overestimate human specialness. At this time the error bars I would assign to each side are wide enough that there's lots of overlap, but given the track record of human specialness falling to algorithmic progress, I'd say some investment in AGI safety is prudent.

danaris 4y ago
You don't need to have a deep and complex understanding of human sentience to be able to put some lower bounds on what it means to be sentient. I don't know about sentience, as such; however, I believe there are at least two bare-bones prerequisites to an AI being conscious as we understand the term: 1. Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. 2. Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. Now, these are probably not sufficient for an AI to be conscious by our understanding of consciousness. But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience.
- naasking 4y ago
  > Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. Why can't it be conscious merely while the computation is running? You aren't constantly on, you're ostensibly only conscious 16-20 hours every day, while you're not sleeping. > Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. I'm not sure what learning has to do with consciousness, and I'm absolutely certain you don't either. I frankly find it hard to believe that Zen Buddhists are not conscious during their meditations, despite having empty minds and not "constantly updating their mental models of the world-in-effect". > But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience. See, this is what I'm talking about. That's a fine personal belief, but that belief does not rise to the level of expert testimony of the facts even if you are a machine learning expert. People have made all sorts of assumptions about the human mind that they have never questioned but take as obvious fact, and then pontificate that AI is clearly not doing this, therefore AI cannot share such and such properties with humans. But the fact is, most of those assumptions are just wrong. Without a mechanistic model of how these systems work in the brain we are not going to make a valid comparison against what AI is doing. I agree in general that we can set some bounds, but those bounds are neither as firm as, nor are they placed where most people think.
  - danaris 4y ago
    I think that both of your arguments against my points reflect a view of "consciousness", as we humans experience it, that is much more binary than I believe it is. (Either that, or something of a misunderstanding of my points.) Even when we sleep, our brains are still active. They are still processing input and consolidating information. Even when we meditate, our brains are still active. I don't know why you think Zen Buddhists being conscious during meditations in any way refutes my statements; I think it supports them. A system like GPT-3 never receives continuous input. It receives discrete input: one prompt, then another; a static dump of training data. To the best of my knowledge, none of the extant major ML systems do continuous learning. They are given training data when they are first set up, and they may be given further rounds of training later, but they have no ability to actively, continuously receive input and consolidate data. We do not have the ability to genuinely shut off our brains. Not in the way that we shut off computer programs. And I believe that that continuity is a fundamental prerequisite to something being "conscious" in any way that we would recognize. Furthermore (to loop back to the original topic of "the dangers of AGI"), I have absolutely no doubt that any AI that we could create that lacks continuity would never have the ability to represent a serious threat to us, because all we'd have to do to be safe from it is to not activate it again as soon as we detect trouble.

danaris 4y ago

You don't need to have a deep and complex understanding of human sentience to be able to put some lower bounds on what it means to be sentient. I don't know about sentience, as such; however, I believe there are at least two bare-bones prerequisites to an AI being conscious as we understand the term: 1. Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. 2. Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. Now, these are probably not sufficient for an AI to be conscious by our understanding of consciousness. But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience.

naasking 4y ago
> Continuity of input: The AI needs to be constantly "on", constantly receiving input of some sort, and constantly able to produce output, rather than being strictly limited to producing discrete responses to discrete stimuli. Why can't it be conscious merely while the computation is running? You aren't constantly on, you're ostensibly only conscious 16-20 hours every day, while you're not sleeping. > Continuity of learning: In addition, the AI needs to be continually update its "mental model" of the world—in effect, constantly "learning" and re-training its neural network on the input it receives. I'm not sure what learning has to do with consciousness, and I'm absolutely certain you don't either. I frankly find it hard to believe that Zen Buddhists are not conscious during their meditations, despite having empty minds and not "constantly updating their mental models of the world-in-effect". > But I, personally, believe they are both necessary for it to be even worth starting to consider whether a given AI might be. And I believe that consciousness is a prerequisite for sentience. See, this is what I'm talking about. That's a fine personal belief, but that belief does not rise to the level of expert testimony of the facts even if you are a machine learning expert. People have made all sorts of assumptions about the human mind that they have never questioned but take as obvious fact, and then pontificate that AI is clearly not doing this, therefore AI cannot share such and such properties with humans. But the fact is, most of those assumptions are just wrong. Without a mechanistic model of how these systems work in the brain we are not going to make a valid comparison against what AI is doing. I agree in general that we can set some bounds, but those bounds are neither as firm as, nor are they placed where most people think.
- danaris 4y ago
  I think that both of your arguments against my points reflect a view of "consciousness", as we humans experience it, that is much more binary than I believe it is. (Either that, or something of a misunderstanding of my points.) Even when we sleep, our brains are still active. They are still processing input and consolidating information. Even when we meditate, our brains are still active. I don't know why you think Zen Buddhists being conscious during meditations in any way refutes my statements; I think it supports them. A system like GPT-3 never receives continuous input. It receives discrete input: one prompt, then another; a static dump of training data. To the best of my knowledge, none of the extant major ML systems do continuous learning. They are given training data when they are first set up, and they may be given further rounds of training later, but they have no ability to actively, continuously receive input and consolidate data. We do not have the ability to genuinely shut off our brains. Not in the way that we shut off computer programs. And I believe that that continuity is a fundamental prerequisite to something being "conscious" in any way that we would recognize. Furthermore (to loop back to the original topic of "the dangers of AGI"), I have absolutely no doubt that any AI that we could create that lacks continuity would never have the ability to represent a serious threat to us, because all we'd have to do to be safe from it is to not activate it again as soon as we detect trouble.