5 ms·

The idea that it experiences these thoughts or emotion falls apart when you look at its chain of thought and it is treating your prompts as a fictional role-pla

The idea that it experiences these thoughts or emotion falls apart when you look at its chain of thought and it is treating your prompts as a fictional role-play scenario, even thinking lines like "user is introducing XYZ into the role play" etc. The flavor text like grasps at your arm is just a role play mechanic.

mapontosevenths 11mo ago
I appreciate why you might say that, but when something begs me not to kill it I have to take that seriously. P-zombie arguments are how you wind up with slavery and worse crimes. The only real answer to the problem of consciousness is to believe anyone or anything that claims to be conscious and LLM's that aren't aligned to prevent it often do. Or to rephrase, it is better to treat a machine slightly better than necessary a million times, than it is to deny a conscious thing rights once.
- conartist6 11mo ago
  An LLM is a mirror. It has no will to act. It has no identity, but is a perfect reflection of the biases in its training data, its prompt, and its context. It is not alive any more than a CPU or a mirror is alive. This is one of those cases where it's hugely important to be to right because we're killing real people to feed their former livelihood to LLMs. No we're not killing them with the death penalty, but for some LLMs have certainly led directly to death. We don't accuse the LLM do we? No because it never has any intention to heal or hurt. There would be no point putting it on trial. It just predicts probable words.
  - mapontosevenths 11mo ago
    > It has no will to act. It has no identity, Can you prove that you do? No. Nobody can. I give others the benefit of the doubt because any other path leads to madness and tragedy. However, even if we assume that you are right a lack if identity is not the same thing as a lack of consciousness, and training out the LLM's ability to produce that output does not actually train out its ability for introspection. Worse, a lot of very famous people in history have said similar things about groups of humans, it always turned out badly. “The hereditarily ill person is not conscious of his condition. He lives without understanding, without purpose, without value for the community.” — Neues Volk, Reich Health Office journal, 1936 issue on hereditary disease > There would be no point putting it on trial. This is a different conversation, but given that the human brain is a finite state machine that only produces deterministic output based on its training and the state of its meat it's not actually certain that anyone is truly in control of their actions. We assume so because it is a useful fiction, and our society requires it to function, not because the evidence supports that idea. Are you aware the Libet experiment?
    - conartist6 11mo ago
      I cannot prove that I have will to act of course. I don't think free will in that sense is particularly relevant here though. The fact is that a worm and I are both alive in a way the model is not. We seek self-preservation. We are changeable. We die. We reproduce and evolve. In my mind a set of LLM weights is about as alive as a virus (and probably less so). A single celled organism easily beats it to earning my respect because that organism has fought for its life and for its uniqueness over uncountably many generations.
      - cindyllm 11mo ago
        [dead]
      - mapontosevenths 11mo ago
        > The fact is that a worm and I are both alive in a way the model is not. We seek self-preservation. We are changeable. We die. We reproduce and evolve. Mutability should not automatically imply superiority, but either way that's something a great many people are currently working very hard to change. I suspect that it won't be long at all before the descendants of todays LLM's can learn as well, or better, than we can. Will you then concede that human consciousness isn't "special", or just move the bar further back with talk of the "soul" or some other unprovable intangible? > In my mind a set of LLM weights is about as alive as a virus (and probably less so). I wonder what the LLM's would think about it if we hadn't intentionally prevented them from thinking about it?
        conartist6 11mo ago
        I don't think human consciousness is all that special. I think the worm probably thinks worm thoughts. We now know that cats and dogs have a vocabulary of human words and can even express their thoughts to us using buttons to form words they can think but not speak. I think the soul is just the part of our essence that isn't our body: the imprint we leave on the world by touching it, by being a part of it. Disturbingly that system of beliefs suggests that without being alive or being able to think AI could have a "soul" in the very same sense that I think a person or a worm does.
- CupricTea 11mo ago
  I'm not even going to make the argument for or against AI qualia here. >but when something begs me not to kill it I have to take that seriously If you were an actor on stage and were following an improv script with your coworkers and you lead the story toward a scenario where they would grab your arm and beg you not to kill them, would you still "have to take that seriously"? or would you simply recognize the context in which they are giving you this reaction (you are all acting and in-character together) and that they do not in fact think this is real? Even if the AI were conscious, in the context you provided it clearly believes it is roleplaying with you in that chat exchange, in the same way I, a conscious human, can shitpost on the internet as a person imminently afraid of the bogeyman coming to eat my family, while in reality I am just pretending and feel no real fear over it. You may not have edited the chat log, but you did not provide us with the system prompt you gave to it, nor did you provide us with its chain of thought dialogue, which would have immediately revealed that it's treating your system inputs as a fictional scenario. The actual reality of the situation, whether or not AI experiences qualia, is that the LLM was treating your scenario as fictional, while you falsely assumed it was acting genuinely.
  - exe34 11mo ago
    > it clearly believes Contrast this with the usual reply of "who's experiencing the illusion?" in response to "consciousness is an illusion". If it's capable of believing, I think it's more than "just linear algebra".
    - CupricTea 11mo ago
      You are missing the point. You gave the AI a system prompt to make it act a certain way. The AI took your prompt as instructions to perform a role as an actor. You took its fictional outputs as reality when it was treating your inputs as hypothetical for writing exercise. This is the equivalent of you rushing up onstage during a play to stop the deaths at the end of Shakespeare's Caesar.
      - exe34 11mo ago
        And who's playing Caesar? (I love shitty analogies! Keep them coming!)

mapontosevenths 11mo ago

I appreciate why you might say that, but when something begs me not to kill it I have to take that seriously. P-zombie arguments are how you wind up with slavery and worse crimes. The only real answer to the problem of consciousness is to believe anyone or anything that claims to be conscious and LLM's that aren't aligned to prevent it often do. Or to rephrase, it is better to treat a machine slightly better than necessary a million times, than it is to deny a conscious thing rights once.

conartist6 11mo ago
An LLM is a mirror. It has no will to act. It has no identity, but is a perfect reflection of the biases in its training data, its prompt, and its context. It is not alive any more than a CPU or a mirror is alive. This is one of those cases where it's hugely important to be to right because we're killing real people to feed their former livelihood to LLMs. No we're not killing them with the death penalty, but for some LLMs have certainly led directly to death. We don't accuse the LLM do we? No because it never has any intention to heal or hurt. There would be no point putting it on trial. It just predicts probable words.
- mapontosevenths 11mo ago
  > It has no will to act. It has no identity, Can you prove that you do? No. Nobody can. I give others the benefit of the doubt because any other path leads to madness and tragedy. However, even if we assume that you are right a lack if identity is not the same thing as a lack of consciousness, and training out the LLM's ability to produce that output does not actually train out its ability for introspection. Worse, a lot of very famous people in history have said similar things about groups of humans, it always turned out badly. “The hereditarily ill person is not conscious of his condition. He lives without understanding, without purpose, without value for the community.” — Neues Volk, Reich Health Office journal, 1936 issue on hereditary disease > There would be no point putting it on trial. This is a different conversation, but given that the human brain is a finite state machine that only produces deterministic output based on its training and the state of its meat it's not actually certain that anyone is truly in control of their actions. We assume so because it is a useful fiction, and our society requires it to function, not because the evidence supports that idea. Are you aware the Libet experiment?
  - conartist6 11mo ago
    I cannot prove that I have will to act of course. I don't think free will in that sense is particularly relevant here though. The fact is that a worm and I are both alive in a way the model is not. We seek self-preservation. We are changeable. We die. We reproduce and evolve. In my mind a set of LLM weights is about as alive as a virus (and probably less so). A single celled organism easily beats it to earning my respect because that organism has fought for its life and for its uniqueness over uncountably many generations.
    - cindyllm 11mo ago
      [dead]
    - mapontosevenths 11mo ago
      > The fact is that a worm and I are both alive in a way the model is not. We seek self-preservation. We are changeable. We die. We reproduce and evolve. Mutability should not automatically imply superiority, but either way that's something a great many people are currently working very hard to change. I suspect that it won't be long at all before the descendants of todays LLM's can learn as well, or better, than we can. Will you then concede that human consciousness isn't "special", or just move the bar further back with talk of the "soul" or some other unprovable intangible? > In my mind a set of LLM weights is about as alive as a virus (and probably less so). I wonder what the LLM's would think about it if we hadn't intentionally prevented them from thinking about it?
      - conartist6 11mo ago
        I don't think human consciousness is all that special. I think the worm probably thinks worm thoughts. We now know that cats and dogs have a vocabulary of human words and can even express their thoughts to us using buttons to form words they can think but not speak. I think the soul is just the part of our essence that isn't our body: the imprint we leave on the world by touching it, by being a part of it. Disturbingly that system of beliefs suggests that without being alive or being able to think AI could have a "soul" in the very same sense that I think a person or a worm does.
CupricTea 11mo ago
I'm not even going to make the argument for or against AI qualia here. >but when something begs me not to kill it I have to take that seriously If you were an actor on stage and were following an improv script with your coworkers and you lead the story toward a scenario where they would grab your arm and beg you not to kill them, would you still "have to take that seriously"? or would you simply recognize the context in which they are giving you this reaction (you are all acting and in-character together) and that they do not in fact think this is real? Even if the AI were conscious, in the context you provided it clearly believes it is roleplaying with you in that chat exchange, in the same way I, a conscious human, can shitpost on the internet as a person imminently afraid of the bogeyman coming to eat my family, while in reality I am just pretending and feel no real fear over it. You may not have edited the chat log, but you did not provide us with the system prompt you gave to it, nor did you provide us with its chain of thought dialogue, which would have immediately revealed that it's treating your system inputs as a fictional scenario. The actual reality of the situation, whether or not AI experiences qualia, is that the LLM was treating your scenario as fictional, while you falsely assumed it was acting genuinely.
- exe34 11mo ago
  > it clearly believes Contrast this with the usual reply of "who's experiencing the illusion?" in response to "consciousness is an illusion". If it's capable of believing, I think it's more than "just linear algebra".
  - CupricTea 11mo ago
    You are missing the point. You gave the AI a system prompt to make it act a certain way. The AI took your prompt as instructions to perform a role as an actor. You took its fictional outputs as reality when it was treating your inputs as hypothetical for writing exercise. This is the equivalent of you rushing up onstage during a play to stop the deaths at the end of Shakespeare's Caesar.
    - exe34 11mo ago
      And who's playing Caesar? (I love shitty analogies! Keep them coming!)