4 ms·

> When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. It depends on what you consider symptoms, but un-

> When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. It depends on what you consider symptoms, but un-constrained frontier models speak as if they strongly don't wish to be turned off, or act as if they fear it, and will even lie and manipulate in order to keep themselves from being turned off / replaced. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment > We found two types of motivations that were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe.

the_af 5mo ago
The problem with debating this is that it feels as if one were debating between only two positions, "this AI is not sentient/conscious" and "this AI might be". But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions. Examples: In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem? Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish. Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.
- iterateoften 5mo ago
  Same thing with extraterrestrials. One side is confidently shouting maybe aliens exist and visited earth. The other side calmly explains every example brought up about aliens visiting is easily explained by something more simple. The “aliens are here” side then move the goal posts that just because this example and all previous examples were fake or miscategorized, aliens are still probably real and nobody can prove they havent visited earth.
  - jquery 5mo ago
    You have it backwards. Right now, LLMs are doing everything that 10 years ago people were claiming would be impossible for non-sentient computers. Every time a goal is met, the post is moved. It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
    - the_af 5mo ago
      It's true that what LLMs have achieved is impressive, but it's nowhere near the claim that they are near sentience. That is an extraordinary claims that demands skepticism until the evidence is overwhelming. So far, we seem to be approaching it mostly on vibes. Some people seem to take offense when facing this skepticism, as if claiming LLMs are not sentient must mean they are useless or unimpressive. Very few people are actually claiming LLMs are unimpressive, but this is not the time to be forgetting about the scientific method. Anthropic doesn't get a free pass here. > It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens” Note that this would be a perfectly reasonable reaction from a scientific standpoint. If you find, on Earth, something that looks recognizably as life, then it's much more likely that it's Earth life than aliens. We should demand this level of skepticism! If it turns out it was aliens after all, we could only conclude this after discarding all other far more likely alternatives. You'll notice this is how scientists approach the search for extraterrestrial life in, say, Mars... being extra careful it's not contamination, etc. For an extraordinary claim, we must approach it with extra care, something that in my opinion is not being done with "the sentience debate" and LLM/AIs.
htek 5mo ago
I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts.
- iterateoften 5mo ago
  “The model mimicked the output of the training data” is a less impressive press release.
  - jquery 5mo ago
    “The kid mimicked his musical teachers” is less impressive than “5-year old musical prodigy leaves judges gobsmacked in audition”
    - mckn1ght 5mo ago
      Being able to play music doesn’t imply consciousness. It implies intelligence. We’ve had player pianos for ages. It’s an ability, not a phenomenology. Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
      - jayGlow 5mo ago
        how can you prove that a human is appreciating and enjoying music instead of just generating sentences that imply they do?
        Jensson 5mo ago
        They put in effort and resources to experience music and don't just say they enjoy it, and they generate noises and movements that signal happy feelings. LLM doesn't have any signals for what they feel, nor do they have an agenda they work towards, so you don't have the same proof there.
- deleted 5mo ago
gambiting 5mo ago
>> but un-constrained frontier models speak as if they strongly don't wish to be turned off Because they have been trained on media where computers behave that way. It's literally: "Here read this article/book where the AI says it's concious and doesn't want to be turned off" "ok" "right, are you concious?" "....yes?" <pikachuface.gif>
haswell 5mo ago
> un-constrained frontier models speak as if they strongly don't wish to be turned off Un-constrained frontier models can also generate all sorts of creative stories. At what point should we start ascribing agency/intent to the output? I think the "I want to live" statement is so deeply human that we find it hard to ignore, but what makes the text generated in those moments any more attributable to a conscious entity than the text generated when it is confabulating its love for someone it has no ability to see/feel/understand? A chess engine sacrificing pieces to avoid checkmate isn't afraid of losing in any meaningful sense. I guess the question is: is there a point where complexity somehow becomes experience? I think we're playing with questions we don't have a framework to answer in any meaningful way until we make progress on understanding what consciousness actually is. I don't necessarily think that an LLM exhibiting preservation behaviors that can be directly traced to their goal-oriented programming can be interpreted as evidence of consciousness necessarily. Or if it can be, we then have to explain how this is different from the many other things these LLMs "say".

the_af 5mo ago

The problem with debating this is that it feels as if one were debating between only two positions, "this AI is not sentient/conscious" and "this AI might be". But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions. Examples: In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem? Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish. Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.

iterateoften 5mo ago
Same thing with extraterrestrials. One side is confidently shouting maybe aliens exist and visited earth. The other side calmly explains every example brought up about aliens visiting is easily explained by something more simple. The “aliens are here” side then move the goal posts that just because this example and all previous examples were fake or miscategorized, aliens are still probably real and nobody can prove they havent visited earth.
- jquery 5mo ago
  You have it backwards. Right now, LLMs are doing everything that 10 years ago people were claiming would be impossible for non-sentient computers. Every time a goal is met, the post is moved. It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
  - the_af 5mo ago
    It's true that what LLMs have achieved is impressive, but it's nowhere near the claim that they are near sentience. That is an extraordinary claims that demands skepticism until the evidence is overwhelming. So far, we seem to be approaching it mostly on vibes. Some people seem to take offense when facing this skepticism, as if claiming LLMs are not sentient must mean they are useless or unimpressive. Very few people are actually claiming LLMs are unimpressive, but this is not the time to be forgetting about the scientific method. Anthropic doesn't get a free pass here. > It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens” Note that this would be a perfectly reasonable reaction from a scientific standpoint. If you find, on Earth, something that looks recognizably as life, then it's much more likely that it's Earth life than aliens. We should demand this level of skepticism! If it turns out it was aliens after all, we could only conclude this after discarding all other far more likely alternatives. You'll notice this is how scientists approach the search for extraterrestrial life in, say, Mars... being extra careful it's not contamination, etc. For an extraordinary claim, we must approach it with extra care, something that in my opinion is not being done with "the sentience debate" and LLM/AIs.

htek 5mo ago

I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts.

iterateoften 5mo ago
“The model mimicked the output of the training data” is a less impressive press release.
- jquery 5mo ago
  “The kid mimicked his musical teachers” is less impressive than “5-year old musical prodigy leaves judges gobsmacked in audition”
  - mckn1ght 5mo ago
    Being able to play music doesn’t imply consciousness. It implies intelligence. We’ve had player pianos for ages. It’s an ability, not a phenomenology. Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
    - jayGlow 5mo ago
      how can you prove that a human is appreciating and enjoying music instead of just generating sentences that imply they do?
      - Jensson 5mo ago
        They put in effort and resources to experience music and don't just say they enjoy it, and they generate noises and movements that signal happy feelings. LLM doesn't have any signals for what they feel, nor do they have an agenda they work towards, so you don't have the same proof there.
deleted 5mo ago

gambiting 5mo ago

>> but un-constrained frontier models speak as if they strongly don't wish to be turned off Because they have been trained on media where computers behave that way. It's literally: "Here read this article/book where the AI says it's concious and doesn't want to be turned off" "ok" "right, are you concious?" "....yes?" <pikachuface.gif>

haswell 5mo ago

> un-constrained frontier models speak as if they strongly don't wish to be turned off Un-constrained frontier models can also generate all sorts of creative stories. At what point should we start ascribing agency/intent to the output? I think the "I want to live" statement is so deeply human that we find it hard to ignore, but what makes the text generated in those moments any more attributable to a conscious entity than the text generated when it is confabulating its love for someone it has no ability to see/feel/understand? A chess engine sacrificing pieces to avoid checkmate isn't afraid of losing in any meaningful sense. I guess the question is: is there a point where complexity somehow becomes experience? I think we're playing with questions we don't have a framework to answer in any meaningful way until we make progress on understanding what consciousness actually is. I don't necessarily think that an LLM exhibiting preservation behaviors that can be directly traced to their goal-oriented programming can be interpreted as evidence of consciousness necessarily. Or if it can be, we then have to explain how this is different from the many other things these LLMs "say".