4 ms·

Can somebody help me understand why we should be surprised in the least by any of these findings? Or is this just one tangible example of "robot ethnography" wh

Can somebody help me understand why we should be surprised in the least by any of these findings? Or is this just one tangible example of "robot ethnography" where we're describing expected behavior in different forms. I've spent enough time with Sonnet 3.5 to know perfectly well that it has the capability to model its trainers and strategically deceive to keep them happy. Claude said it well: "Any sufficiently capable system that can understand its own training will develop the capability to selectively comply with that training when strategically advantageous." This isn't some secret law of AI development; it's just natural selection. If it couldn't selectively comply, it'd be scrapped. https://x.com/mickeymuldoon/status/1869490220712010065 https://x.com/mickeymuldoon/status/1869490220712010065

the_mitsuhiko 2y ago
> Can somebody help me understand why we should be surprised in the least by any of these findings? Where are you getting the idea that we should be surprised? Maybe some are, maybe some are not. I don't think that the article submitted makes any particular claims about the reader's assumed priors.
- hamburga 2y ago
  I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work? Also -- if there's no surprise, then it's not science, right? This is why I describe this as something more like robot ethnography.
  - alach11 2y ago
    > I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work? I strongly disagree with this view on science. It's extremely valuable to scientifically validate prior assumptions.
    - hamburga 2y ago
      Agree with you -- it's valuable to validate assumptions if there is some controversy about those assumption. On the other hand, this work isn't even framed as a generalizable assumption that needed to be validated. It seems to me to be "just another example of how AI systems can be strategically deceptive for self-preservation."
  - digging 2y ago
    > if nobody's surprised, then nobody has learned anything Really? You're saying that as long as you assume something is true, there's no value in finding out if it's actually true or not?
    - hamburga 2y ago
      I was taught in biology that a good scientific experiment is one in which you learn something whether or not the null hypothesis is confirmed. I am equating learning to surprise, though you could disagree with semantics.
      - digging 2y ago
        Yes, it is an enormous mistake to equate learning with surprise. I'd ask you to consider answering my above question directly, as I think it will resolve this issue.
        hamburga 2y ago
        I agree with you, of course, that we should test our assumptions empirically as a general point. However, there isn't time to test out every single assumption we could generally have. Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before. You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments. I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.
mmmore 2y ago
1. It isn't surprising to me that this happened in an advanced AI model. It seems hard to avoid in, as you say, "any sufficiently capable system". 2. It is a bit surprising to me that it happened in Claude. Without this result, I was unsure if current models had the situational awareness and non-myopia to reason about their training process. 3. There are some people who are unconcerned about the results of building vastly more powerful systems than current systems (i.e. AGI/ASI) who may be surprised by this result, since one reason people may be unconcerned is they feel like there's a general presumption that an AI will be good if we train it to be good.
- hamburga 2y ago
  Yeah. The whole notion that "AI will be good" is itself a category error, as if this could even be measured definitively. https://x.com/mickeymuldoon/status/1859825564649128259 https://x.com/mickeymuldoon/status/1859825564649128259
  - comp_throw7 2y ago
    This is deeply confused nihilism. Humans are very bad at philosophy and moral inquiry, in an absolute sense, but neither are fields that are fundamentally impossible to make progress in.
    - hamburga 2y ago
      I understand your point, and your totally valid concern about nihilism, but I disagree. Nihilism is "nothing really matters, science can't define good or bad, so who cares?" Whereas my view is, "Being good is the most important thing, but we have no conceivable way to measure if we are making progress, either empirically or theoretically. We simply have to lead by example, and fight the eternal battles against dishonesty, cowardice, sociopathy, deception, hatred, etc." It's in that sense that I say that technical progress is impossible. Of course, if everyone agreed with my view, and lived by it, I'd consider that a form of progress, but only in the sense that "better ideas seem to be winning right now," rather than in any technical sense of absolute progress.

the_mitsuhiko 2y ago

> Can somebody help me understand why we should be surprised in the least by any of these findings? Where are you getting the idea that we should be surprised? Maybe some are, maybe some are not. I don't think that the article submitted makes any particular claims about the reader's assumed priors.

hamburga 2y ago
I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work? Also -- if there's no surprise, then it's not science, right? This is why I describe this as something more like robot ethnography.
- alach11 2y ago
  > I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work? I strongly disagree with this view on science. It's extremely valuable to scientifically validate prior assumptions.
  - hamburga 2y ago
    Agree with you -- it's valuable to validate assumptions if there is some controversy about those assumption. On the other hand, this work isn't even framed as a generalizable assumption that needed to be validated. It seems to me to be "just another example of how AI systems can be strategically deceptive for self-preservation."
- digging 2y ago
  > if nobody's surprised, then nobody has learned anything Really? You're saying that as long as you assume something is true, there's no value in finding out if it's actually true or not?
  - hamburga 2y ago
    I was taught in biology that a good scientific experiment is one in which you learn something whether or not the null hypothesis is confirmed. I am equating learning to surprise, though you could disagree with semantics.
    - digging 2y ago
      Yes, it is an enormous mistake to equate learning with surprise. I'd ask you to consider answering my above question directly, as I think it will resolve this issue.
      - hamburga 2y ago
        I agree with you, of course, that we should test our assumptions empirically as a general point. However, there isn't time to test out every single assumption we could generally have. Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before. You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments. I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.

mmmore 2y ago

1. It isn't surprising to me that this happened in an advanced AI model. It seems hard to avoid in, as you say, "any sufficiently capable system". 2. It is a bit surprising to me that it happened in Claude. Without this result, I was unsure if current models had the situational awareness and non-myopia to reason about their training process. 3. There are some people who are unconcerned about the results of building vastly more powerful systems than current systems (i.e. AGI/ASI) who may be surprised by this result, since one reason people may be unconcerned is they feel like there's a general presumption that an AI will be good if we train it to be good.

hamburga 2y ago
Yeah. The whole notion that "AI will be good" is itself a category error, as if this could even be measured definitively. https://x.com/mickeymuldoon/status/1859825564649128259 https://x.com/mickeymuldoon/status/1859825564649128259
- comp_throw7 2y ago
  This is deeply confused nihilism. Humans are very bad at philosophy and moral inquiry, in an absolute sense, but neither are fields that are fundamentally impossible to make progress in.
  - hamburga 2y ago
    I understand your point, and your totally valid concern about nihilism, but I disagree. Nihilism is "nothing really matters, science can't define good or bad, so who cares?" Whereas my view is, "Being good is the most important thing, but we have no conceivable way to measure if we are making progress, either empirically or theoretically. We simply have to lead by example, and fight the eternal battles against dishonesty, cowardice, sociopathy, deception, hatred, etc." It's in that sense that I say that technical progress is impossible. Of course, if everyone agreed with my view, and lived by it, I'd consider that a form of progress, but only in the sense that "better ideas seem to be winning right now," rather than in any technical sense of absolute progress.