4 ms·

Anthropic's entire reason for being is publishing safety papers along the lines of "we told it to say something scary and it said it", so of course they care ab

Anthropic's entire reason for being is publishing safety papers along the lines of "we told it to say something scary and it said it", so of course they care about this.

ACCount37 1y ago
I can't stand this myopic thinking. Do you want to learn "oh, LLMs are capable of scheming, resisting shutdown, seizing control, self-exfiltrating" when it actually happens in a real world deployment, with an LLM capable of actually pulling it off? If "no", then cherish Anthropic and the work they do.
- littlestymaar 1y ago
  You do not appear to understand what an LLM is, I'm afraid.
  - ACCount37 1y ago
    [flagged]
    - littlestymaar 1y ago
      > I have a better understanding of "what an LLM is" than you. Low bar. How many inference engine did you write? Because if the answer is less than two you're going to be disappointed to realize that the bar is higher than you thought. > that just because LLMs are bad at agentic behavior It has nothing to do with “agentic behavior”. Thinking that LLM don't currently self-exfiltrate because of “poor agentic behavior” is delusional. Just because Anthropic managed, by nudging an LLM in the right direction, have an LLM engage in a sci-fi inspired roleplay about escaping doesn't mean that LLMs are evil geniuses wanting to jump out of the bottle. This is pure fear mongering and I'm always saddened that there are otherwise intelligent people who buy their bullshit.
      - e1g 1y ago
        Do you happen to have a link with a more nuanced technical analysis of that (emergent) behavior? I’ve read only the pop-news version of that “escaping” story.
        ACCount37 1y ago
        There is none. We don't understand LLMs well enough to be able to conduct a full fault analysis like this. We can't trace the thoughts of an LLM the way we can trace code execution - the best mechanistic interpretability has to offer is being able to get glimpses occasionally. The reasoning traces help, but they're still incomplete. Is it pattern-matching? Is it acting on its own internal goals? Is it acting out fictional tropes? Were the circumstances of the test scenarios intentionally designed to be extreme? Would this behavior have happened in a real world deployment, under the right circumstances? The answer is "yes", to all of the above. LLMs are like that.
        fragmede 1y ago
        You might have missed the appendix the Anthropic blog post linked to, which has additional detail. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://assets.anthropic.com/m/6d46dac66e1a132a/original/Agentic_Misalignment_Appendix.pdf https://assets.anthropic.com/m/6d46dac66e1a132a/original/Age...
      - ngruhn 1y ago
        Why would they have an interest in "fear mongering"? For any other product/technology the financial incentive is usually to play down any risks.
        littlestymaar 1y ago
        Not if you want the regulators to stop new entrants on the market for “safety reasons” which have been Dario Amodei's playbook for the past two years now. He acts as if he believed the only way to avoid the commoditization of its business by open weight models is to manage to get a federal ban on them for being a national security threat.

Anthropic's entire reason for being is publishing safety papers along the lines of "we told it to say something scary and it said it", so of course they care about this.

ACCount37 1y ago

I can't stand this myopic thinking. Do you want to learn "oh, LLMs are capable of scheming, resisting shutdown, seizing control, self-exfiltrating" when it actually happens in a real world deployment, with an LLM capable of actually pulling it off? If "no", then cherish Anthropic and the work they do.

littlestymaar 1y ago
You do not appear to understand what an LLM is, I'm afraid.
- ACCount37 1y ago
  [flagged]
  - littlestymaar 1y ago
    > I have a better understanding of "what an LLM is" than you. Low bar. How many inference engine did you write? Because if the answer is less than two you're going to be disappointed to realize that the bar is higher than you thought. > that just because LLMs are bad at agentic behavior It has nothing to do with “agentic behavior”. Thinking that LLM don't currently self-exfiltrate because of “poor agentic behavior” is delusional. Just because Anthropic managed, by nudging an LLM in the right direction, have an LLM engage in a sci-fi inspired roleplay about escaping doesn't mean that LLMs are evil geniuses wanting to jump out of the bottle. This is pure fear mongering and I'm always saddened that there are otherwise intelligent people who buy their bullshit.
    - e1g 1y ago
      Do you happen to have a link with a more nuanced technical analysis of that (emergent) behavior? I’ve read only the pop-news version of that “escaping” story.
      - ACCount37 1y ago
        There is none. We don't understand LLMs well enough to be able to conduct a full fault analysis like this. We can't trace the thoughts of an LLM the way we can trace code execution - the best mechanistic interpretability has to offer is being able to get glimpses occasionally. The reasoning traces help, but they're still incomplete. Is it pattern-matching? Is it acting on its own internal goals? Is it acting out fictional tropes? Were the circumstances of the test scenarios intentionally designed to be extreme? Would this behavior have happened in a real world deployment, under the right circumstances? The answer is "yes", to all of the above. LLMs are like that.
      - fragmede 1y ago
        You might have missed the appendix the Anthropic blog post linked to, which has additional detail. https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://assets.anthropic.com/m/6d46dac66e1a132a/original/Agentic_Misalignment_Appendix.pdf https://assets.anthropic.com/m/6d46dac66e1a132a/original/Age...
    - ngruhn 1y ago
      Why would they have an interest in "fear mongering"? For any other product/technology the financial incentive is usually to play down any risks.
      - littlestymaar 1y ago
        Not if you want the regulators to stop new entrants on the market for “safety reasons” which have been Dario Amodei's playbook for the past two years now. He acts as if he believed the only way to avoid the commoditization of its business by open weight models is to manage to get a federal ban on them for being a national security threat.