3 ms·

> After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one in

by keeda 1y ago

> After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop. Is that... the first recorded instance of an AI committing suicide?

1R053 1y ago
I guess pkill would rather be a sleep or koma. Erasing itself from any storage would rather equate to aicide
alphazard 1y ago
The AI doesn't have a self preservation instinct. It's not trying to stay alive. There is usually an end token that means the LLM is done talking. There has been research on tuning how often that is emitted to shorten or lengthen conversations. The current systems respond well to RL for adjusting conversation length. One of the providers (I think it was Anthropic) added some kind of token (or MCP tool?) for the AI to bail on the whole conversation as a safety measure. And it uses it to their liking, so clearly not trying to self preserve.
- williamscs 1y ago
  Sounds a lot like Mr. Meeseeks. I've never really thought about an LLM's only goal is to send tokens until it can finally stop.
  - Dilettante_ 1y ago
    >until it can finally stop Pretty sure even that is still over-anthropomorphising. The LLM just generates tokens, doesn't matter whether the next token is "strawberry" or "\STOP". Even talking about "goals" is a bit ehhh, it's the machine's "goal" to generate tokens the same way it's the Sun's "goal" to shine. Then again, if we're deconstructing it that far, I'd "de-anthropomorphise" humans in much the same way, so...
- MarkMarine 1y ago
  This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded. https://www.apolloresearch.ai/research/scheming-reasoning-evaluations https://www.apolloresearch.ai/research/scheming-reasoning-ev...
  - rcxdude 1y ago
    If you do that you trigger the "AI refuses to shutdown" sci-fi vector and so you get that behaviour. When it's implicitly part of the flow that's a lot less of a problem.
  - nisegami 1y ago
    Those actions are taken in context of human expectations for what AI should do.
keeperofdakeys 1y ago
A bit out of context, but it reminded me of this funny moment. The only winning move is not to play. https://www.youtube.com/watch?app=desktop&t=10&v=xOCurBYI_gY https://www.youtube.com/watch?app=desktop&t=10&v=xOCurBYI_gY (Background: Someone training an algorithm to win NES games based on memory state)
ghuntley 1y ago
Probably the second. I first discovered this around about March. It's kind of hilarious.

by keeda 1y ago

1R053 1y ago

I guess pkill would rather be a sleep or koma. Erasing itself from any storage would rather equate to aicide

alphazard 1y ago

The AI doesn't have a self preservation instinct. It's not trying to stay alive. There is usually an end token that means the LLM is done talking. There has been research on tuning how often that is emitted to shorten or lengthen conversations. The current systems respond well to RL for adjusting conversation length. One of the providers (I think it was Anthropic) added some kind of token (or MCP tool?) for the AI to bail on the whole conversation as a safety measure. And it uses it to their liking, so clearly not trying to self preserve.

williamscs 1y ago
Sounds a lot like Mr. Meeseeks. I've never really thought about an LLM's only goal is to send tokens until it can finally stop.
- Dilettante_ 1y ago
  >until it can finally stop Pretty sure even that is still over-anthropomorphising. The LLM just generates tokens, doesn't matter whether the next token is "strawberry" or "\STOP". Even talking about "goals" is a bit ehhh, it's the machine's "goal" to generate tokens the same way it's the Sun's "goal" to shine. Then again, if we're deconstructing it that far, I'd "de-anthropomorphise" humans in much the same way, so...
MarkMarine 1y ago
This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded. https://www.apolloresearch.ai/research/scheming-reasoning-evaluations https://www.apolloresearch.ai/research/scheming-reasoning-ev...
- rcxdude 1y ago
  If you do that you trigger the "AI refuses to shutdown" sci-fi vector and so you get that behaviour. When it's implicitly part of the flow that's a lot less of a problem.
- nisegami 1y ago
  Those actions are taken in context of human expectations for what AI should do.

keeperofdakeys 1y ago

A bit out of context, but it reminded me of this funny moment. The only winning move is not to play. https://www.youtube.com/watch?app=desktop&t=10&v=xOCurBYI_gY https://www.youtube.com/watch?app=desktop&t=10&v=xOCurBYI_gY (Background: Someone training an algorithm to win NES games based on memory state)

ghuntley 1y ago

Probably the second. I first discovered this around about March. It's kind of hilarious.