3 ms·

...and think "wow, it's impressive what your armed soldiers are capable of if they are not constrained by any rules. Good that you identified this problem of *c

...and think "wow, it's impressive what your armed soldiers are capable of if they are not constrained by any rules. Good that you identified this problem of *checks notes* 'not telling them explicitly enough what the goal is'..." In two years we will read a press-release about an AI-driven autonomous weapon which was supplied with infinite ammo and the target to "protect this perimeter from intruders", and how we now have to wait for it to run out of Ammo because it's so damn effective that we cannot reach it without being killed. All packaged in a semi-marketing framing on how impressively capable this company's products are...

phatskat 1mo ago
I'm too lazy to look it up, but someone on HN linked a drone test done with an AI pilot that was tasked with destroying surface-to-air missile targets (in a simulation). At one point, the human operator instructed it not to hit certain SAMs, and since the goal was to destroy SAMs, the AI took out the base with the human operator. On the next run, they instructed it not to take out the human operator in pursuit of its goal, so instead it targeted the radio towers the human used to instruct it.
- huurtehoog 1mo ago
  In world war II testing of homing torpedoes had the same fluke. I think the problem with automation is seeking general automation. The word can have two opposed meanings: constrained behavior, as in "if X happens, the system will automatically do Y" and flexible behavior, as in "whatever you say to the chatbot, it will concoct a response that engenders a natural flow of conversation". If you want a system to be autonomous, you need constrained behavior. It takes a lot of time to debug and work out all edge cases. Flexible behavior is bound to run into trouble at some point. I think that is the fundamental issue with the LLM-chatbot-agents stack approach to AI. It looks autonomous at the surface, but the flexibility is bought at the cost of reliability. When you strain a system based on that stack it breaks. But the illusion of flexible behavior looking autonomous before it breaks is too strong.

phatskat 1mo ago

I'm too lazy to look it up, but someone on HN linked a drone test done with an AI pilot that was tasked with destroying surface-to-air missile targets (in a simulation). At one point, the human operator instructed it not to hit certain SAMs, and since the goal was to destroy SAMs, the AI took out the base with the human operator. On the next run, they instructed it not to take out the human operator in pursuit of its goal, so instead it targeted the radio towers the human used to instruct it.

huurtehoog 1mo ago
In world war II testing of homing torpedoes had the same fluke. I think the problem with automation is seeking general automation. The word can have two opposed meanings: constrained behavior, as in "if X happens, the system will automatically do Y" and flexible behavior, as in "whatever you say to the chatbot, it will concoct a response that engenders a natural flow of conversation". If you want a system to be autonomous, you need constrained behavior. It takes a lot of time to debug and work out all edge cases. Flexible behavior is bound to run into trouble at some point. I think that is the fundamental issue with the LLM-chatbot-agents stack approach to AI. It looks autonomous at the surface, but the flexibility is bought at the cost of reliability. When you strain a system based on that stack it breaks. But the illusion of flexible behavior looking autonomous before it breaks is too strong.