3 ms·

I'm too lazy to look it up, but someone on HN linked a drone test done with an AI pilot that was tasked with destroying surface-to-air missile targets (in a sim

I'm too lazy to look it up, but someone on HN linked a drone test done with an AI pilot that was tasked with destroying surface-to-air missile targets (in a simulation). At one point, the human operator instructed it not to hit certain SAMs, and since the goal was to destroy SAMs, the AI took out the base with the human operator. On the next run, they instructed it not to take out the human operator in pursuit of its goal, so instead it targeted the radio towers the human used to instruct it.

huurtehoog 1mo ago
In world war II testing of homing torpedoes had the same fluke. I think the problem with automation is seeking general automation. The word can have two opposed meanings: constrained behavior, as in "if X happens, the system will automatically do Y" and flexible behavior, as in "whatever you say to the chatbot, it will concoct a response that engenders a natural flow of conversation". If you want a system to be autonomous, you need constrained behavior. It takes a lot of time to debug and work out all edge cases. Flexible behavior is bound to run into trouble at some point. I think that is the fundamental issue with the LLM-chatbot-agents stack approach to AI. It looks autonomous at the surface, but the flexibility is bought at the cost of reliability. When you strain a system based on that stack it breaks. But the illusion of flexible behavior looking autonomous before it breaks is too strong.