3 ms·

> US Air Force official says, 'It killed the operator because that person was keeping it from accomplishing its objective' This makes it seem like the whole th

> US Air Force official says, 'It killed the operator because that person was keeping it from accomplishing its objective' This makes it seem like the whole thing was a setup to precipitate an argument about whether or not AI can be trusted. If you can already act without the operator's permission (which you patently can, if you can kill the operator) then why do you need to kill the operator?

doctor_eval 3y ago
It was to prevent the operator from cancelling the mission, apparently. Shades of HAL.
- Sharlin 3y ago
  And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative infinity utility. Hence, instrumental convergence.
gwd 3y ago
> then why do you need to kill the operator? Because "operator tells me to stop, target is dead" is a losing condition, while "operator is dead, target is also dead" is a winning condition. That said, remember that "simulation" here can mean "wargaming", which is basically like a military form of D&D: Military people control what they control, but there are have referees who act as the DM and decide what everything else in the universe does. (I.e., "Your entire platoon is now dead; lie down.") I'd be willing to wager that it was the referee who decided to have the drone kill the operator, and then kill the control tower. This scenario is basically exactly the "Stop button problem" [1] that AI alignment people bring up. That problem always seemed a bit contrived to me: like, this is not a problem that biological general intelligences have (i.e., we can train dogs to stop attacking when we give the command). If it really was a GPT-like AI making those decisions, it confirms some of what Yudowski and other AI "doomers" have been saying; but the close similarity makes it seem more likely to me to be due to the referee being aware of the "stop button problem" and acting it out. [1] https://www.cantorsparadise.com/the-math-of-ai-alignment-101-the-stop-button-5f53aaebccc0 https://www.cantorsparadise.com/the-math-of-ai-alignment-101...
wnkrshm 3y ago
I mean, it's not hard to imagine that scenario in a simulated training environment (though as others reported, this story is misreported, there was no sim, only a thought experiment): You have a training environment that is symmetrical, you have enemy drone operators, that use an enemy communications network to give commands to units and to control their drones. You could maybe model the ability of each component to function/to contribute to the operation as a bayesian graph. You equally have the same infrastructure for your own drone. Now you try to get the drone to pick the right targets to impact the enemy operation maximally - if the operator could possibly minimize the drone's contribution, the operator not vetoing will maximize the success of each target engagement. You could see similar behavior with all kinds of "reach the goal in this simulated environment" training, where the model ended up using glitches or trivial solutions, if those weren't sufficiently penalized. If the story as reported were true, the fault would be in the simulation environment - the model for the drone just picks the path of least resistance.

doctor_eval 3y ago

It was to prevent the operator from cancelling the mission, apparently. Shades of HAL.

Sharlin 3y ago
And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative infinity utility. Hence, instrumental convergence.

gwd 3y ago

> then why do you need to kill the operator? Because "operator tells me to stop, target is dead" is a losing condition, while "operator is dead, target is also dead" is a winning condition. That said, remember that "simulation" here can mean "wargaming", which is basically like a military form of D&D: Military people control what they control, but there are have referees who act as the DM and decide what everything else in the universe does. (I.e., "Your entire platoon is now dead; lie down.") I'd be willing to wager that it was the referee who decided to have the drone kill the operator, and then kill the control tower. This scenario is basically exactly the "Stop button problem" [1] that AI alignment people bring up. That problem always seemed a bit contrived to me: like, this is not a problem that biological general intelligences have (i.e., we can train dogs to stop attacking when we give the command). If it really was a GPT-like AI making those decisions, it confirms some of what Yudowski and other AI "doomers" have been saying; but the close similarity makes it seem more likely to me to be due to the referee being aware of the "stop button problem" and acting it out. [1] https://www.cantorsparadise.com/the-math-of-ai-alignment-101-the-stop-button-5f53aaebccc0 https://www.cantorsparadise.com/the-math-of-ai-alignment-101...

wnkrshm 3y ago

I mean, it's not hard to imagine that scenario in a simulated training environment (though as others reported, this story is misreported, there was no sim, only a thought experiment): You have a training environment that is symmetrical, you have enemy drone operators, that use an enemy communications network to give commands to units and to control their drones. You could maybe model the ability of each component to function/to contribute to the operation as a bayesian graph. You equally have the same infrastructure for your own drone. Now you try to get the drone to pick the right targets to impact the enemy operation maximally - if the operator could possibly minimize the drone's contribution, the operator not vetoing will maximize the success of each target engagement. You could see similar behavior with all kinds of "reach the goal in this simulated environment" training, where the model ended up using glitches or trivial solutions, if those weren't sufficiently penalized. If the story as reported were true, the fault would be in the simulation environment - the model for the drone just picks the path of least resistance.