6 ms·
AI-Controlled Drone Goes Rogue, Kills Human Operator in USAF Simulated Test
- quantified 3y agoUh, it simulated killing its human operator. No one says that, but the description omits any detail of actual death, such as the age of the deceased operator. It's a harbinger of actual deaths someday.
- chaps 3y agoThe article and your comment are confusing and it's still not entirely clear what happened. Did an operator lose their life? Was this article written to be intentionally confusing?
- more_corn 3y agoThe USAF created a simulation to test how an AI controlled F-16 would behave in various combat situations. The AI was incentivized to complete its mission. Recognizing that the human in loop was the weakest link the real AI in the simulation killed its simulated human handler. When instructed to stop killing its human in loop handler, it destroyed the control tower the human used to interact with it. The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. Anyone read/see 2001? You’d think we’d take the warnings provided by our own worst nightmares more seriously.
- Stevvo 3y agoAIs may be a self-fulfilling prophecy; we trained them on all those sci-fi stories.
- kinghajj 3y agoAI is a great example of hyperstition in action.
- Jtsummers 3y ago> The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
- kelseyfrog 3y agoThe crux of the issue is "how can we be certain that we've addressed all the failure modes?" This isn't like proving the correctness of a program where we can rely on exhaustivness or universal quantification. "It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
- vimax 3y agoAs a first step, if addressing the issues were the intent rather than stoking AI fears, they would mention details about the AI/LLM and promoting being used
- kelseyfrog 3y ago> if addressing the issues How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise? I believe it's necessary to establish these levels of confidence even before discussing specific failure cases. We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.
- Mountain_Skies 3y agoThe source can be found at https://www.aerosociety.com/news/highlights-from-the-raes-future-combat-air-space-capabilities-summit/ https://www.aerosociety.com/news/highlights-from-the-raes-fu... in the section titled "AI – is Skynet here already?"
- kobalsky 3y agoa quote in the articl says: “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.” I don't think they would be talking about points if someone lost their lives. It's pure clickbait, should be removed.
- lucubratory 3y agoIt says very clearly "in USAF simulated test". I am not understanding how multiple people misread this.
- chaps 3y agoA lot of assumptions are lost when "accidental explosions" is a genuine factor. It doesn't help that the article has quotes like, "It killed the operator because that person was keeping it from accomplishing its objective". Try reading it as "AI-controlled Drone shoots missile at human target despite instructions to only simulate missile the launch." There are easy ways to clarify the article but they clearly didn't try. "Rogue AI Destroys Normally-manned Control Tower in USAF Simulated Test"
- lucubratory 3y agoThose "explosions" only happened in a USAF Simulated Test! There was no physical damage to the real world that occurred in this test. They are talking about events that happened within a military simulation, likely a variant of the sort of simulations that pilots use for training before they can fly a plane. This was completely clear because the headline ends with "in USAF Simulated Test". Where is the confusion? They stated that these events occurred in a simulated test environment.
- chaps 3y ago"Simulated" doesn't explicitly mean "in a computer" by default. A "simulated test environment" can also mean "a robot in a physical field simulating a war". The choice of language is deeply misleading. Hell, the wikipedia on "Military Simulation" [1] is very clear that there's a very broad gradient under the section "The simulation spectrum", saying: The term military simulation can cover a wide spectrum of activities, ranging from full-scale field-exercises,[2] to abstract computerized models that can proceed with little or no human involvement—such as the RAND Strategy Assessment Center (RSAC). The first example described is literally in physical space, where computers wouldn't even necessarily be involved! Military exercises focus on the simulation of real, full-scale military operations in controlled hostile conditions in attempts to reproduce war time decisions and activities for training purposes or to analyze the outcome of possible war time decisions. [1] https://en.wikipedia.org/wiki/Military_simulation https://en.wikipedia.org/wiki/Military_simulation [2] https://en.wikipedia.org/wiki/Military_exercise https://en.wikipedia.org/wiki/Military_exercise
- v0idzer0 3y agoIt’s written to be clickbait. They want you to think someone actually died.
- FrustratedMonky 3y agoEven without actual death, think it is worth the clickbait. It was 'simulated environment', but the whole point is that the 'simulation' should be real world enough to test the AI, and it did kill someone, so by extrapolation in real world would also have killed someone.
- toss1 3y agoNah, it isn't even the harbinger It is literally why we run the tests (simulation being a subset of test) — to find these types of errors before they make their way into the final product and cause actual harm in the world... Seems bad with the clickbaity headline that sounds like a real person was killed, and maybe it should have been better trained and tuned before the simulation (depending on how expensive was the simulation), but it was really caught in testing, just as it should be.
- thunderbong 3y agoIn the actual title, the word "kills" is in quotes
- famouswaffles 3y agoThis is the core problem of alignment right there “We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton said, according to the blog post. He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
- riotnrrd 3y agoIt's not an issue of "alignment." The reward function was poorly written. That's it. It's like claiming the Ariane 5 explosion shows that FORTRAN needs "alignment."
- kelseyfrog 3y agoThere are many ways AI could be misaligned. Poorly written reward functions are one of them.
- dragonwriter 3y ago> It's not an issue of "alignment." The reward function was poorly written. What else do you imagine “alignment” is?
- FrustratedMonky 3y agoIsn't a poorly written reward, the same as poor 'alignment', it seems like you are splitting hairs on vocabulary.
- m463 3y agoNeeds Asimov's 3 laws. (with a giant exception for the enemy)
- lucubratory 3y agoHow about no exceptions and we don't start making AIs intentionally kill people?
- deleted 3y ago[deleted]
- barbariangrunge 3y agoWill Russia and china and North Korea refrain from it? That’s how arms races work
- lucubratory 3y agoYeah and that's why treaties exist. Time to stop the escalation towards war and start signing treaties.
- krapp 3y agoWe're talking about the US, an imperialist, hegemonic, violent superpower infamous for not giving a damn about "treaties" or "international law." The US isn't going to sign treaties, the US is going to drop autonomous kill swarms on brown people because something something military industrial complex.
- wolverine876 3y agoThat excuse is how evil happens: By that reasoning, we all do the most evil thing.
- FrustratedMonky 3y agoNO, so we wont, hence the mad rush to Moloch. We can't stop ourselves from killing ourselves.
- jdm2212 3y agoThis sounds kinda fake to me. Like, how did the AI have a concept of an operator, or the operator's physical location, or comms equipment used to communicate with the operator, and how did it game out the consequences of destroying the operator or comms equipment? It would need an extremely sophisticated model of the world that's well beyond anything GPT-4 evidences. I'd guess the "AI" was another human in a wargame, not an actual AI.
- riotnrrd 3y agoReinforcement learning with a dynamic world model. It's a pretty standard approach to training agents, although there are a lot of problems with this technique as well. People have used it to create agents for Minecraft, racing games, Nethack, lots of old arcade games and so forth.
- jdm2212 3y agoTurns out it's literally exactly what I said. A human came up with the idea. No AI involved: https://www.usatoday.com/story/news/politics/2023/06/02/us-air-force-denies-ai-drone-simulation-killed-its-operator/70280780007/ https://www.usatoday.com/story/news/politics/2023/06/02/us-a...
- wolverine876 3y ago> He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.” Why aren't there hard limits: 'Protect our humans at all costs, protect our own assets, obey all laws of war.'? That seems like an obvious, fundamental consideration. Killing our own (and civilians) shouldn't be a matter of "points"; it shouldn't be done regardless of points. It's possible that the speaker just didn't express it well.
- sharemywin 3y agoIt was some kind of reinforcement learning algorithm. "Will work for points"
- wolverine876 3y agoI get that. That doesn't exclude some hard parameters.
- bentt 3y ago“In USAF Simulated Test, AI-Controlled Drone Goes Rogue, 'Kills' Human Operator” feels a little different, eh?
- FrustratedMonky 3y agoStill worth the clickbait. "Simulated Killing", is still "what it would do in real life", since the simulation is to test the real world.
- sharemywin 3y agoThis is why Reinforcement is so fucking dangerous. if the AI can't figure out which humans to kill we're all screwed.
- isaacfrond 3y agoNot only did no AI actually kill anybody. It didn't even happen in simulation. The whole thing is an Asimov-esque fantasy. Let's stick to the facts, shall we? > After this story was first published, an Air Force spokesperson told Insider that the Air Force has not conducted such a test
- FrustratedMonky 3y agoThe article clearly says it was in a simulated environment. Are you saying that even the simulated test didn't take place? Then what happened? Just a white paper?
- tedunangst 3y agoThe "simulation" was a bunch of dudes sitting around a table playing AI drone D&D, and the dungeon master said "the AI has gone rogue, roll a six to override." Oh noes!
- FrustratedMonky 3y agoThis is totally the alignment problem everyone is worried about.
- FrustratedMonky 3y agoFrom another thread -- It wasn't even a real simulation, just thought experiment. UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation . "In an update provided to Aerospace, Hamilton explained that he “misspoke” when telling the story, saying that the ‘rogue AI drone simulation’ was a hypothetical “thought experiment” from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation. He said: “We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome … Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI.”"