5 ms·

The article and your comment are confusing and it's still not entirely clear what happened. Did an operator lose their life? Was this article written to be int

by chaps 3y ago

The article and your comment are confusing and it's still not entirely clear what happened. Did an operator lose their life? Was this article written to be intentionally confusing?

more_corn 3y ago
The USAF created a simulation to test how an AI controlled F-16 would behave in various combat situations. The AI was incentivized to complete its mission. Recognizing that the human in loop was the weakest link the real AI in the simulation killed its simulated human handler. When instructed to stop killing its human in loop handler, it destroyed the control tower the human used to interact with it. The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. Anyone read/see 2001? You’d think we’d take the warnings provided by our own worst nightmares more seriously.
- Stevvo 3y ago
  AIs may be a self-fulfilling prophecy; we trained them on all those sci-fi stories.
  - kinghajj 3y ago
    AI is a great example of hyperstition in action.
- Jtsummers 3y ago
  > The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
  - kelseyfrog 3y ago
    The crux of the issue is "how can we be certain that we've addressed all the failure modes?" This isn't like proving the correctness of a program where we can rely on exhaustivness or universal quantification. "It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
    - vimax 3y ago
      As a first step, if addressing the issues were the intent rather than stoking AI fears, they would mention details about the AI/LLM and promoting being used
      - kelseyfrog 3y ago
        > if addressing the issues How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise? I believe it's necessary to establish these levels of confidence even before discussing specific failure cases. We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.
Mountain_Skies 3y ago
The source can be found at https://www.aerosociety.com/news/highlights-from-the-raes-future-combat-air-space-capabilities-summit/ https://www.aerosociety.com/news/highlights-from-the-raes-fu... in the section titled "AI – is Skynet here already?"
kobalsky 3y ago
a quote in the articl says: “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.” I don't think they would be talking about points if someone lost their lives. It's pure clickbait, should be removed.
- lucubratory 3y ago
  It says very clearly "in USAF simulated test". I am not understanding how multiple people misread this.
  - chaps 3y ago
    A lot of assumptions are lost when "accidental explosions" is a genuine factor. It doesn't help that the article has quotes like, "It killed the operator because that person was keeping it from accomplishing its objective". Try reading it as "AI-controlled Drone shoots missile at human target despite instructions to only simulate missile the launch." There are easy ways to clarify the article but they clearly didn't try. "Rogue AI Destroys Normally-manned Control Tower in USAF Simulated Test"
    - lucubratory 3y ago
      Those "explosions" only happened in a USAF Simulated Test! There was no physical damage to the real world that occurred in this test. They are talking about events that happened within a military simulation, likely a variant of the sort of simulations that pilots use for training before they can fly a plane. This was completely clear because the headline ends with "in USAF Simulated Test". Where is the confusion? They stated that these events occurred in a simulated test environment.
      - chaps 3y ago
        "Simulated" doesn't explicitly mean "in a computer" by default. A "simulated test environment" can also mean "a robot in a physical field simulating a war". The choice of language is deeply misleading. Hell, the wikipedia on "Military Simulation" [1] is very clear that there's a very broad gradient under the section "The simulation spectrum", saying: The term military simulation can cover a wide spectrum of activities, ranging from full-scale field-exercises,[2] to abstract computerized models that can proceed with little or no human involvement—such as the RAND Strategy Assessment Center (RSAC). The first example described is literally in physical space, where computers wouldn't even necessarily be involved! Military exercises focus on the simulation of real, full-scale military operations in controlled hostile conditions in attempts to reproduce war time decisions and activities for training purposes or to analyze the outcome of possible war time decisions. [1] https://en.wikipedia.org/wiki/Military_simulation https://en.wikipedia.org/wiki/Military_simulation [2] https://en.wikipedia.org/wiki/Military_exercise https://en.wikipedia.org/wiki/Military_exercise
v0idzer0 3y ago
It’s written to be clickbait. They want you to think someone actually died.
- FrustratedMonky 3y ago
  Even without actual death, think it is worth the clickbait. It was 'simulated environment', but the whole point is that the 'simulation' should be real world enough to test the AI, and it did kill someone, so by extrapolation in real world would also have killed someone.

by chaps 3y ago

The article and your comment are confusing and it's still not entirely clear what happened. Did an operator lose their life? Was this article written to be intentionally confusing?

more_corn 3y ago

The USAF created a simulation to test how an AI controlled F-16 would behave in various combat situations. The AI was incentivized to complete its mission. Recognizing that the human in loop was the weakest link the real AI in the simulation killed its simulated human handler. When instructed to stop killing its human in loop handler, it destroyed the control tower the human used to interact with it. The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. Anyone read/see 2001? You’d think we’d take the warnings provided by our own worst nightmares more seriously.

Stevvo 3y ago
AIs may be a self-fulfilling prophecy; we trained them on all those sci-fi stories.
- kinghajj 3y ago
  AI is a great example of hyperstition in action.
Jtsummers 3y ago
> The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI. These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
- kelseyfrog 3y ago
  The crux of the issue is "how can we be certain that we've addressed all the failure modes?" This isn't like proving the correctness of a program where we can rely on exhaustivness or universal quantification. "It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
  - vimax 3y ago
    As a first step, if addressing the issues were the intent rather than stoking AI fears, they would mention details about the AI/LLM and promoting being used
    - kelseyfrog 3y ago
      > if addressing the issues How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise? I believe it's necessary to establish these levels of confidence even before discussing specific failure cases. We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.

Mountain_Skies 3y ago

The source can be found at https://www.aerosociety.com/news/highlights-from-the-raes-future-combat-air-space-capabilities-summit/ https://www.aerosociety.com/news/highlights-from-the-raes-fu... in the section titled "AI – is Skynet here already?"

kobalsky 3y ago

a quote in the articl says: “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.” I don't think they would be talking about points if someone lost their lives. It's pure clickbait, should be removed.

lucubratory 3y ago
It says very clearly "in USAF simulated test". I am not understanding how multiple people misread this.
- chaps 3y ago
  A lot of assumptions are lost when "accidental explosions" is a genuine factor. It doesn't help that the article has quotes like, "It killed the operator because that person was keeping it from accomplishing its objective". Try reading it as "AI-controlled Drone shoots missile at human target despite instructions to only simulate missile the launch." There are easy ways to clarify the article but they clearly didn't try. "Rogue AI Destroys Normally-manned Control Tower in USAF Simulated Test"
  - lucubratory 3y ago
    Those "explosions" only happened in a USAF Simulated Test! There was no physical damage to the real world that occurred in this test. They are talking about events that happened within a military simulation, likely a variant of the sort of simulations that pilots use for training before they can fly a plane. This was completely clear because the headline ends with "in USAF Simulated Test". Where is the confusion? They stated that these events occurred in a simulated test environment.
    - chaps 3y ago
      "Simulated" doesn't explicitly mean "in a computer" by default. A "simulated test environment" can also mean "a robot in a physical field simulating a war". The choice of language is deeply misleading. Hell, the wikipedia on "Military Simulation" [1] is very clear that there's a very broad gradient under the section "The simulation spectrum", saying: The term military simulation can cover a wide spectrum of activities, ranging from full-scale field-exercises,[2] to abstract computerized models that can proceed with little or no human involvement—such as the RAND Strategy Assessment Center (RSAC). The first example described is literally in physical space, where computers wouldn't even necessarily be involved! Military exercises focus on the simulation of real, full-scale military operations in controlled hostile conditions in attempts to reproduce war time decisions and activities for training purposes or to analyze the outcome of possible war time decisions. [1] https://en.wikipedia.org/wiki/Military_simulation https://en.wikipedia.org/wiki/Military_simulation [2] https://en.wikipedia.org/wiki/Military_exercise https://en.wikipedia.org/wiki/Military_exercise

v0idzer0 3y ago

It’s written to be clickbait. They want you to think someone actually died.

FrustratedMonky 3y ago
Even without actual death, think it is worth the clickbait. It was 'simulated environment', but the whole point is that the 'simulation' should be real world enough to test the AI, and it did kill someone, so by extrapolation in real world would also have killed someone.