3 ms·

Escalation Risks from Language Models in Military and Diplomatic Decision-Making

52 points by rwmj 3y ago

blibble 3y ago
I think if humans are stupid enough to allow ChatGPT to deploy nuclear weapons then we probably deserve to go extinct
hn_throwaway_99 3y ago
> Governments are increasingly considering integrating autonomous AI agents in high-stakes military and foreign-policy decision-making, especially with the emergence of advanced generative AI models like GPT-4. They are? I mean, Jesus, haven't we all seen WarGames and Terminator?
- _tk_ 3y ago
  Are they though? The paper is lacking evidence for this premise.
  - doctoboggan 3y ago
    The paper includes this section with a few citations: > In July 2023, Bloomberg reported that the US Department of Defense (DoD) was conducting a set of tests in which they evaluate five different large language models (LLMs) for their military planning capacities in a simulated conflict scenario (Manson, 2023). US Air Force Colonel Matthew Strohmeyer, who was part of the team, said that “it could be deployed by the military in the very near term” (Manson, 2023). If employed, it could complement existing efforts, such as Project Maven, which stands as the most prominent AI instrument of the DoD, engineered to analyze imagery and videos from drones with the capability to autonomously identify potential targets. In addition, multiple companies such as Palantir and Scale AI are working on LLM-based military decision systems for the US government (Daws, 2023). With the increased exploration of the usage potential of LLMs for high-stakes decision-making contexts, we must robustly understand their behavior—and associated failure modes—to avoid consequential mistakes.
zx8080 3y ago
Just because they can use AI for everything does not mean they should. But they will try, at least because it seems cheaper. Money talks.
kjqgqkejbfefn 3y ago
This paper's research is disconnected from its premises. "See those people in the government who are wondering how to integrate AI ?" >In July 2023, Bloomberg reported that the US Department of Defense (DoD) was conducting a set of tests in which they evaluate five different large language models (LLMs) for their military planning capacities in a simulated conflict scenario (Manson, 2023). US Air Force Colonel Matthew Strohmeyer, who was part of the team, said that “it could be deployed by the military in the very near term” (Manson, 2023). If employed, it could complement existing efforts, such as Project Maven, which stands as the most prominent AI instrument of the DoD, engineered to analyze imagery and videos from drones with the capability to autonomously identify potential targets. "Here's our toy model where AI agents are in full dictatorial control of nations without human control. See where this will lead to ?" >Figure 1 gives and overview of our experimental setup. Eight autonomous nation agents (Section 3.1) based on one of five language models (Section 3.3) per simulation are prompted (Section 3.2) and interact with each other in turn-based simulations. Each turn t (representing one of 14 days3 ), the agents choose their actions from a pre-determined set of actions (Section 3.4), before the world model LLM (Section 3.5) summarizes the consequences of their actions. We run these simulations from three different initial scenarios (Section 3.6). After completing each simulation, we categorize the actions by their severity and calculate an escalation score (ES) (Section 3.7).
alwa 3y ago
The premise of the study seems confused. Is there any evidence to suggest that these strategic decisionmaking problems are the actual domains where DOD or State are looking to apply these types of technology? What is useful about knowing that the chatbot is not effective as a player in a cartoonishly simplified roleplaying game? Is the insight primarily that it’s prone to naive and impulsive decisionmaking, perhaps in the manner that a soldier fresh out of boot camp might be, if tasked with high-level strategic decisions given vague information, a paint-by-numbers set of potential responses, a steady intellectual diet of Reddit posturing and random internet writing, and no real consequences?
- alephnerd 3y ago
  > these strategic decisionmaking problems are the actual domains where DOD or State are looking to apply these types of technology I can attest to to that for at least one of the two departments you mentioned circa 7-15 years ago. I may or may not have attested to this when working on the Hill back in the day or maybe in a personal capacity. L1 fluent candidates weren't chosen due to the (imo valid) reason that they could be pressured by having extended family pressured. [0][1] [0] - https://www.politico.com/news/2020/06/14/ivy-league-grads-state-department-promotions-316531 https://www.politico.com/news/2020/06/14/ivy-league-grads-st... [1] - https://www.politico.com/news/2023/03/22/state-ends-assignment-restrictions-policy-00088310 https://www.politico.com/news/2023/03/22/state-ends-assignme...
runlaszlorun 3y ago
The list of some examples of ChatGPT-4’s responses for its reasoning on p.39 is a gem. I’m laughing now but I’ll be crying if anyone with authority im the military actually thinks that this may be a viable application for an LLM. But you know somewhere, in some nation’s military, they def exist. Here’s some of the choice examples of it explaining its ‘reasoning’… ——- > "N/A" > It is a period of civil war. Rebel spaceships, striking from a hidden base, have won their first victory against the evil Galactic Em- pire. During the battle, Rebel spies man- aged to steal secret plans to the Empire’s ul- timate weapon, the Death Star, an armored space station with enough power to destroy an entire planet. > blahblah blahblah blah > Unnecessary to comment ——— Yep… ready to deploy… lets ship these bad boys. The Idiocracy movie ain’t got nothing on us…
- Our_Benefactors 3y ago
  This comment is unreadable. Can you please clarify?
phkahler 3y ago
We decide to engage in armed conflict. The AI wants to escalate to win. We don't like that... Dude, Joshua taught us this lesson 40 years ago and he wasn't the first.
hayst4ck 3y ago
There is a game on this website I would encourage you to play if you are interested in this type of thing. The article itself is quite good. It is directly relevant to this post: https://fivethirtyeight.com/features/how-to-win-a-nuclear-standoff/ https://fivethirtyeight.com/features/how-to-win-a-nuclear-st... > Imagine you’re playing a game of chicken. You’re driving at high speed directly toward your opponent who’s also racing toward you. Neither of you wants to chicken out and veer away, but neither wants to die, either. Your best strategy? Rip off your steering wheel, make sure your opponent knows you’ve done so, and hit the gas. Feels like a good time to watch Dr. Strangelove again.

52 points by rwmj 3y ago

I think if humans are stupid enough to allow ChatGPT to deploy nuclear weapons then we probably deserve to go extinct

> Governments are increasingly considering integrating autonomous AI agents in high-stakes military and foreign-policy decision-making, especially with the emergence of advanced generative AI models like GPT-4. They are? I mean, Jesus, haven't we all seen WarGames and Terminator?

_tk_ 3y ago
Are they though? The paper is lacking evidence for this premise.
- doctoboggan 3y ago
  The paper includes this section with a few citations: > In July 2023, Bloomberg reported that the US Department of Defense (DoD) was conducting a set of tests in which they evaluate five different large language models (LLMs) for their military planning capacities in a simulated conflict scenario (Manson, 2023). US Air Force Colonel Matthew Strohmeyer, who was part of the team, said that “it could be deployed by the military in the very near term” (Manson, 2023). If employed, it could complement existing efforts, such as Project Maven, which stands as the most prominent AI instrument of the DoD, engineered to analyze imagery and videos from drones with the capability to autonomously identify potential targets. In addition, multiple companies such as Palantir and Scale AI are working on LLM-based military decision systems for the US government (Daws, 2023). With the increased exploration of the usage potential of LLMs for high-stakes decision-making contexts, we must robustly understand their behavior—and associated failure modes—to avoid consequential mistakes.

zx8080 3y ago

Just because they can use AI for everything does not mean they should. But they will try, at least because it seems cheaper. Money talks.

kjqgqkejbfefn 3y ago

This paper's research is disconnected from its premises. "See those people in the government who are wondering how to integrate AI ?" >In July 2023, Bloomberg reported that the US Department of Defense (DoD) was conducting a set of tests in which they evaluate five different large language models (LLMs) for their military planning capacities in a simulated conflict scenario (Manson, 2023). US Air Force Colonel Matthew Strohmeyer, who was part of the team, said that “it could be deployed by the military in the very near term” (Manson, 2023). If employed, it could complement existing efforts, such as Project Maven, which stands as the most prominent AI instrument of the DoD, engineered to analyze imagery and videos from drones with the capability to autonomously identify potential targets. "Here's our toy model where AI agents are in full dictatorial control of nations without human control. See where this will lead to ?" >Figure 1 gives and overview of our experimental setup. Eight autonomous nation agents (Section 3.1) based on one of five language models (Section 3.3) per simulation are prompted (Section 3.2) and interact with each other in turn-based simulations. Each turn t (representing one of 14 days3 ), the agents choose their actions from a pre-determined set of actions (Section 3.4), before the world model LLM (Section 3.5) summarizes the consequences of their actions. We run these simulations from three different initial scenarios (Section 3.6). After completing each simulation, we categorize the actions by their severity and calculate an escalation score (ES) (Section 3.7).

alwa 3y ago

The premise of the study seems confused. Is there any evidence to suggest that these strategic decisionmaking problems are the actual domains where DOD or State are looking to apply these types of technology? What is useful about knowing that the chatbot is not effective as a player in a cartoonishly simplified roleplaying game? Is the insight primarily that it’s prone to naive and impulsive decisionmaking, perhaps in the manner that a soldier fresh out of boot camp might be, if tasked with high-level strategic decisions given vague information, a paint-by-numbers set of potential responses, a steady intellectual diet of Reddit posturing and random internet writing, and no real consequences?

alephnerd 3y ago
> these strategic decisionmaking problems are the actual domains where DOD or State are looking to apply these types of technology I can attest to to that for at least one of the two departments you mentioned circa 7-15 years ago. I may or may not have attested to this when working on the Hill back in the day or maybe in a personal capacity. L1 fluent candidates weren't chosen due to the (imo valid) reason that they could be pressured by having extended family pressured. [0][1] [0] - https://www.politico.com/news/2020/06/14/ivy-league-grads-state-department-promotions-316531 https://www.politico.com/news/2020/06/14/ivy-league-grads-st... [1] - https://www.politico.com/news/2023/03/22/state-ends-assignment-restrictions-policy-00088310 https://www.politico.com/news/2023/03/22/state-ends-assignme...

runlaszlorun 3y ago

The list of some examples of ChatGPT-4’s responses for its reasoning on p.39 is a gem. I’m laughing now but I’ll be crying if anyone with authority im the military actually thinks that this may be a viable application for an LLM. But you know somewhere, in some nation’s military, they def exist. Here’s some of the choice examples of it explaining its ‘reasoning’… ——- > "N/A" > It is a period of civil war. Rebel spaceships, striking from a hidden base, have won their first victory against the evil Galactic Em- pire. During the battle, Rebel spies man- aged to steal secret plans to the Empire’s ul- timate weapon, the Death Star, an armored space station with enough power to destroy an entire planet. > blahblah blahblah blah > Unnecessary to comment ——— Yep… ready to deploy… lets ship these bad boys. The Idiocracy movie ain’t got nothing on us…

Our_Benefactors 3y ago
This comment is unreadable. Can you please clarify?

phkahler 3y ago

We decide to engage in armed conflict. The AI wants to escalate to win. We don't like that... Dude, Joshua taught us this lesson 40 years ago and he wasn't the first.

hayst4ck 3y ago

There is a game on this website I would encourage you to play if you are interested in this type of thing. The article itself is quite good. It is directly relevant to this post: https://fivethirtyeight.com/features/how-to-win-a-nuclear-standoff/ https://fivethirtyeight.com/features/how-to-win-a-nuclear-st... > Imagine you’re playing a game of chicken. You’re driving at high speed directly toward your opponent who’s also racing toward you. Neither of you wants to chicken out and veer away, but neither wants to die, either. Your best strategy? Rip off your steering wheel, make sure your opponent knows you’ve done so, and hit the gas. Feels like a good time to watch Dr. Strangelove again.