6 ms·

Large language models in national security applications

89 points by bindidwodtj 2y ago

robertkoss 2y ago
Some related news: https://investors.palantir.com/news-details/2024/Anthropic-and-Palantir-Partner-to-Bring-Claude-AI-Models-to-AWS-for-U.S.-Government-Intelligence-and-Defense-Operations/ https://investors.palantir.com/news-details/2024/Anthropic-a...
Syonyk 2y ago
Abstract: > The overwhelming success of GPT-4 in early 2023 highlighted the transformative potential of large language models (LLMs) across various sectors, including national security. This article explores the implications of LLM integration within national security contexts, analyzing their potential to revolutionize information processing, decision-making, and operational efficiency. Whereas LLMs offer substantial benefits, such as automating tasks and enhancing data analysis, they also pose significant risks, including hallucinations, data privacy concerns, and vulnerability to adversarial attacks. Through their coupling with decision-theoretic principles and Bayesian reasoning, LLMs can significantly improve decision-making processes within national security organizations. Namely, LLMs can facilitate the transition from data to actionable decisions, enabling decision-makers to quickly receive and distill available information with less manpower. Current applications within the US Department of Defense and beyond are explored, e.g., the USAF's use of LLMs for wargaming and automatic summarization, that illustrate their potential to streamline operations and support decision-making. However, these applications necessitate rigorous safeguards to ensure accuracy and reliability. The broader implications of LLM integration extend to strategic planning, international relations, and the broader geopolitical landscape, with adversarial nations leveraging LLMs for disinformation and cyber operations, emphasizing the need for robust countermeasures. Despite exhibiting "sparks" of artificial general intelligence, LLMs are best suited for supporting roles rather than leading strategic decisions. Their use in training and wargaming can provide valuable insights and personalized learning experiences for military personnel, thereby improving operational readiness. I mean, I'm glad they suggest that LLMs be used in "supporting roles rather than leading strategic decisions," but... no? Let's please not go down this route for international politics and national security. "Twitch Plays CIA" and "Reddit Plays International Geopolitical Negotiations" sound like bad movies, let's not make them our new reality...
- deleted 2y ago
  [deleted]
gmaster1440 2y ago
The paper argues against using LLMs for military strategy, claiming "no textbook contains the right answers" and strategy can't be learned from text alone (the "Virtual Clausewitz" Problem). But this seems to underestimate LLMs' demonstrated ability to reason through novel situations. Rather than just pattern-matching historical examples, modern LLMs can synthesize insights across domains, identify non-obvious patterns, and generate novel strategic approaches. The real question isn't whether perfect answers exist in training data, but whether LLMs can engage in effective strategic reasoning—which increasingly appears to be the case, especially with reasoning models like o1.
- ben_w 2y ago
  LLMs can combine cross-domain insights, but the insights they have — that I've seen them have in the models I've used — are around the level of a second year university student. I would concur with what the abstract says: incredibly valuable (IMO the breadth of easily discoverable knowledge is a huge plus all by itself), but don't put them in charge.
  - gmaster1440 2y ago
    The "second year university student" analogy is interesting, but might not fully capture what's unique about LLMs in strategic analysis. Unlike students, LLMs can simultaneously process and synthesize insights from thousands of historical conflicts, military doctrines, and real-time data points without human cognitive limitations or biases. The paper actually makes a stronger case for using LLMs to enhance rather than replace human strategists - imagine a military commander with instant access to an aide that has deeply analyzed every military campaign in history and can spot relevant patterns. The question isn't about putting LLMs "in charge," but whether we're fully leveraging their unique capabilities for strategic innovation while maintaining human oversight.
    - ben_w 2y ago
      > Unlike students, LLMs can simultaneously process and synthesize insights from thousands of historical conflicts, military doctrines, and real-time data points without human cognitive limitations or biases. Yes, indeed. Unfortunately (/fortunately depending on who you ask) despite this the actual quality of the output is merely "ok" rather than "fantastic". If you need an answer immediately on any topic where "second year university student" is good enough, these are amazing tools. I don't have that skill level in, say, Chinese, where I can't tell 你好 (hello) from 泥壕 (mud hole/trench)* but ChatGPT can at least manage mediocre jokes that Google Translate turns back into English: 问: 什么东西越洗越脏？答: 水！ But! My experience with LLM translation is much the same as with LLM code generation or GenAI images: anyone with actual skill in whatever field you're asking for support with, can easily do better than the AI. It's a fantastic help when you would otherwise have an intern, and that's a lot of things, but it's not the right tool for every job. * I assume this is grammatically gibberish in Chinese, I'm relying on Google Translate here: https://translate.google.com/?sl=zh-TW&tl=en&text=泥%20壕%20%2F%20你%20好 https://translate.google.com/?sl=zh-TW&tl=en&text=泥%20壕%20%2...
perihelions 2y ago
The most obvious way the US national security industry could use LLM's right now is simply to spam foreign adversaries with chatbots. That's their greatest strength right now—a use-case they have amply proven themselves for. This paper comes off as eager to avoid this topic: they (briefly) talk about detecting foreign LLM spam, which is called propaganda, but sidestep the idea of our own side using it. If we were considering talking about that, we wouldn't choose negative-sentiment descriptors like (quoting the paper) "nation-state sponsored propaganda", or "disinformation campaigns"; we'd use our own netural-sentiment jargon, which is "psychological operations" ("psyops") [0]. That we're not widely debating this question right now *probably* means it's far too late to have a chance of stopping it. edit: Or, to rephrase this as a question: Is it ethical to spam another democracy with millions of chatbots pretending to be citizens of that country—if the effect is to manipulate those citizens to not go to war with our own side, saving our own lives? Is that an atrocity or is that legitimate warfare? [0] https://en.wikipedia.org/wiki/Psychological_operations_(United_States) https://en.wikipedia.org/wiki/Psychological_operations_(Unit...
- joe_the_user 2y ago
  Oh, national security professionals aren't going to be talking about psyops, offensive applications and etc, because such things make a given state look bad - they're an offense against democracy and respect for facts and they make the overt media of a given nation look bad. But hey, leave it to HN commentators to root for taking the gloves off. Not to worry post, I'd bet dollars to donuts the actual classified discussions of such things aren't worried about such niceties. But even more, in those activities of the US and other secret states, that have come to light, these state have propagandized not only enemy populations but also their own. After, have to counter enemies trying to nefariously prevent wars as well.
  - deleted 2y ago
    [deleted]
  - deleted 2y ago
    [deleted]
  - 2y ago
Jerrrrrrry 2y ago
If the probability beats human error margin in regards to collateral damage, then sure. That was the sentiment in regards to Level 5 automaton driven vehicles. I see no logical difference, only human sentiment ones.
- joe_the_user 2y ago
  The problem you have is there's no way to estimate probability in situations like warfare or similar chaotic environments.
  - Jerrrrrrry 2y ago
    Sure you do, it's accumulated heuristics, no different than meteorology, or other macro-sims of chaotic systems. The difference is that human lives are intentioned for different fates; so the negative cognitive dissonance is going to persist consciously, then sub-consciously.
    - joe_the_user 2y ago
      it's accumulated heuristics, no different than meteorology Meteorology is based on physics, meteorology doesn't have a hostile agent attempt counter prediction attempts, meteorology doesn't involve a constantly changing technological landscape, meteorology has access to vast amounts data whereas data that's key to military decisions is generally scarce - you know the phrase "fog of war"? I mean, LLMs in fact, don't provide probabilities for their predictions and indeed the advance of deep learning has hinge "just predict, ignore all considerations of 'good statistics' (knowing probabilities, estimating bias)".
      - Jerrrrrrry 2y ago
        >Meteorology is based on physics, Good start. >meteorology doesn't have a hostile agent attempt counter prediction attempts All chaotic systems have second-order reinforcing feedback loops by "tautological" definition of what a feedback loop is and what a sufficiently-complex system is. >meteorology doesn't involve a constantly changing technological landscape, it does, it just has little incentive/motive/order to change, because knowing the probability of it going to rain vs. the cost of it changing the weather is so uneconomically comically juxtapositioned that technology wasn't advanced enough make any progress. All phones being 5G'd into the weather apparatus would give certainty, but the pressure to do that collectively is not there. >meteorology has access to vast amounts data whereas data that's key to military decisions is generally scarce - you know the phrase "fog of war"? "Fog of war" is how it is perceived from outside. Inside, it is those who have a monopoly on viol/power/surv that ultimately wins. and that is us.
htrp 2y ago
This is how you get CIAGPT
- kevmo 2y ago
  PRISMGPT
  - Jerrrrrrry 2y ago
    https://en.wikipedia.org/wiki/Singleton_(global_governance) https://en.wikipedia.org/wiki/Singleton_(global_governance)
  - hammock 2y ago
    Palantir
photochemsyn 2y ago
It's rather conspicuous that the most well-known use of AI systems in warfare at present, the Lavender / Gospel / Where's Daddy systems used by the IDF, don't get any mention. It's true that LLMs are not the central component of these systems, which have much more in common with Google's targeted ad serving algorithms, in the broader category of machine learning, but a no-code LLM interface is a likely component. In defensive red team scenarios, such an LLM system could be used in all kinds of nefarious ways, using prompts like "provide a list of everyone associated with the US nuclear weapons program, including their immediate friend and family circles, and ranking them by vulnerability to blackmail based on their personal web browsing history" and so on.
- immibis 2y ago
  We're not allowed to talk about Israel.
ofslidingfeet 2y ago
Alternative title: "Obviously Irresponsible, Intellectually Lazy Things that We Definitely Haven't Been Doing for Fifteen Years."

89 points by bindidwodtj 2y ago

Some related news: https://investors.palantir.com/news-details/2024/Anthropic-and-Palantir-Partner-to-Bring-Claude-AI-Models-to-AWS-for-U.S.-Government-Intelligence-and-Defense-Operations/ https://investors.palantir.com/news-details/2024/Anthropic-a...

Syonyk 2y ago

Abstract: > The overwhelming success of GPT-4 in early 2023 highlighted the transformative potential of large language models (LLMs) across various sectors, including national security. This article explores the implications of LLM integration within national security contexts, analyzing their potential to revolutionize information processing, decision-making, and operational efficiency. Whereas LLMs offer substantial benefits, such as automating tasks and enhancing data analysis, they also pose significant risks, including hallucinations, data privacy concerns, and vulnerability to adversarial attacks. Through their coupling with decision-theoretic principles and Bayesian reasoning, LLMs can significantly improve decision-making processes within national security organizations. Namely, LLMs can facilitate the transition from data to actionable decisions, enabling decision-makers to quickly receive and distill available information with less manpower. Current applications within the US Department of Defense and beyond are explored, e.g., the USAF's use of LLMs for wargaming and automatic summarization, that illustrate their potential to streamline operations and support decision-making. However, these applications necessitate rigorous safeguards to ensure accuracy and reliability. The broader implications of LLM integration extend to strategic planning, international relations, and the broader geopolitical landscape, with adversarial nations leveraging LLMs for disinformation and cyber operations, emphasizing the need for robust countermeasures. Despite exhibiting "sparks" of artificial general intelligence, LLMs are best suited for supporting roles rather than leading strategic decisions. Their use in training and wargaming can provide valuable insights and personalized learning experiences for military personnel, thereby improving operational readiness. I mean, I'm glad they suggest that LLMs be used in "supporting roles rather than leading strategic decisions," but... no? Let's please not go down this route for international politics and national security. "Twitch Plays CIA" and "Reddit Plays International Geopolitical Negotiations" sound like bad movies, let's not make them our new reality...

deleted 2y ago
[deleted]

gmaster1440 2y ago

The paper argues against using LLMs for military strategy, claiming "no textbook contains the right answers" and strategy can't be learned from text alone (the "Virtual Clausewitz" Problem). But this seems to underestimate LLMs' demonstrated ability to reason through novel situations. Rather than just pattern-matching historical examples, modern LLMs can synthesize insights across domains, identify non-obvious patterns, and generate novel strategic approaches. The real question isn't whether perfect answers exist in training data, but whether LLMs can engage in effective strategic reasoning—which increasingly appears to be the case, especially with reasoning models like o1.

ben_w 2y ago
LLMs can combine cross-domain insights, but the insights they have — that I've seen them have in the models I've used — are around the level of a second year university student. I would concur with what the abstract says: incredibly valuable (IMO the breadth of easily discoverable knowledge is a huge plus all by itself), but don't put them in charge.
- gmaster1440 2y ago
  The "second year university student" analogy is interesting, but might not fully capture what's unique about LLMs in strategic analysis. Unlike students, LLMs can simultaneously process and synthesize insights from thousands of historical conflicts, military doctrines, and real-time data points without human cognitive limitations or biases. The paper actually makes a stronger case for using LLMs to enhance rather than replace human strategists - imagine a military commander with instant access to an aide that has deeply analyzed every military campaign in history and can spot relevant patterns. The question isn't about putting LLMs "in charge," but whether we're fully leveraging their unique capabilities for strategic innovation while maintaining human oversight.
  - ben_w 2y ago
    > Unlike students, LLMs can simultaneously process and synthesize insights from thousands of historical conflicts, military doctrines, and real-time data points without human cognitive limitations or biases. Yes, indeed. Unfortunately (/fortunately depending on who you ask) despite this the actual quality of the output is merely "ok" rather than "fantastic". If you need an answer immediately on any topic where "second year university student" is good enough, these are amazing tools. I don't have that skill level in, say, Chinese, where I can't tell 你好 (hello) from 泥壕 (mud hole/trench)* but ChatGPT can at least manage mediocre jokes that Google Translate turns back into English: 问: 什么东西越洗越脏？答: 水！ But! My experience with LLM translation is much the same as with LLM code generation or GenAI images: anyone with actual skill in whatever field you're asking for support with, can easily do better than the AI. It's a fantastic help when you would otherwise have an intern, and that's a lot of things, but it's not the right tool for every job. * I assume this is grammatically gibberish in Chinese, I'm relying on Google Translate here: https://translate.google.com/?sl=zh-TW&tl=en&text=泥%20壕%20%2F%20你%20好 https://translate.google.com/?sl=zh-TW&tl=en&text=泥%20壕%20%2...

perihelions 2y ago

The most obvious way the US national security industry could use LLM's right now is simply to spam foreign adversaries with chatbots. That's their greatest strength right now—a use-case they have amply proven themselves for. This paper comes off as eager to avoid this topic: they (briefly) talk about detecting foreign LLM spam, which is called propaganda, but sidestep the idea of our own side using it. If we were considering talking about that, we wouldn't choose negative-sentiment descriptors like (quoting the paper) "nation-state sponsored propaganda", or "disinformation campaigns"; we'd use our own netural-sentiment jargon, which is "psychological operations" ("psyops") [0]. That we're not widely debating this question right now *probably* means it's far too late to have a chance of stopping it. edit: Or, to rephrase this as a question: Is it ethical to spam another democracy with millions of chatbots pretending to be citizens of that country—if the effect is to manipulate those citizens to not go to war with our own side, saving our own lives? Is that an atrocity or is that legitimate warfare? [0] https://en.wikipedia.org/wiki/Psychological_operations_(United_States) https://en.wikipedia.org/wiki/Psychological_operations_(Unit...

joe_the_user 2y ago
Oh, national security professionals aren't going to be talking about psyops, offensive applications and etc, because such things make a given state look bad - they're an offense against democracy and respect for facts and they make the overt media of a given nation look bad. But hey, leave it to HN commentators to root for taking the gloves off. Not to worry post, I'd bet dollars to donuts the actual classified discussions of such things aren't worried about such niceties. But even more, in those activities of the US and other secret states, that have come to light, these state have propagandized not only enemy populations but also their own. After, have to counter enemies trying to nefariously prevent wars as well.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- 2y ago

Jerrrrrrry 2y ago

If the probability beats human error margin in regards to collateral damage, then sure. That was the sentiment in regards to Level 5 automaton driven vehicles. I see no logical difference, only human sentiment ones.

joe_the_user 2y ago
The problem you have is there's no way to estimate probability in situations like warfare or similar chaotic environments.
- Jerrrrrrry 2y ago
  Sure you do, it's accumulated heuristics, no different than meteorology, or other macro-sims of chaotic systems. The difference is that human lives are intentioned for different fates; so the negative cognitive dissonance is going to persist consciously, then sub-consciously.
  - joe_the_user 2y ago
    it's accumulated heuristics, no different than meteorology Meteorology is based on physics, meteorology doesn't have a hostile agent attempt counter prediction attempts, meteorology doesn't involve a constantly changing technological landscape, meteorology has access to vast amounts data whereas data that's key to military decisions is generally scarce - you know the phrase "fog of war"? I mean, LLMs in fact, don't provide probabilities for their predictions and indeed the advance of deep learning has hinge "just predict, ignore all considerations of 'good statistics' (knowing probabilities, estimating bias)".
    - Jerrrrrrry 2y ago
      >Meteorology is based on physics, Good start. >meteorology doesn't have a hostile agent attempt counter prediction attempts All chaotic systems have second-order reinforcing feedback loops by "tautological" definition of what a feedback loop is and what a sufficiently-complex system is. >meteorology doesn't involve a constantly changing technological landscape, it does, it just has little incentive/motive/order to change, because knowing the probability of it going to rain vs. the cost of it changing the weather is so uneconomically comically juxtapositioned that technology wasn't advanced enough make any progress. All phones being 5G'd into the weather apparatus would give certainty, but the pressure to do that collectively is not there. >meteorology has access to vast amounts data whereas data that's key to military decisions is generally scarce - you know the phrase "fog of war"? "Fog of war" is how it is perceived from outside. Inside, it is those who have a monopoly on viol/power/surv that ultimately wins. and that is us.

htrp 2y ago

This is how you get CIAGPT

kevmo 2y ago
PRISMGPT
- Jerrrrrrry 2y ago
  https://en.wikipedia.org/wiki/Singleton_(global_governance) https://en.wikipedia.org/wiki/Singleton_(global_governance)
- hammock 2y ago
  Palantir

photochemsyn 2y ago

It's rather conspicuous that the most well-known use of AI systems in warfare at present, the Lavender / Gospel / Where's Daddy systems used by the IDF, don't get any mention. It's true that LLMs are not the central component of these systems, which have much more in common with Google's targeted ad serving algorithms, in the broader category of machine learning, but a no-code LLM interface is a likely component. In defensive red team scenarios, such an LLM system could be used in all kinds of nefarious ways, using prompts like "provide a list of everyone associated with the US nuclear weapons program, including their immediate friend and family circles, and ranking them by vulnerability to blackmail based on their personal web browsing history" and so on.

immibis 2y ago
We're not allowed to talk about Israel.

ofslidingfeet 2y ago

Alternative title: "Obviously Irresponsible, Intellectually Lazy Things that We Definitely Haven't Been Doing for Fifteen Years."