5 ms·

And what happens when the defense AI 'hallucinates' and suggests that somebody is a terrorist when they are not?

rodgerd 2y ago
Security podcasters will cheer you killing kids because you might have hit a few terrorists in the process.
JumpCrisscross 2y ago
> what happens when the defense AI 'hallucinates' and suggests that somebody is a terrorist when they are not? Collateral damage. Same thing that happens in any war when an analyst or soldier misreads the battlespace. War is hell. We won’t change that by making it pleasant. We can only avoid it by not going to war.
- protomolecule 2y ago
  [flagged]
  - JumpCrisscross 2y ago
    > Acts like firebombing of Tokyo or bombing of Dresden or atomic bombing don't happen now We still raze cities and drop incendiaries. America hasn’t gone to war with a near-peer nonnuclear power like Japan since WWII. To the extent we were faced with the prospect in the Cold War, both we and the Soviets were committed to MAD, i.e. using nukes. (Do you think unilateral disarmament in the Cold War would have lead to peace?) There has been no militarily useful technology that was voluntarily abandoned. Just constrained. You can’t constrain a technology you don’t bother understanding.
    - protomolecule 2y ago
      [flagged]
      - JumpCrisscross 2y ago
        > during the firebombing of Tokyo the US murdered 100,000 civilians Are you arguing there was a war in which firebombing would have been useful but someone decided it was too mean? Since WWII we invented better high explosives and stand-off precision weapons. If there were a strategic case for firebombing in a future war, have no delusions: it will happen. (Last year, incendiary weapons were used in “ in the Gaza Strip, Lebanon, Ukraine, and Syria” [1].) [1] https://www.hrw.org/news/2024/11/07/incendiary-weapons-new-use-calls-immediate-action https://www.hrw.org/news/2024/11/07/incendiary-weapons-new-u...
        protomolecule 2y ago
        [flagged]
        JumpCrisscross 2y ago
        > literally 'making war different' What? Who argued war hasn’t changed with technology? > civilians weren't murdered on the same scale War wasn’t conducted on the same scale. > why you are conflating the use of certain types of weapons and willingly allowing enormous collateral damage I’m not. Nobody in this thread is. The point is the weapons are still stockpiled and used. We have never agreed to ban a useful military technology. Just contained or surpassed it. AI will be used by militaries as long as it’s useful, even if it causes collateral damage. We will obviously try to reduce collateral damage. But in part because that makes the weapon more useful.
        protomolecule 2y ago
        [flagged]
        JumpCrisscross 2y ago
        > You argued that war is the same hell as it was 80 years ago and it can't be changed by making war different Where? I certainly did not. War is hell and always has been, but it's obviously a different hell than it was in earlier eras. Going back on piste: if AI has military applications, it will be developed and use for them. > If you say that all hells are made equal, I won't agree Not how a discussion works. > question is why. Maybe it's because something changed Yes. Nukes and precision stand-off weapons.
  - blooalien 2y ago
    > "Acts like firebombing of Tokyo or bombing of Dresden or atomic bombing don't happen now." For the time being... Humanity's leaders are increasingly as insane (sometimes more-so) as their worshipers. I feel it's only a matter of time before atrocities and crimes against humanity skyrocket again. :(
    - JumpCrisscross 2y ago
      > For the time being Not even. We haven’t avoided nuclear war by not building nukes. And we still raze cities and manufacture incendiary weapons.
  - edanm 2y ago
    That's very misleading. Terrible war keeps on happening all the time, just not so much in the US and Europe for the last 70 years. Yes, since WWII things have been relatively peaceful, but the key term there is relatively. As we speak a pretty awful war is happening in Gaza, in the last twenty years we've had multiple wars with pretty severe casualties, and if you go a little farther back you get to things like Vietnam. It's true that specifically atomic bombs haven't been used.
jsheard 2y ago
Going by recent events I think the convention is to drone strike them and their entire family anyway, and then tally them up as a confirmed dead terrorist. https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/
- sourcepluck 2y ago
  Ahh, I hadn't seen this before posting. Thank you for providing a dose of reality, excellent link
woadwarrior01 2y ago
That's been done over a decade ago with random forests. There's no need to apply anything as advanced as generative AI 'hallucinations' for such a trivial problem. /s https://www.theguardian.com/science/the-lay-scientist/2016/feb/18/has-a-rampaging-ai-algorithm-really-killed-thousands-in-pakistan https://www.theguardian.com/science/the-lay-scientist/2016/f...
sourcepluck 2y ago
The cynic in me finds this quite naive - there are territories of the Earth where if you are a male of a certain age and you are killed in a drone strike you are automatically classified as a "military-aged male", i.e. a non-civilian, regardless of the existence of any other evidence. So what will happen if AI suggests someone is a terrorist when they are not? Well, in the worst scenario they'll be killed, and it'll be very close to what we have today, except somewhere in a private military database there might be an automatically generated record from an LLM ok-ing the target.

Security podcasters will cheer you killing kids because you might have hit a few terrorists in the process.

> what happens when the defense AI 'hallucinates' and suggests that somebody is a terrorist when they are not? Collateral damage. Same thing that happens in any war when an analyst or soldier misreads the battlespace. War is hell. We won’t change that by making it pleasant. We can only avoid it by not going to war.

protomolecule 2y ago
[flagged]
- JumpCrisscross 2y ago
  > Acts like firebombing of Tokyo or bombing of Dresden or atomic bombing don't happen now We still raze cities and drop incendiaries. America hasn’t gone to war with a near-peer nonnuclear power like Japan since WWII. To the extent we were faced with the prospect in the Cold War, both we and the Soviets were committed to MAD, i.e. using nukes. (Do you think unilateral disarmament in the Cold War would have lead to peace?) There has been no militarily useful technology that was voluntarily abandoned. Just constrained. You can’t constrain a technology you don’t bother understanding.
  - protomolecule 2y ago
    [flagged]
    - JumpCrisscross 2y ago
      > during the firebombing of Tokyo the US murdered 100,000 civilians Are you arguing there was a war in which firebombing would have been useful but someone decided it was too mean? Since WWII we invented better high explosives and stand-off precision weapons. If there were a strategic case for firebombing in a future war, have no delusions: it will happen. (Last year, incendiary weapons were used in “ in the Gaza Strip, Lebanon, Ukraine, and Syria” [1].) [1] https://www.hrw.org/news/2024/11/07/incendiary-weapons-new-use-calls-immediate-action https://www.hrw.org/news/2024/11/07/incendiary-weapons-new-u...
      - protomolecule 2y ago
        [flagged]
        JumpCrisscross 2y ago
        > literally 'making war different' What? Who argued war hasn’t changed with technology? > civilians weren't murdered on the same scale War wasn’t conducted on the same scale. > why you are conflating the use of certain types of weapons and willingly allowing enormous collateral damage I’m not. Nobody in this thread is. The point is the weapons are still stockpiled and used. We have never agreed to ban a useful military technology. Just contained or surpassed it. AI will be used by militaries as long as it’s useful, even if it causes collateral damage. We will obviously try to reduce collateral damage. But in part because that makes the weapon more useful.
        protomolecule 2y ago
        [flagged]
        JumpCrisscross 2y ago
        > You argued that war is the same hell as it was 80 years ago and it can't be changed by making war different Where? I certainly did not. War is hell and always has been, but it's obviously a different hell than it was in earlier eras. Going back on piste: if AI has military applications, it will be developed and use for them. > If you say that all hells are made equal, I won't agree Not how a discussion works. > question is why. Maybe it's because something changed Yes. Nukes and precision stand-off weapons.
- blooalien 2y ago
  > "Acts like firebombing of Tokyo or bombing of Dresden or atomic bombing don't happen now." For the time being... Humanity's leaders are increasingly as insane (sometimes more-so) as their worshipers. I feel it's only a matter of time before atrocities and crimes against humanity skyrocket again. :(
  - JumpCrisscross 2y ago
    > For the time being Not even. We haven’t avoided nuclear war by not building nukes. And we still raze cities and manufacture incendiary weapons.
- edanm 2y ago
  That's very misleading. Terrible war keeps on happening all the time, just not so much in the US and Europe for the last 70 years. Yes, since WWII things have been relatively peaceful, but the key term there is relatively. As we speak a pretty awful war is happening in Gaza, in the last twenty years we've had multiple wars with pretty severe casualties, and if you go a little farther back you get to things like Vietnam. It's true that specifically atomic bombs haven't been used.

jsheard 2y ago

Going by recent events I think the convention is to drone strike them and their entire family anyway, and then tally them up as a confirmed dead terrorist. https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/

sourcepluck 2y ago
Ahh, I hadn't seen this before posting. Thank you for providing a dose of reality, excellent link

woadwarrior01 2y ago

That's been done over a decade ago with random forests. There's no need to apply anything as advanced as generative AI 'hallucinations' for such a trivial problem. /s https://www.theguardian.com/science/the-lay-scientist/2016/feb/18/has-a-rampaging-ai-algorithm-really-killed-thousands-in-pakistan https://www.theguardian.com/science/the-lay-scientist/2016/f...

sourcepluck 2y ago

The cynic in me finds this quite naive - there are territories of the Earth where if you are a male of a certain age and you are killed in a drone strike you are automatically classified as a "military-aged male", i.e. a non-civilian, regardless of the existence of any other evidence. So what will happen if AI suggests someone is a terrorist when they are not? Well, in the worst scenario they'll be killed, and it'll be very close to what we have today, except somewhere in a private military database there might be an automatically generated record from an LLM ok-ing the target.