6 ms·

I don't see how a ban could work. We already have technology that can identify targets, so the issue is whether a human is involved to approve each kill decisio

I don't see how a ban could work. We already have technology that can identify targets, so the issue is whether a human is involved to approve each kill decision. But what constitutes human approval? Is it enough for a human to sit and watch target details flashing up on a screen, and intervene if they see an incorrect target? What exactly would be banned?

pjc50 11y ago
This is something of a problem. The system which scans mobile phone metadata (among other things) and turns it into drone bombing targets is already enough like the latter. A human is involved in pulling the trigger, but really they're following orders rather than having made the target/not target decision. And they may not even have the security clearance for the information on which the decision is based.
- bitJericho 11y ago
  I don't think that's so much a problem. The problem is literally robots pulling the trigger. What happens when the president is targeted due to error/hacking? A member of the military would have a major problem with assassination of her own leader; a robot may not care. What happens if everybody becomes a target? The military personnel would figure things out real quick, a robot, again, may not care.
  - TheOtherHobbes 11y ago
    A member of the military might not have any problem at all with the "accidental" assassination of their leader, whether that leader is the president or the company sergeant. AI could easily provide plausible deniability. I've been very critical of claims of the dangers of AI, but this isn't about AI - this is about giving lethal weapons to autonomous systems. And that's just stupid. All software has bugs, and even if you don't have a problem with autonomous killing systems - I certainly do - it's incredibly irresponsible to build systems that could kill just because someone left out a semicolon or couldn't convert between miles and km. It's taking the "blue screen of death" experience just a little too literally.
    - bryondowd 11y ago
      Umm, we already have machines that can kill people if someone "left out a semicolon". Airplane autopilots, medical equipment, car computers, etc. Not saying this makes it OK, but that isn't the reason. Life-critical code isn't a new thing.
      - jaawn 11y ago
        And those systems all have humans operating in conjunction with them, none of those things are (currently) autonomous. The success thus far of life-critical code usually depends partially on the assistance of humans.
        bryondowd 11y ago
        Most of those systems operate on their own unless/until they hit something they can't handle and a human is standing by as failover. So, unless you're saying it's OK to have deathbots as long as a human is on hand as failover if the deathbot gets confused, I think my point stands.
        TheOtherHobbes 11y ago
        IMO, the current state of the F-35 rather suggests it doesn't.
  - pjc50 11y ago
    A member of the military would have a major problem with assassination of her own leader https://en.wikipedia.org/wiki/List_of_coups_d%27%C3%A9tat_and_coup_attempts https://en.wikipedia.org/wiki/List_of_coups_d%27%C3%A9tat_an... Usually if a leader is assassinated, the assassin is a member of the country's military. The US is comparatively rare in having had no coups since independence and only one civil war. The US also has very strong indoctrination against this kind of thing, but individuals are just as vulnerable as robots to making the wrong decision if they are fed false information. Such as endless attacks on the legitimacy of the president's election, birth certificate, allegations of being a foreign agent, etc. The US is of course not above assassinating foreign leaders, conspiring to have them deposed, or providing arms to military coups against democratically elected governments. Allende is probably the biggest example. Edit: and of course there's always the risk of a single rogue individual, from Dr. Strangelove's " Brigadier General Jack D. Ripper" to the very real Ford Hood massacre. In fact, Dr. Strangelove should be required viewing on this subject; not only is it darkly hilarious, it's all about setting in motion destructive partly-automated systems which cannot be countermanded.
  - RobertoG 11y ago
    It's funny, because probably is more relevant to think the problem the other way. What happens when leaders know that, no matter what, never are going to be "targeted" by their own automatic military?
- jmngomes 11y ago
  You've had drone operators coming forward and exposing problems with the "program". That's a very good reason to keep humans close to these tasks.
- TeMPOraL 11y ago
  Reminds me of The Machine in Person of Interest. (Spoiler alert for those who haven't seen the first 3 seasons.) The Samaritan works like you described - turns data and metadata into kill decisions. The Machine just outputs a social security number of a "person of interest". Figuring out why is the job of humans. The show features an interesting discussion about what happens when an AI following the first method (data -> kill decision) starts lying to you to further its own agenda. The same issue applies when you have a black box of sub-AI complicated analysis algorithms that make the decision.
  - tedunangst 11y ago
    Why in the world would you design such a system that deliberately fails to explain its decisions?
    - pjc50 11y ago
      Classification. If the system is using all available classified information, then an explanation of its output would have to include the security classification of every input, and the system of classification may also be such that no one person is cleared to know everything. Therefore nobody can know the rationale. (I've made this up, but I think it's the most plausible way such a thing would get built, and it's the logical outcome of "no read up/no write down" security rules.)
      - TeMPOraL 11y ago
        I don't know much about the exact structure of classification. The situation you described would be certainly possible if the classification system is not a tree but a general graph. Is that the case with clearance levels?
        Retric 11y ago
        Yes, it's designed that way for counter espionage reasons. In theory only the president gets full clearance for everything.
notahacker 11y ago
The advantage of humans sitting and watching target details flash up on screen is that not being in any immediate danger themselves they should have fewer excuses for misindentification. Technology should make humans more acccountable for kill decisions rather than less.
- coldtea 11y ago
  In practice it would just make it easier to kill -- so more killings. In wars, invasions, etc, including those done by "democratic powers" few care about "misindentification" (or even far worse offenses to the enemies including civilians). That's why the mostly go unpunished or just get a slap on the wrist. "Kill the all and let god sort them out" is the modus operandi.
rquantz 11y ago
We have bans on chemical and biological weapons. You don't say "but everything is made of chemicals. What are we even banning?" The law will create a legal definition and courts will interpret it. It's not the hard.

pjc50 11y ago

This is something of a problem. The system which scans mobile phone metadata (among other things) and turns it into drone bombing targets is already enough like the latter. A human is involved in pulling the trigger, but really they're following orders rather than having made the target/not target decision. And they may not even have the security clearance for the information on which the decision is based.

bitJericho 11y ago
I don't think that's so much a problem. The problem is literally robots pulling the trigger. What happens when the president is targeted due to error/hacking? A member of the military would have a major problem with assassination of her own leader; a robot may not care. What happens if everybody becomes a target? The military personnel would figure things out real quick, a robot, again, may not care.
- TheOtherHobbes 11y ago
  A member of the military might not have any problem at all with the "accidental" assassination of their leader, whether that leader is the president or the company sergeant. AI could easily provide plausible deniability. I've been very critical of claims of the dangers of AI, but this isn't about AI - this is about giving lethal weapons to autonomous systems. And that's just stupid. All software has bugs, and even if you don't have a problem with autonomous killing systems - I certainly do - it's incredibly irresponsible to build systems that could kill just because someone left out a semicolon or couldn't convert between miles and km. It's taking the "blue screen of death" experience just a little too literally.
  - bryondowd 11y ago
    Umm, we already have machines that can kill people if someone "left out a semicolon". Airplane autopilots, medical equipment, car computers, etc. Not saying this makes it OK, but that isn't the reason. Life-critical code isn't a new thing.
    - jaawn 11y ago
      And those systems all have humans operating in conjunction with them, none of those things are (currently) autonomous. The success thus far of life-critical code usually depends partially on the assistance of humans.
      - bryondowd 11y ago
        Most of those systems operate on their own unless/until they hit something they can't handle and a human is standing by as failover. So, unless you're saying it's OK to have deathbots as long as a human is on hand as failover if the deathbot gets confused, I think my point stands.
        TheOtherHobbes 11y ago
        IMO, the current state of the F-35 rather suggests it doesn't.
- pjc50 11y ago
  A member of the military would have a major problem with assassination of her own leader https://en.wikipedia.org/wiki/List_of_coups_d%27%C3%A9tat_and_coup_attempts https://en.wikipedia.org/wiki/List_of_coups_d%27%C3%A9tat_an... Usually if a leader is assassinated, the assassin is a member of the country's military. The US is comparatively rare in having had no coups since independence and only one civil war. The US also has very strong indoctrination against this kind of thing, but individuals are just as vulnerable as robots to making the wrong decision if they are fed false information. Such as endless attacks on the legitimacy of the president's election, birth certificate, allegations of being a foreign agent, etc. The US is of course not above assassinating foreign leaders, conspiring to have them deposed, or providing arms to military coups against democratically elected governments. Allende is probably the biggest example. Edit: and of course there's always the risk of a single rogue individual, from Dr. Strangelove's " Brigadier General Jack D. Ripper" to the very real Ford Hood massacre. In fact, Dr. Strangelove should be required viewing on this subject; not only is it darkly hilarious, it's all about setting in motion destructive partly-automated systems which cannot be countermanded.
- RobertoG 11y ago
  It's funny, because probably is more relevant to think the problem the other way. What happens when leaders know that, no matter what, never are going to be "targeted" by their own automatic military?
jmngomes 11y ago
You've had drone operators coming forward and exposing problems with the "program". That's a very good reason to keep humans close to these tasks.
TeMPOraL 11y ago
Reminds me of The Machine in Person of Interest. (Spoiler alert for those who haven't seen the first 3 seasons.) The Samaritan works like you described - turns data and metadata into kill decisions. The Machine just outputs a social security number of a "person of interest". Figuring out why is the job of humans. The show features an interesting discussion about what happens when an AI following the first method (data -> kill decision) starts lying to you to further its own agenda. The same issue applies when you have a black box of sub-AI complicated analysis algorithms that make the decision.
- tedunangst 11y ago
  Why in the world would you design such a system that deliberately fails to explain its decisions?
  - pjc50 11y ago
    Classification. If the system is using all available classified information, then an explanation of its output would have to include the security classification of every input, and the system of classification may also be such that no one person is cleared to know everything. Therefore nobody can know the rationale. (I've made this up, but I think it's the most plausible way such a thing would get built, and it's the logical outcome of "no read up/no write down" security rules.)
    - TeMPOraL 11y ago
      I don't know much about the exact structure of classification. The situation you described would be certainly possible if the classification system is not a tree but a general graph. Is that the case with clearance levels?
      - Retric 11y ago
        Yes, it's designed that way for counter espionage reasons. In theory only the president gets full clearance for everything.

notahacker 11y ago

The advantage of humans sitting and watching target details flash up on screen is that not being in any immediate danger themselves they should have fewer excuses for misindentification. Technology should make humans more acccountable for kill decisions rather than less.

coldtea 11y ago
In practice it would just make it easier to kill -- so more killings. In wars, invasions, etc, including those done by "democratic powers" few care about "misindentification" (or even far worse offenses to the enemies including civilians). That's why the mostly go unpunished or just get a slap on the wrist. "Kill the all and let god sort them out" is the modus operandi.

rquantz 11y ago

We have bans on chemical and biological weapons. You don't say "but everything is made of chemicals. What are we even banning?" The law will create a legal definition and courts will interpret it. It's not the hard.