9 ms·

I think the authors are 100% right on everything, but they failed to bring up one of the most important factors that I believe is driving US policy in this area

I think the authors are 100% right on everything, but they failed to bring up one of the most important factors that I believe is driving US policy in this area: if Russian or Chinese weapons are automatically firing at the speed of a neural network while our weapons are waiting for meatware actuation then it's going to be a short battle.

binarymax 8y ago
Similar to chemical warfare. There needs to be a UN resolution to ban these types of weapons. Make it so use of them constitutes war crimes.
- philwelch 8y ago
  Chemical weapons aren't used because they're not worth the risk. There was virtually zero battlefield use of chemical weapons in the Second World War, which was otherwise as "no holds barred" as you can imagine.
  - gotocake 8y ago
    It’s not really risk that’s an issue, it’s a combination of expensive production, dangerous and expensive storage, and expensive deployment contrasted with a total lack of efficacy in killing the intended target. Unless your goal is to murder untrained and unequipped civilians, chemical weapons are just not very good weapons. Soldiers are trained, have vehicles with overpressure seals, masks and MOPP suits, detectors and a command structure. The real use for chemical weapons is to terrify civilians and poorly organized/equipped militias. Most of all it’s a way to murder a large number of civilians without destroying infrastructure. Needless to say that’s not something you’d typically do in a war, where destruction of infrastructure is typically step one! As a result most countries recognize the economic and social burden of chemical weapons and their use, and strongly oppose them.
    - coolspot 8y ago
      > The real use for chemical weapons is to terrify civilians and poorly organized/equipped militias. See Vietnam War, Napalm, Agent Orange etc.
      - gotocake 8y ago
        By that definition explosives are chemical weapons, and you know that isn’t what’s meant by the term. Napalm is an incendiary and equally effective against military and civilians. Agent Orange is a defoliant, and while it’s long term effects are horrendous, it’s not effective as a chemical weapon; you don’t want to sit around and wait for people to develop cancers.
        philwelch 8y ago
        Hell, by that standard, bullets are chemical weapons because they can cause lead poisoning.
- Datenstrom 8y ago
  A large issue with this is that like the DARPA ALIAS project[1], and another project which I do not have a reference to but saw a demonstration of last year which made a traditional helicopter autonomous with just a bolt on avionics box, it is entirely possible to have "manned" systems that are made autonomous with a bolt on. This means that any country could develop autonomous warfare systems entirely in secret and then convert everything to unmanned overnight just before an attack. You can't see software. [1]: http://www.aurora.aero/wp-content/uploads/2016/10/ALIAS-Brochure_X8.pdf http://www.aurora.aero/wp-content/uploads/2016/10/ALIAS-Broc... Edit, found a reference for the autonomous helicopter: https://www.navy.mil/submit/display.asp?story_id=103769 https://www.navy.mil/submit/display.asp?story_id=103769
  - binarymax 8y ago
    I don't understand your argument. Laws are created irregardless of the potential for illegal circumvention. Nobody ever decided "we shouldn't bother to outlaw theft because shoplifting is too easy". Creating the law provides the teeth to prosecute someone found to be breaking it.
    - Datenstrom 8y ago
      Agreed, the threat I am concerned about is development and preparation of deployment in secret of these systems by a country in secret. If outlawed and other countries obey the laws, and a bad actor develops in secret and waits for the tech to become sufficiently advanced, they could launch a __very__ effective blitzkrieg. Unlike chemical/nuclear warfare, no factories, material transportation, etc. out of the ordinary observable by other countries. Sensors needed for the systems would be entirely justified for other reasons and software to change behavior can be loaded invisibly. There would be no way to police it.
    - deleted 8y ago
      [deleted]
  - jacquesm 8y ago
    Here it is in flight and a shot of the interior as well as the sensor array bolted onto the nose: https://www.youtube.com/watch?v=WxfNChMNQyM https://www.youtube.com/watch?v=WxfNChMNQyM
Datenstrom 8y ago
Yes, the fact that these systems will be built by someone regardless needs to be addressed unless we want to become the equivalent of redcoats lining up in neat rows to fight people who are not playing by the rules.
xoa 8y ago
>if Russian or Chinese weapons are automatically firing at the speed of a neural network while our weapons are waiting for meatware actuation then it's going to be a short battle. While I see your point on the ever present escalation risk in these things, are you sure you're not taking an overly restrictive view of "pull the trigger"? Take long existing weapons like just plain old guided missiles. In one respect it could be argued a computer system is making the kill choice there, after firing there will be a long period where the computer is seeking the target and the computer is what actually detonates the warhead that makes the kill. But in practice it's hard to argue that a human isn't still "pulling the trigger", making the decision that "this target should be killed" and then the level of intelligence the weapon employs to accomplish that order afterwards isn't so relevant. The real comfort factor seems come from there being inherent limits in terms of time/space (a missile only has so much run time and range) and specificity (a missile is going after a specific designated target). If we consider the most problematic weapons of modern history, I think the real uniting factor is when they get away from all those limits at once. Landmines can linger a long time and activate indiscriminately without further human intervention. Chemical and bioweapons also not well controlled in terms of time, space or specificity despite no AI. Project Pluto, the early Cold War R&D by the US military to create a hypersonic nuclear scramjet cruise "missile" (more of a autonomous bomber) that could be launched then linger flying over the ocean for months (due to nuclear reactor power) was abandoned precisely because it was considered far too destabilizing even for then. Despite still being a "missile", the longer time/space/area effect made a difference. Heck, even cluster munitions on conventional missiles, while not considered in the same category as landmines exactly, are still pretty hotly debated. By the same token in a future conflict with more advanced NN/ML weapons, isn't there still room for operator decisions? Ie., "here are cryptographically authenticated orders to kill targets matching X profile within Y geographic area for Z time period". It would seem like setting international restrictions on X, Y and Z there (requirements for safety interlocks and for any given kill authorization to be valid for no more then 20 minutes over no more then 400 km^2 to toss something out) could be a viable approach. So in the specific minutes and seconds and milliseconds of machine/machine conflict there'd be no disadvantage, but humans would still be need to be constantly "involved" at a higher level. Having killbots just roaming around or lying in wait (and potentially lasting after the conflict is totally over, ala landmines) would be banned, but sending robot tanks to attack a specific fortified position on one single day with active strategic direction would not be. Basically moving humanity "up the chain" a bit seems like something countries could practically agree to without feeling massively behind and could be monitored/verified. Edit: updated to add some comparison to non-AI weapon systems that have been agreed to be restricted with some success.
decebalus1 8y ago
this just proves we haven't learned anything from the nuclear arms race.
PavlovsCat 8y ago
The line isn't between Russians and Chinese and American citizens, it's between the power centers within them and the citizens in their own countries as well as the others, and these power centers will be more loyal to each other than the respective populations they "represent" when it's time to consent and pay taxes for something, but which they don't know at all when it comes to dissidence or poverty. Robots firing at robots or soldiers isn't going to the end result, robots firing at civilians is. IMO the closing window of opportunity, that is, the fleeting opportunity that still exist, would be a culture of intellectual honesty and getting rid of moral double standards rather than goosestepping, so maybe people in Russia, China and elsewhere might think hey, maybe we'd actually like to be free, too. But, instead let's just play another round of this game. Remember the cold war, how giant that was, yet when it ended, the military budget just stayed as big? It's a racket.
alkonaut 8y ago
That’s why it needs to be a treaty. There needs to be an understanding that it’s as unacceptable as biological warfare and that the gloves are off once they are used. No one will use autonomous fire in some backwater proxy war skirmish (which is where superpowers meet) if the response would be the same as it would be to a nuclear first strike.
- i_am_proteus 8y ago
  Even in a controlled testing environment, it would be difficult to distinguish the signature of an autonomous weapon system from a remote-controlled unmanned system. On the battlefield, it will be impossible. The only difference between a system with a human firebreak and a fully autonomous weapon is the flip of a bit. Treaties will not be effective at preventing the development of these weapons. Any illusions will be shattered by the first bad actor. An arms race in this space is inevitable; realistically it has already begun.
  - alkonaut 8y ago
    That's true: you will have deniability. To begin with, all autonomous sysems will also be able to operate as remote controlled, or semi-autonomous with human triggering. If you are the kind of state that will, say, repaint military vehicles before sending them across the border to a country where you "have no military engagement" - you would also no doubt operate autonomous weapons and call them remote controlled. Still - using what amounts to a nuclear first strike is a dangerous game to play, so it might at least make superpowers think twice if it's made clear that this is how it's seen. Russia wouldn't try to use a bioweapon or tiny nuke and claim it was conventional, because it's a risky gamble.
- Recurecur 8y ago
  > There needs to be an understanding that it’s as unacceptable as biological warfare Are you under the impression that the major powers don't have biological warfare capabilities? > and that the gloves are off once they are used. So, China uses autonomous killbots in Tibet to control an uprising. The US "takes its gloves off", and...sends a strongly worded letter? Sends the world economy into a freefall? Further, China and Russia could do some limited testing, and then stockpile millions of killbots/killdrones. Once the gloves have already "come off", they can then rule the battlefield, since their opponent(s) won't have a similar massive force. I think the combination of cheap and effective will be impossible for the less scrupulous countries to resist...and possibly the more scrupulous countries as well. It is also a slippery slope - the US is already working on autonomous anti-ship cruise missiles that search an area and look for the correct ship (think "Chinese aircraft carrier"). That implicitly means the missile will autonomously "decide" which ship containing hundreds or thousands of sailors gets hit. Full autonomy will also be a requirement for any effective UCAS capability. I expect that regardless of what treaty is signed, the various powers will make sure there are loopholes permitting what they wish to do - or they will simply ignore the treaty. A concrete example of such behavior is Russia's long-term violations of the INF Treaty.

binarymax 8y ago

Similar to chemical warfare. There needs to be a UN resolution to ban these types of weapons. Make it so use of them constitutes war crimes.

philwelch 8y ago
Chemical weapons aren't used because they're not worth the risk. There was virtually zero battlefield use of chemical weapons in the Second World War, which was otherwise as "no holds barred" as you can imagine.
- gotocake 8y ago
  It’s not really risk that’s an issue, it’s a combination of expensive production, dangerous and expensive storage, and expensive deployment contrasted with a total lack of efficacy in killing the intended target. Unless your goal is to murder untrained and unequipped civilians, chemical weapons are just not very good weapons. Soldiers are trained, have vehicles with overpressure seals, masks and MOPP suits, detectors and a command structure. The real use for chemical weapons is to terrify civilians and poorly organized/equipped militias. Most of all it’s a way to murder a large number of civilians without destroying infrastructure. Needless to say that’s not something you’d typically do in a war, where destruction of infrastructure is typically step one! As a result most countries recognize the economic and social burden of chemical weapons and their use, and strongly oppose them.
  - coolspot 8y ago
    > The real use for chemical weapons is to terrify civilians and poorly organized/equipped militias. See Vietnam War, Napalm, Agent Orange etc.
    - gotocake 8y ago
      By that definition explosives are chemical weapons, and you know that isn’t what’s meant by the term. Napalm is an incendiary and equally effective against military and civilians. Agent Orange is a defoliant, and while it’s long term effects are horrendous, it’s not effective as a chemical weapon; you don’t want to sit around and wait for people to develop cancers.
      - philwelch 8y ago
        Hell, by that standard, bullets are chemical weapons because they can cause lead poisoning.
Datenstrom 8y ago
A large issue with this is that like the DARPA ALIAS project[1], and another project which I do not have a reference to but saw a demonstration of last year which made a traditional helicopter autonomous with just a bolt on avionics box, it is entirely possible to have "manned" systems that are made autonomous with a bolt on. This means that any country could develop autonomous warfare systems entirely in secret and then convert everything to unmanned overnight just before an attack. You can't see software. [1]: http://www.aurora.aero/wp-content/uploads/2016/10/ALIAS-Brochure_X8.pdf http://www.aurora.aero/wp-content/uploads/2016/10/ALIAS-Broc... Edit, found a reference for the autonomous helicopter: https://www.navy.mil/submit/display.asp?story_id=103769 https://www.navy.mil/submit/display.asp?story_id=103769
- binarymax 8y ago
  I don't understand your argument. Laws are created irregardless of the potential for illegal circumvention. Nobody ever decided "we shouldn't bother to outlaw theft because shoplifting is too easy". Creating the law provides the teeth to prosecute someone found to be breaking it.
  - Datenstrom 8y ago
    Agreed, the threat I am concerned about is development and preparation of deployment in secret of these systems by a country in secret. If outlawed and other countries obey the laws, and a bad actor develops in secret and waits for the tech to become sufficiently advanced, they could launch a __very__ effective blitzkrieg. Unlike chemical/nuclear warfare, no factories, material transportation, etc. out of the ordinary observable by other countries. Sensors needed for the systems would be entirely justified for other reasons and software to change behavior can be loaded invisibly. There would be no way to police it.
  - deleted 8y ago
    [deleted]
- jacquesm 8y ago
  Here it is in flight and a shot of the interior as well as the sensor array bolted onto the nose: https://www.youtube.com/watch?v=WxfNChMNQyM https://www.youtube.com/watch?v=WxfNChMNQyM

Datenstrom 8y ago

Yes, the fact that these systems will be built by someone regardless needs to be addressed unless we want to become the equivalent of redcoats lining up in neat rows to fight people who are not playing by the rules.

xoa 8y ago

>if Russian or Chinese weapons are automatically firing at the speed of a neural network while our weapons are waiting for meatware actuation then it's going to be a short battle. While I see your point on the ever present escalation risk in these things, are you sure you're not taking an overly restrictive view of "pull the trigger"? Take long existing weapons like just plain old guided missiles. In one respect it could be argued a computer system is making the kill choice there, after firing there will be a long period where the computer is seeking the target and the computer is what actually detonates the warhead that makes the kill. But in practice it's hard to argue that a human isn't still "pulling the trigger", making the decision that "this target should be killed" and then the level of intelligence the weapon employs to accomplish that order afterwards isn't so relevant. The real comfort factor seems come from there being inherent limits in terms of time/space (a missile only has so much run time and range) and specificity (a missile is going after a specific designated target). If we consider the most problematic weapons of modern history, I think the real uniting factor is when they get away from all those limits at once. Landmines can linger a long time and activate indiscriminately without further human intervention. Chemical and bioweapons also not well controlled in terms of time, space or specificity despite no AI. Project Pluto, the early Cold War R&D by the US military to create a hypersonic nuclear scramjet cruise "missile" (more of a autonomous bomber) that could be launched then linger flying over the ocean for months (due to nuclear reactor power) was abandoned precisely because it was considered far too destabilizing even for then. Despite still being a "missile", the longer time/space/area effect made a difference. Heck, even cluster munitions on conventional missiles, while not considered in the same category as landmines exactly, are still pretty hotly debated. By the same token in a future conflict with more advanced NN/ML weapons, isn't there still room for operator decisions? Ie., "here are cryptographically authenticated orders to kill targets matching X profile within Y geographic area for Z time period". It would seem like setting international restrictions on X, Y and Z there (requirements for safety interlocks and for any given kill authorization to be valid for no more then 20 minutes over no more then 400 km^2 to toss something out) could be a viable approach. So in the specific minutes and seconds and milliseconds of machine/machine conflict there'd be no disadvantage, but humans would still be need to be constantly "involved" at a higher level. Having killbots just roaming around or lying in wait (and potentially lasting after the conflict is totally over, ala landmines) would be banned, but sending robot tanks to attack a specific fortified position on one single day with active strategic direction would not be. Basically moving humanity "up the chain" a bit seems like something countries could practically agree to without feeling massively behind and could be monitored/verified. Edit: updated to add some comparison to non-AI weapon systems that have been agreed to be restricted with some success.

decebalus1 8y ago

this just proves we haven't learned anything from the nuclear arms race.

PavlovsCat 8y ago

The line isn't between Russians and Chinese and American citizens, it's between the power centers within them and the citizens in their own countries as well as the others, and these power centers will be more loyal to each other than the respective populations they "represent" when it's time to consent and pay taxes for something, but which they don't know at all when it comes to dissidence or poverty. Robots firing at robots or soldiers isn't going to the end result, robots firing at civilians is. IMO the closing window of opportunity, that is, the fleeting opportunity that still exist, would be a culture of intellectual honesty and getting rid of moral double standards rather than goosestepping, so maybe people in Russia, China and elsewhere might think hey, maybe we'd actually like to be free, too. But, instead let's just play another round of this game. Remember the cold war, how giant that was, yet when it ended, the military budget just stayed as big? It's a racket.

alkonaut 8y ago

That’s why it needs to be a treaty. There needs to be an understanding that it’s as unacceptable as biological warfare and that the gloves are off once they are used. No one will use autonomous fire in some backwater proxy war skirmish (which is where superpowers meet) if the response would be the same as it would be to a nuclear first strike.

i_am_proteus 8y ago
Even in a controlled testing environment, it would be difficult to distinguish the signature of an autonomous weapon system from a remote-controlled unmanned system. On the battlefield, it will be impossible. The only difference between a system with a human firebreak and a fully autonomous weapon is the flip of a bit. Treaties will not be effective at preventing the development of these weapons. Any illusions will be shattered by the first bad actor. An arms race in this space is inevitable; realistically it has already begun.
- alkonaut 8y ago
  That's true: you will have deniability. To begin with, all autonomous sysems will also be able to operate as remote controlled, or semi-autonomous with human triggering. If you are the kind of state that will, say, repaint military vehicles before sending them across the border to a country where you "have no military engagement" - you would also no doubt operate autonomous weapons and call them remote controlled. Still - using what amounts to a nuclear first strike is a dangerous game to play, so it might at least make superpowers think twice if it's made clear that this is how it's seen. Russia wouldn't try to use a bioweapon or tiny nuke and claim it was conventional, because it's a risky gamble.
Recurecur 8y ago
> There needs to be an understanding that it’s as unacceptable as biological warfare Are you under the impression that the major powers don't have biological warfare capabilities? > and that the gloves are off once they are used. So, China uses autonomous killbots in Tibet to control an uprising. The US "takes its gloves off", and...sends a strongly worded letter? Sends the world economy into a freefall? Further, China and Russia could do some limited testing, and then stockpile millions of killbots/killdrones. Once the gloves have already "come off", they can then rule the battlefield, since their opponent(s) won't have a similar massive force. I think the combination of cheap and effective will be impossible for the less scrupulous countries to resist...and possibly the more scrupulous countries as well. It is also a slippery slope - the US is already working on autonomous anti-ship cruise missiles that search an area and look for the correct ship (think "Chinese aircraft carrier"). That implicitly means the missile will autonomously "decide" which ship containing hundreds or thousands of sailors gets hit. Full autonomy will also be a requirement for any effective UCAS capability. I expect that regardless of what treaty is signed, the various powers will make sure there are loopholes permitting what they wish to do - or they will simply ignore the treaty. A concrete example of such behavior is Russia's long-term violations of the INF Treaty.