8 ms·

The substantive part of this conversation comes about 35 minutes in, when Allen describes the DoD's five AI ethics principles: that any deployed AI systems must

by lwneal 7y ago

The substantive part of this conversation comes about 35 minutes in, when Allen describes the DoD's five AI ethics principles: that any deployed AI systems must be "Responsible, Equitable, Traceable, Reliable, and Governable." [1]. I think "Responsible" raises some hard questions. Today, any weapons use by a robot might require a responsible human operator to first (remotely) pull the trigger. But what if an adversary allows their robots to fire without waiting for human permission? The first side to do so would gain a critical advantage, and we cannot have an autonomy gap. Some have raised conerns that in the extreme case, this could lead to a world of machines anonymously killing without attribution. If you haven't, see Stuart Russell's compelling short film "Slaughterbots" [2]. [1] https://media.defense.gov/2019/Oct/31/2002204459/-1/-1/0/DIB_AI_PRINCIPLES_SUPPORTING_DOCUMENT.PDF https://media.defense.gov/2019/Oct/31/2002204459/-1/-1/0/DIB... [2] https://www.youtube.com/watch?v=9CO6M2HsoIA https://www.youtube.com/watch?v=9CO6M2HsoIA

dx87 7y ago
Autonomous killing could be something that needs to be added to international laws of war. The USA already follows restrictions that could give it an advantage in war, but it doesn't because of international agreements. For example, our military has to wear uniforms during war, and we have to follow proportional force that restricts what weapons can be used in response to an attack. Countries largely follow the rules because nobody wants a free-for-all war with no rules.
- tehjoker 7y ago
  The US follows rules to the extent they can get away with breaking them, no more no less. Witness the Hauge invasion act of the early 'oughts that sought to avoid accountability by the international criminal court by threatening to invade it if a US military person was ever put on the dock. https://en.wikipedia.org/wiki/American_Service-Members%27_Protection_Act https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...
  - umvi 7y ago
    I disagree. I think the majority of us military personnel are good people and would disobey orders to attack civilians or otherwise commit evil acts (even if it was in the middle of nowhere and they could get away with it).
    - threatripper 7y ago
      Of course the majority of people is good people almost everywhere and yet our prisons are full with bad people that we need to lock away because they disobeyed the written rules of society and couldn't get away with it.
    - seer 7y ago
      In this context the story of Hugh Thompson [1] is very interesting - a clear cut “kill the civilians“ order and people risking their careers and lives to disobey and protect the village. Whats relevant though is that most soldiers obeyed the orders and preferred to commit war crimes. Even in the US military which has a strong moral ethos (or likes to portray itself that way which for the common soldier is the same thing) But what about all the other, smaller countries? [1]https://en.m.wikipedia.org/wiki/Hugh_Thompson_Jr https://en.m.wikipedia.org/wiki/Hugh_Thompson_Jr.
    - nitwit005 7y ago
      Possibly, in a conflict where the US isn't particularly threatened. The last time the US felt in real danger it flattened German and Japanese cities, and dropped two nukes. I don't feel we've become profoundly nicer since then.
      - umvi 7y ago
        We used the weapons we had at the time to stop a World War. We've definitely become profoundly nicer since then. When was the last time we flattened a city or dropped a nuke? We've since developed PGMs which allow us to minimize collateral damage more than ever before. We can destroy strategic targets of military interest with pinpoint accuracy. People that commit war crimes are reported by their fellow soldiers and justice is (usually) served. Sometimes war criminals are pardoned even after being turned in (like Gallagher), but this is rare and usually results in national outrage.
- catalogia 7y ago
  How would autonomous killing be precisely defined? Whenever a machine is designed to kill without an affirmative command from a human operator? Would that include landminds? Maybe it should...
  - pageandrew 7y ago
    good point. autonomous killing is nothing new.
  - joshvm 7y ago
    There are plenty of fire-and-forget weapons that will attack a designated target autonomously. Some may even be totally automated, like CIWS, if you put them in some paranoid mode. A landmine is indiscriminate, by that logic anything with a triggerable fuze is autonomous (even a snare trap would be). The difference would be if the landmine chose to explode based on some other factor. Autonomy and intelligence are not the same thing.
    - satanspastaroll 7y ago
      Well landmines can discriminate by atleast weight and magnetism. I think they can also count the tracks and blow up later if they want to
      - cf141q5325 7y ago
        The action is however caused by a human who mined the area, its just delayed, similar to how a grenade has a delay after the human activation, before it explodes. You can make the same distinction with an automatic gun or turret. You position it in an area in which you decide to kill anyone in it. Autonomy gets truly dangerous because you dont need to create death zones and no mans land. They are not a less violent version of just bombing the area. They can run considerations and decide by themselves who to kill and with this, their usage outside of active front lines will be considered. And with that, the people deploying them give up the decision and delegate it to an algorithm or worse yet, a trained black box. Reality will be to have completely unaccountable killers bots flying over countries who cant defend themselves against foreign aggression. This stuff has absolutely horrible potential and i find it prudent to shame and cut out anyone with a sufficient lack of a spine to work in this area. As unpopular as it is to consider the impact of the technology you are developing, it is taught for a reason in a CS degree. This work has the potential to screw over humanity for a while to come.
- Mirioron 7y ago
  >Countries largely follow the rules because nobody wants a free-for-all war with no rules. And somehow these rules tend to put restrictions on things which are a force multiplier for smaller forces against larger ones or would make it very costly for a larger country to attack a smaller one. They protect the incumbents. It makes sense why big militaries would want rules like this - this makes it much less costly for big countries to war with smaller ones. Russia didn't want Ukraine to have nukes. A few decades later Russia invaded Ukraine. It is okay for a US drone to strike a target and kill civilians. The civilians will just be defined as "enemy combatants" and the world moves on. However, if people from that same place somehow manage to kill the drone operator then that would be a terror attack.
  - gdy 7y ago
    "Russia didn't want Ukraine to have nukes." The most concerned party at that time was the US. Having multiple nuclear armed ex-USSR states was contrary to their interests. The better example is the US not wanting Gaddafi to have nukes and later invading Libya.
DoingIsLearning 7y ago
> Traceability A question for the AI Researchers and Engineers On the topic of _Traceability_ and as someone outside the field this is something that has puzzled me: (Broad assumptions) Assuming a 'neural network' is a black box probabilistic model. How do we _prove_ in the context of _safety-critical_ systems that they will behave in a known determined and traceable way for all known cases? I would be curious to understand the process/methods that people use in ADAS system or Medical Imaging (and now Defense applications as well) in order to provide 'verification' evidence for such a system? (Focusing more on verification/provability from a regulatory standpoint)
- kuu 7y ago
  There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example. Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable. [0] https://christophm.github.io/interpretable-ml-book/neural-networks.html https://christophm.github.io/interpretable-ml-book/neural-ne...
  - DoingIsLearning 7y ago
    Indeed ML is vast and not limited to NN. My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry. The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.) Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)
    - yoav_hollander 7y ago
      Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles): https://blog.foretellix.com/2017/07/06/where-machine-learning-meets-rule-based-verification/ https://blog.foretellix.com/2017/07/06/where-machine-learnin...
yorwba 7y ago
> But what if an adversary allows their robots to fire without waiting for human permission? The first side to do so would gain a critical advantage, and we cannot have an autonomy gap. You're making the bold assumption that the robots will actually work, instead of wasting ammunition on all the wrong targets, including friendlies. I'm pretty sure the first side to allow fully autonomous weapons will do so before the technology is ready (humans are never patient enough to wait until the technology is ready) and end up regretting it.
- speedplane 7y ago
  > You're making the bold assumption that the robots will actually work, instead of wasting ammunition I'm pretty skeptical the U.S. military is concerned about wasted ammunition. They waste so much on training and preparation that anything wasted in combat is minimal. > I'm pretty sure the first side to allow fully autonomous weapons will do so before the technology is ready It doesn't have to be autonomous to be deployed before it's ready. Iran managed to should down a civil aircraft based on human error. The U.S. has done the same decades earlier. Clearly there were either technology or human process problems, I suspect that autonomous military robots will follow this trend.
  - mcv 7y ago
    There have also been instances during the war in Iraq where human soldiers clearly had no problem shooting civilians. It's not something that's unique to autonomous systems, though I suppose it might terrify some people a bit more than when it's humans doing it.
  - bumby 7y ago
    >I'm pretty skeptical the U.S. military is concerned about wasted ammunition. They waste so much on training and preparation that anything wasted in combat is minimal. They may not care about the ammunition itself so much but they care about budget and combat effectiveness. It's not uncommon to have training be missed due to ammunition budget shortfalls and a robot without ammunition isn't terribly effective.
- lordlic 7y ago
arh68 7y ago
"Responsible" seems pretty easy: follow the rules of engagement. What if a convoy passes a group of nuns? What if the nuns "fire without waiting"? Should we let the nuns have the "critical advantage" of first shot? Well, yes, it'd be absurd to do otherwise. If some other country wants to develop auto kill bots that commit war crimes, I don't think we have an imperative to match them.
- ethbro 7y ago
  That's not the problem parent is talking about. They're talking about an autonomy gap. Tolerant rules of engagement are feasible because reactions are "similar" in timescale. Enemy fires on you, you fire back, etc. But there are many developments that make that system less tenable. Firstly, increased weapon lethality. When the first shot has a 90%+ kill probability (say, putting a Hellfire missile on a civilian vehicle), can you afford to let the enemy take the first shot? Or do your rules account for reality and back up a step? Maybe now, radar detection, firing preparations, or even entering a given airspace are sufficiently provocative to counterattack. In literature and experience, this is casualty modeling moving from attrition (some constant, scaled by time) to volley (majority of casualties caused in bursts, at specific points in time). Secondly, and this specific issue: reaction time between autonomous vs manual systems. If an opponent is using autonomous targeting and engagement, and you're using autonomous targeting and manual engagement, there may be some orders of magnitude difference in reaction time. Coupled with the first point, that may mean your entire force is wiped out before you have a chance to react.
  - bumby 7y ago
    The idea of a preemptive attack in the face of potentially large risks is nothing new and not unique to weapons utilizing AI.[1][2] [1] https://www.jstor.org/stable/3109845?seq=1 https://www.jstor.org/stable/3109845?seq=1 [2] https://en.wikipedia.org/wiki/The_One_Percent_Doctrine https://en.wikipedia.org/wiki/The_One_Percent_Doctrine
    - ethbro 7y ago
      It's not. And it's led to a reevaluation of rules in those other spaces too.

by lwneal 7y ago

dx87 7y ago

Autonomous killing could be something that needs to be added to international laws of war. The USA already follows restrictions that could give it an advantage in war, but it doesn't because of international agreements. For example, our military has to wear uniforms during war, and we have to follow proportional force that restricts what weapons can be used in response to an attack. Countries largely follow the rules because nobody wants a free-for-all war with no rules.

tehjoker 7y ago
The US follows rules to the extent they can get away with breaking them, no more no less. Witness the Hauge invasion act of the early 'oughts that sought to avoid accountability by the international criminal court by threatening to invade it if a US military person was ever put on the dock. https://en.wikipedia.org/wiki/American_Service-Members%27_Protection_Act https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...
- umvi 7y ago
  I disagree. I think the majority of us military personnel are good people and would disobey orders to attack civilians or otherwise commit evil acts (even if it was in the middle of nowhere and they could get away with it).
  - threatripper 7y ago
    Of course the majority of people is good people almost everywhere and yet our prisons are full with bad people that we need to lock away because they disobeyed the written rules of society and couldn't get away with it.
  - seer 7y ago
    In this context the story of Hugh Thompson [1] is very interesting - a clear cut “kill the civilians“ order and people risking their careers and lives to disobey and protect the village. Whats relevant though is that most soldiers obeyed the orders and preferred to commit war crimes. Even in the US military which has a strong moral ethos (or likes to portray itself that way which for the common soldier is the same thing) But what about all the other, smaller countries? [1]https://en.m.wikipedia.org/wiki/Hugh_Thompson_Jr https://en.m.wikipedia.org/wiki/Hugh_Thompson_Jr.
  - nitwit005 7y ago
    Possibly, in a conflict where the US isn't particularly threatened. The last time the US felt in real danger it flattened German and Japanese cities, and dropped two nukes. I don't feel we've become profoundly nicer since then.
    - umvi 7y ago
      We used the weapons we had at the time to stop a World War. We've definitely become profoundly nicer since then. When was the last time we flattened a city or dropped a nuke? We've since developed PGMs which allow us to minimize collateral damage more than ever before. We can destroy strategic targets of military interest with pinpoint accuracy. People that commit war crimes are reported by their fellow soldiers and justice is (usually) served. Sometimes war criminals are pardoned even after being turned in (like Gallagher), but this is rare and usually results in national outrage.
catalogia 7y ago
How would autonomous killing be precisely defined? Whenever a machine is designed to kill without an affirmative command from a human operator? Would that include landminds? Maybe it should...
- pageandrew 7y ago
  good point. autonomous killing is nothing new.
- joshvm 7y ago
  There are plenty of fire-and-forget weapons that will attack a designated target autonomously. Some may even be totally automated, like CIWS, if you put them in some paranoid mode. A landmine is indiscriminate, by that logic anything with a triggerable fuze is autonomous (even a snare trap would be). The difference would be if the landmine chose to explode based on some other factor. Autonomy and intelligence are not the same thing.
  - satanspastaroll 7y ago
    Well landmines can discriminate by atleast weight and magnetism. I think they can also count the tracks and blow up later if they want to
    - cf141q5325 7y ago
      The action is however caused by a human who mined the area, its just delayed, similar to how a grenade has a delay after the human activation, before it explodes. You can make the same distinction with an automatic gun or turret. You position it in an area in which you decide to kill anyone in it. Autonomy gets truly dangerous because you dont need to create death zones and no mans land. They are not a less violent version of just bombing the area. They can run considerations and decide by themselves who to kill and with this, their usage outside of active front lines will be considered. And with that, the people deploying them give up the decision and delegate it to an algorithm or worse yet, a trained black box. Reality will be to have completely unaccountable killers bots flying over countries who cant defend themselves against foreign aggression. This stuff has absolutely horrible potential and i find it prudent to shame and cut out anyone with a sufficient lack of a spine to work in this area. As unpopular as it is to consider the impact of the technology you are developing, it is taught for a reason in a CS degree. This work has the potential to screw over humanity for a while to come.
Mirioron 7y ago
>Countries largely follow the rules because nobody wants a free-for-all war with no rules. And somehow these rules tend to put restrictions on things which are a force multiplier for smaller forces against larger ones or would make it very costly for a larger country to attack a smaller one. They protect the incumbents. It makes sense why big militaries would want rules like this - this makes it much less costly for big countries to war with smaller ones. Russia didn't want Ukraine to have nukes. A few decades later Russia invaded Ukraine. It is okay for a US drone to strike a target and kill civilians. The civilians will just be defined as "enemy combatants" and the world moves on. However, if people from that same place somehow manage to kill the drone operator then that would be a terror attack.
- gdy 7y ago
  "Russia didn't want Ukraine to have nukes." The most concerned party at that time was the US. Having multiple nuclear armed ex-USSR states was contrary to their interests. The better example is the US not wanting Gaddafi to have nukes and later invading Libya.

DoingIsLearning 7y ago

> Traceability A question for the AI Researchers and Engineers On the topic of _Traceability_ and as someone outside the field this is something that has puzzled me: (Broad assumptions) Assuming a 'neural network' is a black box probabilistic model. How do we _prove_ in the context of _safety-critical_ systems that they will behave in a known determined and traceable way for all known cases? I would be curious to understand the process/methods that people use in ADAS system or Medical Imaging (and now Defense applications as well) in order to provide 'verification' evidence for such a system? (Focusing more on verification/provability from a regulatory standpoint)

kuu 7y ago
There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example. Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable. [0] https://christophm.github.io/interpretable-ml-book/neural-networks.html https://christophm.github.io/interpretable-ml-book/neural-ne...
- DoingIsLearning 7y ago
  Indeed ML is vast and not limited to NN. My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry. The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.) Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)
  - yoav_hollander 7y ago
    Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles): https://blog.foretellix.com/2017/07/06/where-machine-learning-meets-rule-based-verification/ https://blog.foretellix.com/2017/07/06/where-machine-learnin...

yorwba 7y ago

> But what if an adversary allows their robots to fire without waiting for human permission? The first side to do so would gain a critical advantage, and we cannot have an autonomy gap. You're making the bold assumption that the robots will actually work, instead of wasting ammunition on all the wrong targets, including friendlies. I'm pretty sure the first side to allow fully autonomous weapons will do so before the technology is ready (humans are never patient enough to wait until the technology is ready) and end up regretting it.

speedplane 7y ago
> You're making the bold assumption that the robots will actually work, instead of wasting ammunition I'm pretty skeptical the U.S. military is concerned about wasted ammunition. They waste so much on training and preparation that anything wasted in combat is minimal. > I'm pretty sure the first side to allow fully autonomous weapons will do so before the technology is ready It doesn't have to be autonomous to be deployed before it's ready. Iran managed to should down a civil aircraft based on human error. The U.S. has done the same decades earlier. Clearly there were either technology or human process problems, I suspect that autonomous military robots will follow this trend.
- mcv 7y ago
  There have also been instances during the war in Iraq where human soldiers clearly had no problem shooting civilians. It's not something that's unique to autonomous systems, though I suppose it might terrify some people a bit more than when it's humans doing it.
- bumby 7y ago
  >I'm pretty skeptical the U.S. military is concerned about wasted ammunition. They waste so much on training and preparation that anything wasted in combat is minimal. They may not care about the ammunition itself so much but they care about budget and combat effectiveness. It's not uncommon to have training be missed due to ammunition budget shortfalls and a robot without ammunition isn't terribly effective.
lordlic 7y ago

arh68 7y ago

"Responsible" seems pretty easy: follow the rules of engagement. What if a convoy passes a group of nuns? What if the nuns "fire without waiting"? Should we let the nuns have the "critical advantage" of first shot? Well, yes, it'd be absurd to do otherwise. If some other country wants to develop auto kill bots that commit war crimes, I don't think we have an imperative to match them.

ethbro 7y ago
That's not the problem parent is talking about. They're talking about an autonomy gap. Tolerant rules of engagement are feasible because reactions are "similar" in timescale. Enemy fires on you, you fire back, etc. But there are many developments that make that system less tenable. Firstly, increased weapon lethality. When the first shot has a 90%+ kill probability (say, putting a Hellfire missile on a civilian vehicle), can you afford to let the enemy take the first shot? Or do your rules account for reality and back up a step? Maybe now, radar detection, firing preparations, or even entering a given airspace are sufficiently provocative to counterattack. In literature and experience, this is casualty modeling moving from attrition (some constant, scaled by time) to volley (majority of casualties caused in bursts, at specific points in time). Secondly, and this specific issue: reaction time between autonomous vs manual systems. If an opponent is using autonomous targeting and engagement, and you're using autonomous targeting and manual engagement, there may be some orders of magnitude difference in reaction time. Coupled with the first point, that may mean your entire force is wiped out before you have a chance to react.
- bumby 7y ago
  The idea of a preemptive attack in the face of potentially large risks is nothing new and not unique to weapons utilizing AI.[1][2] [1] https://www.jstor.org/stable/3109845?seq=1 https://www.jstor.org/stable/3109845?seq=1 [2] https://en.wikipedia.org/wiki/The_One_Percent_Doctrine https://en.wikipedia.org/wiki/The_One_Percent_Doctrine
  - ethbro 7y ago
    It's not. And it's led to a reevaluation of rules in those other spaces too.