9 ms·

What's wrong with wanting a “human in the loop”?

75 points by Symmetry 4y ago

n4jm4 4y ago
The natural course of war tends toward a future where polities own, but do not operate, fully autonomous weapons. There will come a day when even the targets, be they military or civilian, are selected and attacked solely by machines. The human element is more creative, though bound by a sense of morality. Whereas the machine is faster, bound only by the thinnest of rule sets. If we're lucky, most of the combat occurs virtually rather than mechanically.
jsmith45 4y ago
Keeping a human in the loop is more about beeing willing to trust the tech than about safety. Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched if there were no enemies so obviously the one and only target it sees must be an enemy, and turns around and kills you. After that happens just a few times, the other soldiers would revolt and refuse to use the weapon system anymore. That is the sort of thing that motivates having humans in the loop, even if on average there would be greater safety without humans in the loop.
- xyzzy123 4y ago
  "This side towards enemy'. Doesn't have to be "safe" / "not dangerous" (a lot of weapons systems are very dangerous in different ways) but it should be predictable, using some reasonably simple mental model. "Refusing to deploy" might not be a sensible option if your enemy is willing to deploy. It seems safe to predict that human in the loop systems will be quickly overwhelmed by fully autonomous systems. The tradeoff may end up being having to balance the risk of getting killed by your own systems versus being killed by your enemy's systems, because yours were too "tame".
  - Frost1x 4y ago
    >"Refusing to deploy" might not be a sensible option if your enemy is willing to deploy. In the AI ethics arena, this is really one of the best arguments I've heard for the military to pursue and have automated systems in place. US leadership may agree and see all the issues with using killer robots but other countries are developing the same technology and may have less ethical concern and considerations then you do. They may be willing to let these systems fail and kill innocent bystanders or even their own soldiers. They may even look at these and consider it a necessary loss to move forward and succeed in their goals. As such, if you're not willing to make such compromises as well, unless your approach ultimately achieves the goal or you have other techniques to mitigate these corner cutting strategies, you too must cave. If China decides to engage in conflict and releases killer robots and we don't have killer robots, can we contend? If we can't, we might have to engage in the race to the bottom that is competition unless we have an alternative mitigation strategy that doesn't require us to compromise our ethics. I imagine this is the dilemma scientists faced during WWII when developing nuclear weapons. Part of the argument that lead to nuclear weapons was that Germany was already working on the process. This resulted in other countries engaging and ultimately building successful weapons where Germany failed. Ultimately, we now live in a world where lots of people have nuclear weapons... and overall it's been the proverbial Mexican standoff with nuclear deterrence strategies. Is there an AI deterrence strategy? Nuclear worked because it lead to the case of a global collapse in terms of what nuclear war would bring. Killer robots that aren't sentient don't seem to have quite the same threat and seem like militaries could utilize them as they fall in an ethical grey area globally and have far less risk of ending humanity on a global scale unless we start talking about sentient terminator killer robot level uprising which is not even remotely where AI is currently or in the foreseeable future.
    - deleted 4y ago
      [deleted]
    - andi999 4y ago
      I think currently no other country is even close to the level of autonomous weapon systems the US is developing. An arms race where you are miles ahead cannot be used as a reason why you have tonrun faster.
- j7ake 4y ago
  I think you’ve hit on the concept when humans in the loop are needed. For many black box systems, the average performance may be comparable or better than humans, but when errors occur, they should fail predictably. If systems fail unpredictably (ie unbounded in the types of errors it makes), then they can never be deployed in the field without humans in the loop. This is why self driving cars can, at the same time, be on average safer but still not be accepted by public.
  - Gordonjcp 4y ago
    > This is why self driving cars can, at the same time, be on average safer but still not be accepted by public. They're not safer, though.
- venkat223 4y ago
  Being an expert boss in AI and expert controller in loop in situ are two different perspectives.AI experts will be useless in situ.
- walrus01 4y ago
  > Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched or more realistically, your autonomous version of an aerovironment switchblade launches, has image recognition/onboard trained neural network that can identify a russian spec T-72 tank or BMP or similar, doesn't see any, loiters for a while and then self destructs.
darkerside 4y ago
This article is thoughtful and naive at the same time. Deploying an autonomous weapon is just like deploying any other weapon. Once a gun is fired, the human is out of the loop. Autonomous weapons will end up the same way. People will get to exercise judgement early and late, but AI will make decisions during execution, just like a bullet follows the flight path. The reason we need humans is for accountability. You can't court marshal a machine. Firing the operator doesn't fix anything or prevent it from happening again (next operator will be just as fallible), it just satisfies some primal part of our brains that needs to blame someone when something bad happens.
- taylorius 4y ago
  The difference lies in the predictability, I would say. A bullet flies according to well understood equations of motion, whereas an A.I - no-one really knows how it might be fooled, or what mistakes it could make.
  - darkerside 4y ago
    I guess the better analogue is directing troops.
  - jameshart 4y ago
    Cutting the wires and letting a torpedo run in active homing mode and find and select its own target has been part of submarine warfare practice for decades. We tolerate it because there aren’t generally a lot of civilians or even friendly fire risks in submarine engagement environments.
- jayd16 4y ago
  That's one way to think about it. However, don't we usually just outright ban weapons with longer term lethality? A bullet is lethal for seconds while mines are lethal for years. If autonomous drones are lethal for days, weeks or even months, should we consider them a war crime?
  - darkerside 4y ago
sgt101 4y ago
>When that happens, we identify why the interaction is problematic, why the human is not trusting the machine, or why the way that the data is presented is being misunderstood — and then we change those things. Such changes should make us question to what extent human oversight of algorithms is truly meaningful. Nonsense; this is like saying "we changed the wheel on the bike so that the rider didn't fall off it so often which makes us question if they can ride the bike". If the interface is being changed to better deceive a user then that of course is problematic, but if it is being changed to allow the user to make accurate decisions then that is the function of the system. >But now imagine a system similar to HARPY, a “fire-and-forget” system that targets enemy combatants in an area or enemy ships (to use an example that minimizes the collateral risk). Proponents of meaningful human control do not believe it should ever be left up to a machine to target humans. Strawman - that is not what "Proponents of meaningful human control" believe, I am sure you can find some people who say that, but then they will also say that they don't believe that a bullet should be fired or a bomb dropped. Machines target humans all the time in the sense that unintended collateral damage occurs whenever bombs are dropped. The humans doing it must accept responsibility for this; that is the same for more sophisticated processes - the humans remain on the hook for unleashing the device. If the device decides to slaughter a primary school class that's on you in the same way as it is if you target a school yourself. >I think there are good reasons to question this. Specifically, I am not persuaded that there is a significant moral difference between an operator identifying a single human target based on some data and a human operator drawing a box and defining targets within that box based on equally good data. Which shows that you, dear writer, should never be allowed to decide a god damn thing. There is a chasm of uncertainty between the two scenarios, and a chasm of indirection. >whether in cases when the system works better or as well without a human operator, we have moral reasons to place a human in the loop. Because we live in an open world and the judgement of performance is post hoc, you can't know that it will work until it's done its thing. The judgement has to be when the act is commissioned because the consequences of the act cannot be reversed.
verisimi 4y ago
I can't get past the first paragraph! > At the recent conference on the ethics of AI-enabled weapons systems at the U.S. Naval Academy, well over half the talks discussed meaningful human control of AI to some extent. But, but AI is sentient right?! What business is it of ours to try to exercise control over AI, with its many and various superior attributes? lol. > If you work among the AI ethics community, and especially among those working on AI ethics and governance for the military, you are hard-pressed to find an article or enter a room without stumbling on someone literally or metaphorically slamming their fist on the table while exalting the importance of human control over AI and especially AI-enabled weapons. AI ethics is only there to justify the unjustifiable. They do not have my ethics that's for sure, nor any normal person's. Its a bad joke that allows a corporation to say - 'but it has been subject to an ethical review!' - and the political class can allow themselves to be bamboozled and provide their rubber stamp. > the ethics of AI-enabled weapons ... the ethical implications fall on those coding the dystopia. War is immoral, pretending that wars are real is immoral, handing over decision of life and death to a machine is immoral - regardless of how efficient the machine may be. So, any of those people involved in that circle are immoral. Soldiers may think they are working to defend their 'country' - but 'country' is a fiction and 'defence' is an inversion. People act this way for money.. at least that reason makes sense - but there is nothing moral about it. There are no ethics to be found there.
- astrange 4y ago
  > Soldiers may think they are working to defend their 'country' - but 'country' is a fiction and 'defence' is an inversion. Meaning is collaborative. It doesn’t matter if you’re a nihilist and want to declare your country doesn’t exist; the guy invading you makes it real. You can be a quisling, but defecting to the invader seems to admit they exist, at least.
  - verisimi 4y ago
    Who's invading who now? What are the soldiers "defending"? Do you think that 'Department of Defense' is a misnomer? Howabout 'military aid'? Are these the sorts of linguistic meanings we are meant to collaborate on? I'm no collaborator, you know..
theelous3 4y ago
The article presents two sides - one end advocates humans taking an active role in the loop, like confirming a drone's target classification and authorising it to act lethally. The other side is where the human's role is to shape the operating envelope to what they think will have parameters appropriate for the ethics of the situation. The examples of the systems seems completely out of whack though. The active participant advocates are clearly taking about things like air to ground hunter killer missions on ambiguous targets. For example, a civilian vehicle converted to technical vs a plumber hauling some piping. A farmer with a weed killer in a ditch by the road vs an insurgent constructing an ambush. These are clearly beyond the realm of what the other side is talking about - easily identifying shit like russian airforce craft with no friendly transponders and armed active radar payloads under its wings. I don't understand the conversation that is trying to take place. Clearly they are too different to even consider. Is the conversation supposed to be hypothetical enough that we can pretend we can act in the first case, as we can in the second? Then the conversation seems pointless. Of course you can, if you accept the premise of the second in the first place. As an aside, the section that goes like: > exerting over the system when the system has been fine-tuned to make it easier for the human to do less, and for the operator to accept the machine’s judgment. Is just incredibly bad faith nonsense. Or extremely ignorant. It's hard to tell which.
- bsenftner 4y ago
  It is both incredibly bad faith nonsense and extremely ignorant. The ethics debates around AI are filled with military jargon dropping morons. It is extremely hard to find a rational anyone not pushing some corporate-wealth-scheme or crank-conspiracy nonsense. I work in AI, and the non-technically specific conversations, such as ethics, are extremely shallow.
  - sicp-enjoyer 4y ago
    It's a form of marketing that appeals to people in tech.
  - hutzlibu 4y ago
    The main problem with non technical people talking about AI, is that they mainly have no understanding about AI, but everyone has still a strong opinion (or thinks he has to have one)
ben_w 4y ago
My father had a humans-in-the-loop story about early ICBM warning systems. Turns out the moon doesn’t have an IFF transponder, so when it came over the horizon…
- ovao 4y ago
  I think we all probably get the gist, but would love to hear more details.
  - geoduck14 4y ago
    I'll continue: The moon came over the horizon... then we BLEW IT UP, which is why we only have 1 moon today
  - remram 4y ago
    It's a warning system so it probably sounded a warning.
    - klyrs 4y ago
      Moral: fly your icbm on a moon-eclipsing trajectory.
- IshKebab 4y ago
  Nothing happened because the radar didn't have enough range to reach the freaking moon?
nonrandomstring 4y ago
In all considerations of these arguments Incident RQ-170 2011 [1] should remain foremost. It is to military systems engineering what the Tacoma Bridge [2] is to civil engineering and the Therac-25 [3] is to software engineering. Axiomatically: Any system, no matter how well guarded, can and eventually will be taken by an enemy and used against you. That's why we; Don't take a knife to fight if you're not trained to use it. Don't build offensive cyberweapons your civilian population are not equipt to defend against (looking at you US Cyber Command). [1] https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incid... [2] https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940) https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940) [3] https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- kleinsch 4y ago
  I don’t understand the point you’re trying to make, other than that these are famous events in engineering. What does a bridge collapsing bc of a structural defect or a software bug causing higher doses or radiation have to do with systems being taken by enemies and used against you?
  - jayd16 4y ago
    They use the Therac-25 story to scare young CS students into considering the broader implications of their work, ie people might die. The others might also fall under 'popular professional cautionary tales.'
    - hotpotamus 4y ago
      Interestingly in talking to a developer who worked in defense, they don’t say if you get your code wrong, people might die - they say the wrong people might die.
- kidme5 4y ago
  My greatest concern today is the militarization of space going on right now[0] (Space Development Agency). The system engineer behind it Michael D. Griffin[1] seems to think the US can "win" all future wars by creating a space-based missile defense. Decisions are automatically made in milliseconds, with no room for human input. SpaceX's participation[2] is particularly concerning (they just built and launched four "secret" satellites for it last week) and employees are now complaining about it on Glassdoor[3] Elon talks a lot about AI / Skynet, and it appears there is a serious attempt to construct it. [0] https://crsreports.congress.gov/product/pdf/IF/IF11623 https://crsreports.congress.gov/product/pdf/IF/IF11623 [1] https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career [2] https://mobile.twitter.com/planet4589/status/1538960051414646796 https://mobile.twitter.com/planet4589/status/153896005141464... [3] https://www.glassdoor.com/Reviews/Employee-Review-SpaceX-RVW65728516.htm https://www.glassdoor.com/Reviews/Employee-Review-SpaceX-RVW...
LocalH 4y ago
Real life WOPR incoming. Shall we play a game?
- linksnapzz 4y ago
  Years ago, a late friend and I had a little ritual. One of us would be hunched over a terminal in the operations booth of our workplace's datacenter, trying to figure out why $critical_service was not coming back up after patching. The other would see the alert in Intermapper, come downstairs, and amongst the alarms and ringing phones, solemnly intone: "Mr. McKittrick, after very careful consideration, sir, I have come to the conclusion that your new defense system...sucks." This always managed to lighten the mood.
alanbernstein 4y ago
I once read a quote from a general or something, making the point that the decisions made by these systems, even when a human is in the loop, are very much algorithmic or formulaic, and therefore taking the human out of the loop, making the system entirely formulaic, should be fine. Obviously that can't apply to all situations in the same way, but I thought it was an interesting viewpoint at least. I can't remember the article or the speaker though.
- bell-cot 4y ago
  OTOH - given all the sorts of complete-facepalm mistakes that image recognition systems have been seen to make, and the incentives for an actual wartime enemy to "encourage" your "AI"-run weapon systems to screw up, and the consequences if they are successful...
  - alanbernstein 4y ago
    Yeah, absolutely. The quote may have been from before the modern era of DNN-based image recognition, when I suppose that part of the system was always assumed to be a human's responsibility.
andi999 4y ago
You need a human in the loop, otherwise there would be no legal recourse on war crimes.
- bell-cot 4y ago
  My reaction: It should be the same consequences for war crimes as if you'd fired a barrage of WWII-era VT-fused artillery shells at a refugee camp, or laid an old-fashioned minefield in the playground of an orphanage. Some "smart" gadget you built & deployed did something horrible. Your ass. Or, if you protest that "the AI did it!" - then you have to march across a field guarded by a bunch of "smart" weapon systems. The justice system is not responsible if any of those AI's malfunction and kill you.
gentleman11 4y ago
There was an incident once where a primitive ai system misidentified some birds as a nuclear launch. A submarine captain was supposed to launch missiles vs the United States, which would have ended the entire world. He decided not to. New ai is a lot better than our old ai but we’re talking about the fate of human existence and new ai is still prone to catastrophic errors the few times it does fail, Eg, autopilot mistaking the side of a truck for the sky in a Tesla. What if the training set just didn’t include certain cases?
- stoltzmann 4y ago
  >There was an incident once where a primitive ai system misidentified some birds as a nuclear launch. A submarine captain was supposed to launch missiles vs the United States, which would have ended the entire world. He decided not to. I think you mixed up a couple of stories. There was an incident in 1983 where the Soviet early warning system picked up a reflection off a satellite and high-altitude clouds and interpreted it as a launch of 5 US ICBMs. The "proper" reaction to detecting a US launch would be to counterstrike immediately, but the operator Stanislav Petrov has recognized it as an error - he correctly guessed that if push came to shove, Americans would launch a couple more missiles than just 5. The submarine incident was in 1962 during the Cuban missile crisis. As it was ending, a US destroyer dropped a couple of small signalling depth charges above a Soviet submarine. The submarine did not have ballistic missiles, but it had torpedoes with nuclear warheads. Those weapons required usually permission from both the captain and the political officer - and both agreed that World War 3 has just started and they should engage. Thankfully, the commander of the submarine flotilla Vasily Arkhipov was aboard and in this case also his permission was needed. He instead opted to surface the ship and contact Moscow for orders. There was no AI present during any of this. I don't believe birds possess the required characteristics to be identified as an ICBM launch.
FunnyBadger 4y ago
There's nothing inherently wrong with human-in-the-loop. In many, many, many cases, it's is lack of this that causes death, destruction and grievous mistakes while giving humans a nominal "plausible deniability" of making any mistake at all. - "it's the software that decides; I can't change that.". The biggest reason why human-in-the-loop is necessary is that humans scale better in unique complexity situations than fixed-design machines ever an be. Often you actually want to slow down a system specifically to assure a human can judge. Fully automating assures that the small or large errors will grow far too fast - we see this with social media - that's literally the problem with Twitter et al. - they operate too fast to be safe or effective. It's better to slow things down if it results in correct, moral or fair operation. There is nothing magical or essential or better about maximum speed blindly. The additional big reason is that machines are incapable of morality nor of facing punishment - you need a human to "have skin in the game" to create any force to assure correct behavior or operation. Being afraid of jail or financial ruin is essential to this. A machine or corporation (NOT a person) never can and never will. One data point is drones: currently, like most precision weapons, the precision is nearly 100% - that is, if you point the weapon at a target, the target will be destroyed nearly 100% of the time. The problem then is how you target the weapon - that's where all the "errors" are incurred. This is how you get Afghan wedding parties killed off. It's the intelligence errors and rules of engagement that are actually what kills. Another data point that should be horrifying: the US military is currently working on neuromorphic AI chips. The specific intention is to make early warning satellites capable of identifying targets and triggering a nuclear "response/attack" without a human in the loop. Literal Sky Net. This is intelligence and targeting being automated and humans being pulled out of the loop.
- kragen 4y ago
  You say, "There's nothing inherently wrong with human-in-the-loop." But humans have reaction times around 500 milliseconds; in the best cases, around 200 milliseconds. I routinely write software with worst-case reaction times around 0.001 milliseconds, or 1 millisecond for more complex stimuli. If a force of drones with humans in the loop fights a force of drones without humans in the loop, absent colossal blunders, the latter will win, and it won't even be close. Imagine you get into a fistfight with a guy. He's bigger than you, he's got a loaded 9-millimeter pistol, and he's a good marksman. But his reaction time is 200 times yours --- say, about 50 seconds, if you're a regular human. That is, all his reactions are reacting to what you did 50 seconds ago, at best. Who do you think is going to win? Unless he's inside a tank and you're outside, you will. Even in that case, I think you've got a fighting chance. You argue that it's good for armies to "have skin in the game" in the sense of being vulnerable to jail or financial ruin. And, from a moral point of view, it probably is good: armies that are exposed to retaliation will generally be less willing to do things that provoke retaliation, many of which are immoral. However, the things that provoke retaliation are also the things that win wars. Making your army more vulnerable may reduce war crimes, but it is not a recipe for victory. The logic of warfare is inimical to our survival as a species, which is why our best and brightest have been calling for a pacifist world order since even before World War II. Nuclear warfare were a nice early warning, but they're children's toys compared to precision-guided munitions backed up by pervasive surveillance.
dirtyid 4y ago
Mitigate kill chain distruption in age of electronic warfare.
yboris 4y ago
PSA: Stop Killer Robots https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/

75 points by Symmetry 4y ago

The natural course of war tends toward a future where polities own, but do not operate, fully autonomous weapons. There will come a day when even the targets, be they military or civilian, are selected and attacked solely by machines. The human element is more creative, though bound by a sense of morality. Whereas the machine is faster, bound only by the thinnest of rule sets. If we're lucky, most of the combat occurs virtually rather than mechanically.

jsmith45 4y ago

Keeping a human in the loop is more about beeing willing to trust the tech than about safety. Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched if there were no enemies so obviously the one and only target it sees must be an enemy, and turns around and kills you. After that happens just a few times, the other soldiers would revolt and refuse to use the weapon system anymore. That is the sort of thing that motivates having humans in the loop, even if on average there would be greater safety without humans in the loop.

xyzzy123 4y ago
"This side towards enemy'. Doesn't have to be "safe" / "not dangerous" (a lot of weapons systems are very dangerous in different ways) but it should be predictable, using some reasonably simple mental model. "Refusing to deploy" might not be a sensible option if your enemy is willing to deploy. It seems safe to predict that human in the loop systems will be quickly overwhelmed by fully autonomous systems. The tradeoff may end up being having to balance the risk of getting killed by your own systems versus being killed by your enemy's systems, because yours were too "tame".
- Frost1x 4y ago
  >"Refusing to deploy" might not be a sensible option if your enemy is willing to deploy. In the AI ethics arena, this is really one of the best arguments I've heard for the military to pursue and have automated systems in place. US leadership may agree and see all the issues with using killer robots but other countries are developing the same technology and may have less ethical concern and considerations then you do. They may be willing to let these systems fail and kill innocent bystanders or even their own soldiers. They may even look at these and consider it a necessary loss to move forward and succeed in their goals. As such, if you're not willing to make such compromises as well, unless your approach ultimately achieves the goal or you have other techniques to mitigate these corner cutting strategies, you too must cave. If China decides to engage in conflict and releases killer robots and we don't have killer robots, can we contend? If we can't, we might have to engage in the race to the bottom that is competition unless we have an alternative mitigation strategy that doesn't require us to compromise our ethics. I imagine this is the dilemma scientists faced during WWII when developing nuclear weapons. Part of the argument that lead to nuclear weapons was that Germany was already working on the process. This resulted in other countries engaging and ultimately building successful weapons where Germany failed. Ultimately, we now live in a world where lots of people have nuclear weapons... and overall it's been the proverbial Mexican standoff with nuclear deterrence strategies. Is there an AI deterrence strategy? Nuclear worked because it lead to the case of a global collapse in terms of what nuclear war would bring. Killer robots that aren't sentient don't seem to have quite the same threat and seem like militaries could utilize them as they fall in an ethical grey area globally and have far less risk of ending humanity on a global scale unless we start talking about sentient terminator killer robot level uprising which is not even remotely where AI is currently or in the foreseeable future.
  - deleted 4y ago
    [deleted]
  - andi999 4y ago
    I think currently no other country is even close to the level of autonomous weapon systems the US is developing. An arms race where you are miles ahead cannot be used as a reason why you have tonrun faster.
j7ake 4y ago
I think you’ve hit on the concept when humans in the loop are needed. For many black box systems, the average performance may be comparable or better than humans, but when errors occur, they should fail predictably. If systems fail unpredictably (ie unbounded in the types of errors it makes), then they can never be deployed in the field without humans in the loop. This is why self driving cars can, at the same time, be on average safer but still not be accepted by public.
- Gordonjcp 4y ago
  > This is why self driving cars can, at the same time, be on average safer but still not be accepted by public. They're not safer, though.
venkat223 4y ago
Being an expert boss in AI and expert controller in loop in situ are two different perspectives.AI experts will be useless in situ.
walrus01 4y ago
> Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched or more realistically, your autonomous version of an aerovironment switchblade launches, has image recognition/onboard trained neural network that can identify a russian spec T-72 tank or BMP or similar, doesn't see any, loiters for a while and then self destructs.

darkerside 4y ago

This article is thoughtful and naive at the same time. Deploying an autonomous weapon is just like deploying any other weapon. Once a gun is fired, the human is out of the loop. Autonomous weapons will end up the same way. People will get to exercise judgement early and late, but AI will make decisions during execution, just like a bullet follows the flight path. The reason we need humans is for accountability. You can't court marshal a machine. Firing the operator doesn't fix anything or prevent it from happening again (next operator will be just as fallible), it just satisfies some primal part of our brains that needs to blame someone when something bad happens.

taylorius 4y ago
The difference lies in the predictability, I would say. A bullet flies according to well understood equations of motion, whereas an A.I - no-one really knows how it might be fooled, or what mistakes it could make.
- darkerside 4y ago
  I guess the better analogue is directing troops.
- jameshart 4y ago
  Cutting the wires and letting a torpedo run in active homing mode and find and select its own target has been part of submarine warfare practice for decades. We tolerate it because there aren’t generally a lot of civilians or even friendly fire risks in submarine engagement environments.
jayd16 4y ago
That's one way to think about it. However, don't we usually just outright ban weapons with longer term lethality? A bullet is lethal for seconds while mines are lethal for years. If autonomous drones are lethal for days, weeks or even months, should we consider them a war crime?
- darkerside 4y ago

sgt101 4y ago

>When that happens, we identify why the interaction is problematic, why the human is not trusting the machine, or why the way that the data is presented is being misunderstood — and then we change those things. Such changes should make us question to what extent human oversight of algorithms is truly meaningful. Nonsense; this is like saying "we changed the wheel on the bike so that the rider didn't fall off it so often which makes us question if they can ride the bike". If the interface is being changed to better deceive a user then that of course is problematic, but if it is being changed to allow the user to make accurate decisions then that is the function of the system. >But now imagine a system similar to HARPY, a “fire-and-forget” system that targets enemy combatants in an area or enemy ships (to use an example that minimizes the collateral risk). Proponents of meaningful human control do not believe it should ever be left up to a machine to target humans. Strawman - that is not what "Proponents of meaningful human control" believe, I am sure you can find some people who say that, but then they will also say that they don't believe that a bullet should be fired or a bomb dropped. Machines target humans all the time in the sense that unintended collateral damage occurs whenever bombs are dropped. The humans doing it must accept responsibility for this; that is the same for more sophisticated processes - the humans remain on the hook for unleashing the device. If the device decides to slaughter a primary school class that's on you in the same way as it is if you target a school yourself. >I think there are good reasons to question this. Specifically, I am not persuaded that there is a significant moral difference between an operator identifying a single human target based on some data and a human operator drawing a box and defining targets within that box based on equally good data. Which shows that you, dear writer, should never be allowed to decide a god damn thing. There is a chasm of uncertainty between the two scenarios, and a chasm of indirection. >whether in cases when the system works better or as well without a human operator, we have moral reasons to place a human in the loop. Because we live in an open world and the judgement of performance is post hoc, you can't know that it will work until it's done its thing. The judgement has to be when the act is commissioned because the consequences of the act cannot be reversed.

verisimi 4y ago

I can't get past the first paragraph! > At the recent conference on the ethics of AI-enabled weapons systems at the U.S. Naval Academy, well over half the talks discussed meaningful human control of AI to some extent. But, but AI is sentient right?! What business is it of ours to try to exercise control over AI, with its many and various superior attributes? lol. > If you work among the AI ethics community, and especially among those working on AI ethics and governance for the military, you are hard-pressed to find an article or enter a room without stumbling on someone literally or metaphorically slamming their fist on the table while exalting the importance of human control over AI and especially AI-enabled weapons. AI ethics is only there to justify the unjustifiable. They do not have my ethics that's for sure, nor any normal person's. Its a bad joke that allows a corporation to say - 'but it has been subject to an ethical review!' - and the political class can allow themselves to be bamboozled and provide their rubber stamp. > the ethics of AI-enabled weapons ... the ethical implications fall on those coding the dystopia. War is immoral, pretending that wars are real is immoral, handing over decision of life and death to a machine is immoral - regardless of how efficient the machine may be. So, any of those people involved in that circle are immoral. Soldiers may think they are working to defend their 'country' - but 'country' is a fiction and 'defence' is an inversion. People act this way for money.. at least that reason makes sense - but there is nothing moral about it. There are no ethics to be found there.

astrange 4y ago
> Soldiers may think they are working to defend their 'country' - but 'country' is a fiction and 'defence' is an inversion. Meaning is collaborative. It doesn’t matter if you’re a nihilist and want to declare your country doesn’t exist; the guy invading you makes it real. You can be a quisling, but defecting to the invader seems to admit they exist, at least.
- verisimi 4y ago
  Who's invading who now? What are the soldiers "defending"? Do you think that 'Department of Defense' is a misnomer? Howabout 'military aid'? Are these the sorts of linguistic meanings we are meant to collaborate on? I'm no collaborator, you know..

theelous3 4y ago

The article presents two sides - one end advocates humans taking an active role in the loop, like confirming a drone's target classification and authorising it to act lethally. The other side is where the human's role is to shape the operating envelope to what they think will have parameters appropriate for the ethics of the situation. The examples of the systems seems completely out of whack though. The active participant advocates are clearly taking about things like air to ground hunter killer missions on ambiguous targets. For example, a civilian vehicle converted to technical vs a plumber hauling some piping. A farmer with a weed killer in a ditch by the road vs an insurgent constructing an ambush. These are clearly beyond the realm of what the other side is talking about - easily identifying shit like russian airforce craft with no friendly transponders and armed active radar payloads under its wings. I don't understand the conversation that is trying to take place. Clearly they are too different to even consider. Is the conversation supposed to be hypothetical enough that we can pretend we can act in the first case, as we can in the second? Then the conversation seems pointless. Of course you can, if you accept the premise of the second in the first place. As an aside, the section that goes like: > exerting over the system when the system has been fine-tuned to make it easier for the human to do less, and for the operator to accept the machine’s judgment. Is just incredibly bad faith nonsense. Or extremely ignorant. It's hard to tell which.

bsenftner 4y ago
It is both incredibly bad faith nonsense and extremely ignorant. The ethics debates around AI are filled with military jargon dropping morons. It is extremely hard to find a rational anyone not pushing some corporate-wealth-scheme or crank-conspiracy nonsense. I work in AI, and the non-technically specific conversations, such as ethics, are extremely shallow.
- sicp-enjoyer 4y ago
  It's a form of marketing that appeals to people in tech.
- hutzlibu 4y ago
  The main problem with non technical people talking about AI, is that they mainly have no understanding about AI, but everyone has still a strong opinion (or thinks he has to have one)

ben_w 4y ago

My father had a humans-in-the-loop story about early ICBM warning systems. Turns out the moon doesn’t have an IFF transponder, so when it came over the horizon…

ovao 4y ago
I think we all probably get the gist, but would love to hear more details.
- geoduck14 4y ago
  I'll continue: The moon came over the horizon... then we BLEW IT UP, which is why we only have 1 moon today
- remram 4y ago
  It's a warning system so it probably sounded a warning.
  - klyrs 4y ago
    Moral: fly your icbm on a moon-eclipsing trajectory.
IshKebab 4y ago
Nothing happened because the radar didn't have enough range to reach the freaking moon?

nonrandomstring 4y ago

In all considerations of these arguments Incident RQ-170 2011 [1] should remain foremost. It is to military systems engineering what the Tacoma Bridge [2] is to civil engineering and the Therac-25 [3] is to software engineering. Axiomatically: Any system, no matter how well guarded, can and eventually will be taken by an enemy and used against you. That's why we; Don't take a knife to fight if you're not trained to use it. Don't build offensive cyberweapons your civilian population are not equipt to defend against (looking at you US Cyber Command). [1] https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incid... [2] https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940) https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940) [3] https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25

kleinsch 4y ago
I don’t understand the point you’re trying to make, other than that these are famous events in engineering. What does a bridge collapsing bc of a structural defect or a software bug causing higher doses or radiation have to do with systems being taken by enemies and used against you?
- jayd16 4y ago
  They use the Therac-25 story to scare young CS students into considering the broader implications of their work, ie people might die. The others might also fall under 'popular professional cautionary tales.'
  - hotpotamus 4y ago
    Interestingly in talking to a developer who worked in defense, they don’t say if you get your code wrong, people might die - they say the wrong people might die.
kidme5 4y ago
My greatest concern today is the militarization of space going on right now[0] (Space Development Agency). The system engineer behind it Michael D. Griffin[1] seems to think the US can "win" all future wars by creating a space-based missile defense. Decisions are automatically made in milliseconds, with no room for human input. SpaceX's participation[2] is particularly concerning (they just built and launched four "secret" satellites for it last week) and employees are now complaining about it on Glassdoor[3] Elon talks a lot about AI / Skynet, and it appears there is a serious attempt to construct it. [0] https://crsreports.congress.gov/product/pdf/IF/IF11623 https://crsreports.congress.gov/product/pdf/IF/IF11623 [1] https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career [2] https://mobile.twitter.com/planet4589/status/1538960051414646796 https://mobile.twitter.com/planet4589/status/153896005141464... [3] https://www.glassdoor.com/Reviews/Employee-Review-SpaceX-RVW65728516.htm https://www.glassdoor.com/Reviews/Employee-Review-SpaceX-RVW...

LocalH 4y ago

Real life WOPR incoming. Shall we play a game?

linksnapzz 4y ago
Years ago, a late friend and I had a little ritual. One of us would be hunched over a terminal in the operations booth of our workplace's datacenter, trying to figure out why $critical_service was not coming back up after patching. The other would see the alert in Intermapper, come downstairs, and amongst the alarms and ringing phones, solemnly intone: "Mr. McKittrick, after very careful consideration, sir, I have come to the conclusion that your new defense system...sucks." This always managed to lighten the mood.

alanbernstein 4y ago

I once read a quote from a general or something, making the point that the decisions made by these systems, even when a human is in the loop, are very much algorithmic or formulaic, and therefore taking the human out of the loop, making the system entirely formulaic, should be fine. Obviously that can't apply to all situations in the same way, but I thought it was an interesting viewpoint at least. I can't remember the article or the speaker though.

bell-cot 4y ago
OTOH - given all the sorts of complete-facepalm mistakes that image recognition systems have been seen to make, and the incentives for an actual wartime enemy to "encourage" your "AI"-run weapon systems to screw up, and the consequences if they are successful...
- alanbernstein 4y ago
  Yeah, absolutely. The quote may have been from before the modern era of DNN-based image recognition, when I suppose that part of the system was always assumed to be a human's responsibility.

andi999 4y ago

You need a human in the loop, otherwise there would be no legal recourse on war crimes.

bell-cot 4y ago
My reaction: It should be the same consequences for war crimes as if you'd fired a barrage of WWII-era VT-fused artillery shells at a refugee camp, or laid an old-fashioned minefield in the playground of an orphanage. Some "smart" gadget you built & deployed did something horrible. Your ass. Or, if you protest that "the AI did it!" - then you have to march across a field guarded by a bunch of "smart" weapon systems. The justice system is not responsible if any of those AI's malfunction and kill you.

gentleman11 4y ago

There was an incident once where a primitive ai system misidentified some birds as a nuclear launch. A submarine captain was supposed to launch missiles vs the United States, which would have ended the entire world. He decided not to. New ai is a lot better than our old ai but we’re talking about the fate of human existence and new ai is still prone to catastrophic errors the few times it does fail, Eg, autopilot mistaking the side of a truck for the sky in a Tesla. What if the training set just didn’t include certain cases?

stoltzmann 4y ago
>There was an incident once where a primitive ai system misidentified some birds as a nuclear launch. A submarine captain was supposed to launch missiles vs the United States, which would have ended the entire world. He decided not to. I think you mixed up a couple of stories. There was an incident in 1983 where the Soviet early warning system picked up a reflection off a satellite and high-altitude clouds and interpreted it as a launch of 5 US ICBMs. The "proper" reaction to detecting a US launch would be to counterstrike immediately, but the operator Stanislav Petrov has recognized it as an error - he correctly guessed that if push came to shove, Americans would launch a couple more missiles than just 5. The submarine incident was in 1962 during the Cuban missile crisis. As it was ending, a US destroyer dropped a couple of small signalling depth charges above a Soviet submarine. The submarine did not have ballistic missiles, but it had torpedoes with nuclear warheads. Those weapons required usually permission from both the captain and the political officer - and both agreed that World War 3 has just started and they should engage. Thankfully, the commander of the submarine flotilla Vasily Arkhipov was aboard and in this case also his permission was needed. He instead opted to surface the ship and contact Moscow for orders. There was no AI present during any of this. I don't believe birds possess the required characteristics to be identified as an ICBM launch.

FunnyBadger 4y ago

There's nothing inherently wrong with human-in-the-loop. In many, many, many cases, it's is lack of this that causes death, destruction and grievous mistakes while giving humans a nominal "plausible deniability" of making any mistake at all. - "it's the software that decides; I can't change that.". The biggest reason why human-in-the-loop is necessary is that humans scale better in unique complexity situations than fixed-design machines ever an be. Often you actually want to slow down a system specifically to assure a human can judge. Fully automating assures that the small or large errors will grow far too fast - we see this with social media - that's literally the problem with Twitter et al. - they operate too fast to be safe or effective. It's better to slow things down if it results in correct, moral or fair operation. There is nothing magical or essential or better about maximum speed blindly. The additional big reason is that machines are incapable of morality nor of facing punishment - you need a human to "have skin in the game" to create any force to assure correct behavior or operation. Being afraid of jail or financial ruin is essential to this. A machine or corporation (NOT a person) never can and never will. One data point is drones: currently, like most precision weapons, the precision is nearly 100% - that is, if you point the weapon at a target, the target will be destroyed nearly 100% of the time. The problem then is how you target the weapon - that's where all the "errors" are incurred. This is how you get Afghan wedding parties killed off. It's the intelligence errors and rules of engagement that are actually what kills. Another data point that should be horrifying: the US military is currently working on neuromorphic AI chips. The specific intention is to make early warning satellites capable of identifying targets and triggering a nuclear "response/attack" without a human in the loop. Literal Sky Net. This is intelligence and targeting being automated and humans being pulled out of the loop.

kragen 4y ago
You say, "There's nothing inherently wrong with human-in-the-loop." But humans have reaction times around 500 milliseconds; in the best cases, around 200 milliseconds. I routinely write software with worst-case reaction times around 0.001 milliseconds, or 1 millisecond for more complex stimuli. If a force of drones with humans in the loop fights a force of drones without humans in the loop, absent colossal blunders, the latter will win, and it won't even be close. Imagine you get into a fistfight with a guy. He's bigger than you, he's got a loaded 9-millimeter pistol, and he's a good marksman. But his reaction time is 200 times yours --- say, about 50 seconds, if you're a regular human. That is, all his reactions are reacting to what you did 50 seconds ago, at best. Who do you think is going to win? Unless he's inside a tank and you're outside, you will. Even in that case, I think you've got a fighting chance. You argue that it's good for armies to "have skin in the game" in the sense of being vulnerable to jail or financial ruin. And, from a moral point of view, it probably is good: armies that are exposed to retaliation will generally be less willing to do things that provoke retaliation, many of which are immoral. However, the things that provoke retaliation are also the things that win wars. Making your army more vulnerable may reduce war crimes, but it is not a recipe for victory. The logic of warfare is inimical to our survival as a species, which is why our best and brightest have been calling for a pacifist world order since even before World War II. Nuclear warfare were a nice early warning, but they're children's toys compared to precision-guided munitions backed up by pervasive surveillance.

dirtyid 4y ago

Mitigate kill chain distruption in age of electronic warfare.

yboris 4y ago

PSA: Stop Killer Robots https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/