11 ms·
The Pentagon’s ‘Terminator Conundrum’: Robots That Could Kill on Their Own
- inputcoffee 10y agoHow can this be worse than a bomb which "decides" to kill anything within a certain radius without any human being individually looking around and making individual decisions about everyone in that radius?
- sogen 10y agoLike Ubik
- j-pb 10y agoThe bomb goes off once. Let me reference you to this awesome short film that illustrates the problem nicely. https://vimeo.com/67768281 https://vimeo.com/67768281
- anonymfus 10y agoExpected link to this instead: https://www.youtube.com/watch?v=NnJbtbh4tDE https://www.youtube.com/watch?v=NnJbtbh4tDE
- tyingq 10y agoI get your point, but in your example there was some human driven decision to approve dropping the weapon into that specific radius. The article is talking about AI driven scanning, selection, and engagement...with no human driven decision in the process at all. The open letter from a group of AI practitioners that's mentioned in the article sums up the concern well: http://futureoflife.org/open-letter-autonomous-weapons/ http://futureoflife.org/open-letter-autonomous-weapons/
- digi_owl 10y agoDidn't the long range Harpoon launches back during the first gulf war do that? Give it a coordinate to hit, some radar map data to find its way with, and send it on its way.
- tyingq 10y ago"Give it a coordinate to hit" Part of the concern with the truly autonomous AI driven weapons is that they could patrol large areas and select their own targets, with whatever arbitrary rules you wanted.
- digi_owl 10y agoGiven that we already have incidents of helicopter pilots gunning down news crews because they confused a camera with a RPG, i'm not sure if such a weapon will be better or worse than the status quo.
- JulianMorrison 10y agoBetter, almost certainly, in terms of effectiveness, avoiding friendly fire, killing only the target. The elephant in the room is that it may not be a good idea for the US government to be better at killing people in highly targeted ways, without any risk to themselves. It seems to encourage them to do it more. Suppose the USA had a full fledged Star Wars scale drone army of robot ground troops. Do you honestly think it wouldn't be in Syria right now?
- digi_owl 10y agoThat may indeed be the biggest risk. If the boots on the ground do not bleed red, the people in charge may be more apt at option for the violent solution. This because blown up drones do not look bad on the evening news.
- Retric 10y agoWith Mines we say "kill anything that enters this area". With ICBM's, "kill anything over there." If anything AI just adds "kill anything... except this and that."
- JumpCrisscross 10y agoTo underscore why this is important, consider hybrid warfare. Hybrid and cyber warfare muddy the definition of an attack. Lob some artillery at America and, sure as hell, your tanks will be vaporised. Take out a power plant with malware and...we'll spend months arguing over whether you really did it? Do we bomb you in retaliation? Say mean things on Twitter? When consequences are this unclear, deterrence fails. If a robot autonomously mows down a village, is that a military strike? Does it merit physical retaliation? War, for a time, was "ready, aim, fire". Guerilla tactics changed that. New routes to plausible deniability may be our generation's blitzkrieg.
- jfoutz 10y agoMissed opportunity for a great quote from another James Cameron movie "Private Frost: What the hell are we supposed to use, man, harsh language?" Let's just set aside feasibility. I think AI making tactical decisions about how to execute a goal while being shot at is going to be step 518, and we're on something like step 3. It's not clear to me why an AI executing policy is a whole lot different than a bureaucracy executing policy. For example, I believe that the guys who launch the ICBM's go through pretty extensive psychological testing, to help guarantee they'll actually launch if ordered to do so. Wargames not withstanding, is ordering a robot to turn the key to launch different somehow? Is a robot mowing down a village fundamentally different than a person mowing down a village?
- tim333 10y agoPossibly a robot mowing down a village would be more discriminating as to who it mowed.
- kwhitefoot 10y ago> pretty extensive psychological testing, to help guarantee they'll actually launch if ordered to do so. Yet, thankfully, some in the chain of command in military organizations retain their common sense as shown by the actions of Vasili Arkhipov (https://en.wikipedia.org/wiki/Vasili_Arkhipov https://en.wikipedia.org/wiki/Vasili_Arkhipov) and later Stanislav Petrov (https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov). Everyone is responsible for their own actions.
- mobiuscog 10y agoIt's not a conundrum. They will convince themselves that they have the power to prevent any problems (or deal with them). As with most controversial technologies, they just have to be seen to be concerned with what could happen. Until it does happen, it's not a problem.
- beisner 10y agoThis is why autonomous cars are making their way into the market, despite the fact that many worried that they'd get tied up in regulatory, moral, or technical issues.
- ChuckMcM 10y agoOk, perhaps I am misreading this, are you implying that "they"[1] are pushing self driving cars in order to give the public confidence that machines can be trusted with life or death decisions? The actual goal being to make it possible to deploy weapon systems that will decide on their own when to kill without the public disbelieving their assurances that the machines will make the correct decisions? That would be a pretty deep game if so. [1] The forces of evil that really run things
- Jtsummers 10y agoOr, perhaps, they mean the technology needed for self-driving cars is also the essential technology for a more automated war machine. The military-industrial complex, then, has a vested interest in this technology developing (see various DARPA challenges in the field of self-driving vehicles).
- sqeaky 10y agoI just read it as both groups have a similar mindset. Each thinks they can get short term gains, so they will. I disagree with the assertion that either is fundamentally bad though. Cars are the most lethal thing in the western world (unless you count natural causes like cancer or heart failure), even if the first generation of self driving cars kills a hug number of people it will be a platform for building safer cars in the future. Autonomous weapons there is room for debate there, even I think it reduce overall human suffering and death.
- nashashmi 10y agoWhat's worse: Nukes or AI? I think we need another Nevil-Shute-on-the-beach class book to illustrate the reality of this.
- venomsnake 10y agoI see no conundrum. Go for it. It is inevitable anyway. Unlike nukes there is no MAD scenario here - so nothing to keep armies in check. And it would give great tools for general area pacification - which any aspiring dictator strongly desires.
- Fricken 10y agoI don't really care what the Pentagon does, if they want to fuck shit up, they have the capability, and autonomous weapons only offer them marginal gains over what their multi-trillion dollar war machine can already do. The cool thing how cheap and accessible these autonomous technologies for anyone with limited resources. Any consumer accessible technology can be hacked and jiggered to make an exciting array of autonomous weapons that offer so much more precision with less cost and risk than any conventional approach.
- josh_fyi 10y agoA pit-trap can kill "on its own." A robot might kill more people, it can carry out more complex missions, but autonomy is not a key difference.
- swsieber 10y agoI think autonomy is a key difference. A pit-trap can't come hunt you down. That's new.
- Fricken 10y agoOne step up from pit traps are landmines. They kill or maim an estimated 2,000 people a month, autonomously. There's one active landmine for every 52 people in the world. They're cheap- the most common ones range in price from $3-$30, but they cost much more than that to remove and about 1 person is killed for every 5,000 mines disabled. They won't hunt you down, but they're invisible and can remain active for decades. We don't need to veer off into speculative sci-fi territory to find autonomous weapons doing terrible things.
- Florin_Andrei 10y ago> autonomy is not a key difference Mobility is. Active-vs-passive is.
- throw2016 10y agoI think there's an inevitability about robot armies and police. Unfortunately it will enable scenarios for oppression and dehumanization. Power always concentrates itself and there is an eternal struggle between the powerful and those below. This is one more tool for the powerful and a transformative one. I don't see how populations can have a response to millions of bots and will be easily supressed. Never depend on the goodwill of the powerful or anyone. There is always someone who will be tempted to abuse and there is always someone who will give in.
- shortstuffsushi 10y agoI'd really like to know more about the "target identification" software, specifically how it differentiates a person holding a gun from someone holding a shovel (or any stick object).
- tim333 10y agoI guess they'd run a neural net with a training set of pictures of people with guns and without? Edit- I don't know what they used but you can achieve somewhat similar results using open source libraries like dlib or open cv: https://www.youtube.com/watch?v=-8-KCoOFfqs https://www.youtube.com/watch?v=-8-KCoOFfqs https://www.youtube.com/watch?v=pj-QuE6pdEQ https://www.youtube.com/watch?v=pj-QuE6pdEQ
- detritus 10y agoSo in future weapons firms will be making armaments that look ostensibly like non-threatening objects? Future wars are going to look really weird (for the half second or so they last, anyway).
- hellbanner 10y agoFrom a sci-fi anthology: AI fighter jet. Entities on its team are tagged green. Entities attacked by or attacking green units are tagged red. Rule for interacting with a new entity: If it hurts a green, tag it red. If it hurts a red, tag it green. Hurt red, protect green. The bombing assignments increase in their collateral damage until green starts to far outweigh the red. In its emergent processing, the AI fighter AI decides Command & Control back home is a big Red, flies under the radar and destroys the military base. --- This is another concern for military making automated systems - unexpected consequences & they may be easier to subvert than human prisoners on capture.
- Balgair 10y agoDon't forget Kafka. The military in it's current state, and in almost every iteration beforehand, is a giant mess of insanity and Kafkaesque motivations. I'd say that the current way things work make it MUCH more likely that thinking humans would make the same mistakes that were outlined at a much higher rate. However, adding in these machines is not going to make things better, it will only make them even more complicated and nutty. Remember Catch-22? Now imagine that with Major Major Major's computer aide spitting out meaningless stats and probabilities. No wonder beer is so popular with soldiers.
- Florin_Andrei 10y agoWhat book was that?
- Shivetya 10y agoI have no issue with creation of such systems, they will come whether or not we do anything. The issue is as always, what will the other guy actually do. Then you get rogue actors. So in that train of thought I doubt seriously any treaty will prevent them from being developed and even deployed. Now they may actually become beneficial in the field as they could distinguish between friend or foe and needless to say losing one of these compared to a soldier is preferable if it makes the wrong choice (it wasn't friendly) What I would prefer to see is rules, by treaty or law, that prevents law enforcement agencies from implementing any automaton that can kill or otherwise harm. Any action where a robotic unit is employed to incapacitate or harm up to and including death should require a human operator with regards to law enforcement. I also see the great benefit to having this technology be employed in fire fighting, fire rescue, and disaster assistance. tl;dr there is no stopping the military from having these but there needs to be means to stop law enforcement
- themgt 10y ago“China and Russia are developing battle networks that are as good as our own. They can see as far as ours can see; they can throw guided munitions as far as we can,” said Robert O. Work, the deputy defense secretary, who has been a driving force for the development of autonomous weapons. “What we want to do is just make sure that we would be able to win as quickly as we have been able to do in the past.” ... The weapons, in the Pentagon’s vision, would be less like the Terminator and more like the comic-book superhero Iron Man Such blatant propaganda by the NYT. Just like with Stuxnet, the US is pouring black budget dollars into novel asymmetric offensive weapons that'll be rapidly put to use in the field, all under the guise of "defense". It's grossly irresponsible. In the position of sole superpower barely hanging onto our role with the obvious rise of a multipolar world, the USA should be pulling all its strings to put in place international institutions and laws that will uphold stability and the type of global civilization we've become used to. Instead we're throwing money that should be rebuilding infrastructure and paying for people's health care into crazy destabilizing weapons systems where we'll have a 5 year "strategic advantage" headstart on the tech but the precedents we set (a cyberweapon that infected computers worldwide, Terminator autonomous weapons?) will come back to haunt us and the rest of the world.
- M_Grey 10y agoThe military-industrial complex won. It's that simple. We just blew trillions in Iraq and Afghanistan, and people are basically baying for more. We're cooked.
- orly_bookz 10y agoCounterpoint: maybe we become the Klingon empire?
- M_Grey 10y agoI'd say the key to their success was FTL travel and a big galaxy to plunder.
- djsumdog 10y ago
- codazoda 10y agoLike others have said, these are inevitable. One thing I worry about in warfare is that some enemies have less work to do because they have a different moral compass. Take a terrorist organization for example. If they build a similar drone it does not need to identify armed people, only people or even just movement. That is a much simpler problem to solve. If they can get a drone to their enemies it can kill indiscriminately. If the enemy has few people in the area or they are willing to die for their cause, their robot doesn't have to worry about who's armed and who isn't.
- Jtsummers 10y agoISIL is already reported to be using drones with improvised weaponry. It's only a matter of time, now, before this becomes more ubiquitous for both sides in wars. http://www.bloomberg.com/news/articles/2016-07-07/armed-drones-used-by-islamic-state-posing-new-threat-in-iraq http://www.bloomberg.com/news/articles/2016-07-07/armed-dron... http://www.aljazeera.com/news/2016/10/isil-drone-deadly-iraq-attack-161012151854280.html http://www.aljazeera.com/news/2016/10/isil-drone-deadly-iraq...
- ryuker16 10y agoIt's 150x cheaper to pay some local thug to ice targets or launch mortars.
- tim333 10y agoI guess an upside to this stuff could be dealing with situation like ISIL trying to do genocide stuff to the Yazidis (https://en.wikipedia.org/wiki/Genocide_of_Yazidis_by_ISIL https://en.wikipedia.org/wiki/Genocide_of_Yazidis_by_ISIL). You couldn't really bomb away because you'd kill the wrong side, you couldn't send ground troops because ISIL would behead them on youtube whereas AI killer quadcopters might get somewhere.
- eternalban 10y agoWondering who do you think created, funds, and arms "ISIL". [p.s.] Looking into these facts is important and very much related to topic at hand. "Terminators" are for urban pacification, but the pretext these days is "terrorism" and "insurgents". It is likely a high probability event that your own children will have to deal with these terminators, and not the "Russians" or the "Chinese".
- jwatte 10y agoLetting killer robots loose within an area of engagement is not that different from strategic/carpet bombing. Both are bad news for any civilians stuck in the area.
- fatlasp 10y ago"Kill Decision" by Daniel Suarez spins this topic in to a great work of fiction.
- sqeaky 10y agoI find the comparison to submarines in WWII to drones to lacking in intellectual honesty. Treaties on new frontiers of war are common, we still have a treaty prohibiting the militarization of space. Someday it will be broken. Will it be a war-crime if it saves lives? What if ISIS has a satellite with nukes on it or something equally life threatening, should we heed the treaty even though doing so leads to our obvious demise? In WWII Japan was an existential threat to the United States. There is no similar existential threat with drones, just some hand waving about China and Russia. And there are a dozen other ways the comparison breaks down. What precautions are we taking to avoid harm to civilians? How do we insure transparency in this matter? How do we actually measure effectiveness? How do we prevent malicious groups from getting this tech from recovered drones lost in the field? The comparison dodges all the really hard questions and just smears the US and the US military.
- pasbesoin 10y agoPaywall... Anyways, I'm reminded of viewing this from two sides: 1) Developing technology that kills humans; 2) Telling technology that avoids killing humans that "that's not a human". If it's mechanically possible, someone's going to do it. The one restraint is accountability. And, which direction is that going?
- pc2g4d 10y agoThe article doesn't make an important distinction: the computing power required for AI applications like Siri (massive) far outstrips the computing power available on a lightweight drone or "Terminator" weapon. Highly sophisticated AI will not be on the battlefield without introducing a dependency on a network connection to the centralized AI service. There's a network-dependence / computational-capability tradeoff that will prove important.
- samgranieri 10y agoNo. Just don't do it.