11 ms·

Slaughterbots: Stop Autonomous Weapons [video]

215 points by georgecmu 9y ago

basicplus2 9y ago
I find it Strange that nobody generally comments on this including all the previous postings.. is it because people think it is inevitable and there is nothing they can do to stop it? or is it because people think it is fantasy and could never happen?
- trhway 9y ago
  why stop it? even at this early stage it seems to work really well - in a hall full of people it correctly identified its target (until of course presenter cheated). Also compare to a rifle bullet, this thing is much easier to shoot down. It also fits the progress we've been making - we removed civilians from the battlefield or at least we made it a war crime, and now we need to remove soldiers from it too (especially considering that they will become inferior - slower reaction, no 360 vision, slower speed, can't fly, can't really network and exchange full tactical info, etc. - to drones anyway). Let drones fight each other. For non-military setting it was already addressed in South Park for example - autonomous police drones will control the airspace. I think in the future any UFO will be just taken down like a car without license plates.
  - PhasmaFelis 9y ago
    > we removed civilians from the battlefield or at least we made it a war crime In the Middle East, the US likes to blow up buildings that they think might have terrorists in them with a drone-launched missile, then have the drone circle back and hit the rubble with another missile a few minutes later to kill any neighbors or EMTs coming to help the victims. > Let drones fight each other. Are you imagining a sci-fi future where nations politely agree to decide their disagreements with bloodless robot duels? Wars are won by destroying the enemy's will to fight, not by gentlemen's agreements. You seem to think that war can be made safe and civilized somehow. It reminds me of the 1897 New York Times writer who thought that machine guns would lead to peace on earth because leaders would hesitate to send their soldiers against such deadly weapons.
    - trhway 9y ago
      >Are you imagining a sci-fi future where nations politely agree to decide their disagreements with bloodless robot duels? Wars are won by destroying the enemy's will to fight, not by gentlemen's agreements. Where is no "politely agree" nor "gentlemen's agreements" in international relationships, only sheer power. MAD is an example of power forcing to not fight a war at all. Falkland war was fought without bombing Buenos Aires or any other action outside of immediate war zone. Ukraine/Russia war in Donbass was limited to that area because of possible repercussions to either side for letting it spill out (and for the same reason both sides in general, with some exceptions, didn't indiscriminantly target civilians). US and other developed countries limit sending its soldiers in harms ways and hitting enemy civilians because of internal and external political repercussions. So it is always a power calculation, and as a result in this more and more interconnected and densely populated world we have 2 tendencies - restricting conflicts to specific localities and minimizing human involvement. Those 2 tendencies ultimately lead to drone wars confined to specific areas (preferably in space or in cyberspace - notice how much of adversarial activity happens more and more in cyberspace, yet nobody is going for real life action over it). Attempt to bomb each other's cities will be (and pretty much is already today) unthinkable like in case of Falkland war. Of course, when there is no containing power in sight, we immediately have what Saudis do in Yemen right now - it isn't even a war, it is just pure genocide.
- daxorid 9y ago
  Another option is that some of us really look forward to real-life sentry gun turrets; we actually want this future.
  - deleted 9y ago
    [deleted]
  - jabretti 9y ago
    Let's sit back and take a chill pill. Will this capability exist? Yes, probably. Am I concerned that this capability will be deployed against me? No, not really. Theoretically it could, but then again, so could a dude with a gun. I live in a democratic first-world country, the capabilities of my own government are likely to exceed the capabilities of those who want to kill me. And theoretically maybe my government might want to kill me... but it could already do that anyway, using a dude with a gun.
    - jrochkind1 9y ago
      It's nice to live in the country that's at the top of the global political order, it's just _our_ "enemies" that have to worry about this sort of thing. For now. Things change.
  - sorokod 9y ago
    Systems such as Phalanx CISW with fully automatic mode have been available for ages.
    - daxorid 9y ago
      Yes, but to clarify, I meant access by civilians, given the availability of cheap FLIR systems, stepper motor rigs, and OpenCV processing.
    - gaius 9y ago
      Systems such as Phalanx CISW with fully automatic mode have been available for ages Phalanx doesn't work tho' :-) In 1991 Iraqi forces launched a Silkworm missile against USS Missouri. The ship launched decoys... and its Phalanx guns promptly targeted those decoys. Fortunately there was a Royal Navy ship, HMS Gloucester, in the vicinity which dealt with the problem with a good British weapon, the Sea Dart missile. This was the first real "drone on drone" combat, if you will.
- redeploy-test 9y ago
  I noticed that too. I believe the answer is the repression of fear. The human mind will avoid awareness of a possibility that would involve extreme psychological distress. When something is "too terrible to imagine" ... people will completely not acknowledge it. This applies to many cognitive situations, including an individual's predictions about the future. It also applies to determining the intent of other people, and even a person's self image and memories. What is frightening is that the same principle really makes mass atrocities possible. Psychology and history show clearly that people would rather become docile and distracted than face the possibility that they may be murdered by an overwhelming force. BUT WHO CARES WHAT I THINK. MY IP IS SHADOWBANNED FROM HN.
neuro_imager 9y ago
Unfortunately I don't see a way this can be avoided.There has never been a weapon humanity imagined, designed, and built that it has not used. AI will be no different. A ban would be impossible to implement. What exactly would you ban? AI? Drones? facial recognition? Once you have a certain level of capability in these technologies, weaponising these things is the easy part.
- vadimberman 9y ago
  > There has never been a weapon humanity imagined, designed, and built that it has not used. What about the neutron bomb? But overall, yes, I agree, a rifle hanging on the wall will eventually go off.
  - tehlike 9y ago
    Matter of time.
    - zachrose 9y ago
      This is a temporal version of the No True Scotsman fallacy.
      - reificator 9y ago
        The "No Temporal Scotsman" fallacy if you will.
        aoeuasdf1 9y ago
        The "Non-Timely Scotsman" fallacy?
        afarrell 9y ago
        Outlander fans will be very sad.
  - mikeash 9y ago
boznz 9y ago
I thought it was real for a few seconds.. Wow Scary!
kirykl 9y ago
Software eating society. Suddenly our entire criminal justice paradigm breaks down without corporeal law breakers. Makes the 'corporation as a person' question seem sophomoric
free_everybody 9y ago
This is terrifying!! AHH!!!!
alexS 9y ago
Maybe we can petition.
- petre 9y ago
  Or start building anti drone capabilities.
hitekker 9y ago
Terrifying and fascinating, as all good wake-up calls should be. In the future, miniature exploding drones could indeed autonomously, distinguish between cranium and body, plot a flight route and divebomb at their victim. But, I don't think these skull-crackers will be fully autonomous, even with significant advances and development. Given the chance of collateral damage and the "fuzziness" of algorithms, the kill command or sequence will be still left to a human operator, like how America does so in the skies of Pakistan or with the machine-gun robots of the Middle-East[1]. In my mind, the the computer would display potential targets via live capture, and the human in the chair would cycle through them, and after filtering out the mistakes (a human dummy, a picture on the wall), type "y". Militaries would have this check less out of "consider the ethics and morality!" and more out of rational avoid-friendly-fire and ensure-maximum accuracy-of-the-payload. One exception to this approach could be indiscriminate killing; where the designers intentionally program the drones to lobotomize any target within a geographic range or a time, as long as that target exceeds a certain threshold. This approach is guaranteed to result in a lot of false positives, but would still be remarkably efficient for the purpose of terror or localized warfare. [2] On a side note, YouTube recommended a follow up video, depicting a real-life swarm, howling and circling around a target. https://www.youtube.com/watch?v=PYLP0pAGbE0 https://www.youtube.com/watch?v=PYLP0pAGbE0 [1]: https://en.wikipedia.org/wiki/Foster-Miller_TALON https://en.wikipedia.org/wiki/Foster-Miller_TALON [2]: Even remote controlled drones strikes are pretty iffy on who they kill, so it's not like unlikely that less savory actors won't mind blanketing swaths of countries with skull-crackers. https://www.nytimes.com/2015/04/24/world/asia/drone-strikes-reveal-uncomfortable-truth-us-is-often-unsure-about-who-will-die.html https://www.nytimes.com/2015/04/24/world/asia/drone-strikes-...
- briga 9y ago
  I think with modern advances in facial recognition technologies, a human participant wont be necessary. Have you seen how accurate these models are nowadays? Many big tech companies have already released tools that make it possible for just about anyone who can write code to implement sophisticated CNN models for facial recognition. These technologies are more accessible currently than they've ever been.
- mtgx 9y ago
deleted 9y ago
[deleted]
narrator 9y ago
It's like WWI all over again. In WWI, people went happily off to war thinking it's going to be nice and heroic and adventurous like the good old days and they all get slaughtered in a pathetic pointless way in a blink of an eye by newly developed automated war machinery. The first AI War is going to be a really brutal wakeup call. The part in sci-fi where anyone actually has to aim to kill an enemy was a big romantic lie. The robots don't miss and they draw and shoot before the humans even know they're there. William Gibson, to his credit, did anticipate this with the Slamhound assassin robots in Count Zero.
- reificator 9y ago
  > The first AI War is going to be a really brutal wakeup call. I know. I thought I was decent at strategy games, but AI War shook that confidence to the bone. I don't think I've ever won a game. http://store.steampowered.com/app/40400/AI_War_Fleet_Command/ http://store.steampowered.com/app/40400/AI_War_Fleet_Command... On a serious note, you're absolutely right. Sci-Fi has been lying to us for a very long time and I don't think people are ready to face what the reality will be. > The robots don't miss and they draw and shoot before the humans even know they're there. Maybe, though if those robots are based on neural networks maybe we'll luck out and get a situation like this: https://xkcd.com/1897/ https://xkcd.com/1897/ To download your file: Please click all the squares containing traitorous human scum.
  - RobertoG 9y ago
    "Sci-Fi has been lying to us for a very long time [..]" That's totally unfair. I think you have been reading the wrong science fiction.
- int_19h 9y ago
  You don't even need robots for that. You just need an advanced scope on a rifle that can still be transported by a human, and the human is also the one that identifies targets. Check this out: https://www.youtube.com/watch?v=q0oGZ4TZr5k https://www.youtube.com/watch?v=q0oGZ4TZr5k Now, we're not yet at the point where it's cost-effective to give every infantryman a thing like this - the scope alone costs more than most high-end sniper rifles. But costs will go down...
nopinsight 9y ago
People who can should apply to work on one of the AI Safety Research projects: https://futureoflife.org/ai-safety-research/ https://futureoflife.org/ai-safety-research/ The list covers a number of professors in top graduate programs and several angles should be tackled simultaneously to maximize our chances.
- Pinckney 9y ago
  This seems to be mostly about skynet scenarios, which is an entirely different problem.
  - nopinsight 9y ago
    Yes, I am aware. But weaponized drones and robots will give much more power to a future skynet or a lesser version that autonomously acts with minimal guidance which may unintentionally divert from human goals. This greatly increases the priority of AI Safety research. Some people I know kept arguing we can simply unplug the skynet and be mostly safe (despite some damage). These AI weapons make it clear that we will not be able to largely limit the damage to the virtual world.
    - rl3 9y ago
      >But weaponized drones and robots will give much more power to a future skynet ... Nah, not really. In the face of a proper hard-takeoff superintelligence, Skynet will look positively incompetent by comparison. There's enough manufacturing equipment connected to the internet today that a superintelligence could physically manifest whatever it wishes. That's not to mention all the human labor it could coerce, trick or pay. Put simply: as soon as the thing copied itself into the internet, that's game over. Better hope it likes us.
posnet 9y ago
Daniel Suarez, the author of Daemon, has another book called Kill Decision, which explores a lot of these ideas.
- mtgx 9y ago
  He's also had a great video on this topic for a few years: https://www.youtube.com/watch?v=pMYYx_im5QI https://www.youtube.com/watch?v=pMYYx_im5QI
p1xelHer0 9y ago
I, too, look forward to season 4 of Black Mirror.
- redeploy-test 9y ago
  It seems dangerous to joke that the idea of weapons like that are just science fiction. It isn't far-fetched at all, and technology like that could be here in a short matter of years or perhaps months.
- dag11 9y ago
  Interestingly, this was almost the exact plot to the the latest season finale.
  - tialaramex 9y ago
    I've observed elsewhere that the reason this is different is also the reason they couldn't use footage from "Hated In The Nation" (the Black Mirror episode you're thinking of) Hated In The Nation hypothesises that a lone nutter repurposes existing non-lethal technology to kill people. No explosives, just drill your way into the brain through the ear because the robots are replacements for flying insects. A law saying "Don't make killer robots" is useless in the Black Mirror scenario, nobody made killer robots except the lone nutjob they're already hunting for murder. So for this campaign it's an unhelpful message, it suggests their campaign would be futile.
jrochkind1 9y ago
I for real find it terrifying. If we were sane, we'd be pushing for international treaties against this same as chemical weapons, instead we're making it happen and selling the tech to people.
- int_19h 9y ago
  International treaties around weapons are hard to enact lately - it seems that most of the useful stuff that we have is mostly created in WW1-WW2 era. For more recent treaties, the usual pattern is that one is enacted, but US, Russia and China (and sometimes a few others who happen to be major users) simply refuse to be a party. Most other countries sign up, but it mostly doesn't matter, because they either don't do it on a large scale anyway for other reasons, or they ignore it when it's suddenly convenient. Examples: https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Convention_on_Cluster_Munitions https://en.wikipedia.org/wiki/Convention_on_Cluster_Munition...
speedplane 9y ago
Best and most telling part of the video: "trust me, these were all bad guys"
mickduprez 9y ago
personal EMP??
bawana 9y ago
Cannot be stopped. Whatever human mind imagines can be made. Only defense is counter drones. Everyone will buy one with next cell phone.
l0b0 9y ago
Just the availability of this kind of tech would scare anybody sane shitless. From that day, you'd better hope you don't ever piss off anybody within a 1000 km radius enough to find your photo and address online (easy in most countries) and drive to within drone distance of your house. Want to obliterate anyone's family completely? Done. Kill anyone in session at the government of a country - any country? Done.
- anonymous5133 9y ago
  That technology already exists and is being tested in a military setting https://www.youtube.com/watch?v=CGAk5gRD-t0 https://www.youtube.com/watch?v=CGAk5gRD-t0. Not sure if those ones have explosive on it but I'm sure it isn't harder to implement it.
- nine_k 9y ago
  Try flying a drone next to a major governmental building. In the best case, you'll be politely ordered to stop and leave. In worse cases, e.g. if you tried to conceal yourself, the drone might be shot down, and you face charges.
greggman 9y ago
Is this a leak of Black Mirror Season 4?
nathantross 9y ago
crazy
XorNot 9y ago
The message of this film feels extremely confused - probably because it's trying to single-mindedly push an agenda rather then explore the problem space. In a world where apparently drone-swarms are this cheap...what, no one ever developed counter-drones? It's the reason grey goo isn't a realistic scenario for nanotechnology: the goo has to replicate, whereas all the counter-goo has to do is kill it. Same issue: a world where the technology to build an incredibly cheap drone that size which can hunt humans with facial recognition based on specific parameters exists...is also a world in which an anti-drone can use the exact same technology and just ram into unidentified drone vehicles to disable them. If the swarms are cheap enough for terrorists, then the counter-swarms are monumentally cheaper. EDIT: It's worth noting you could argue other scenarios - like only governments building these, but that's what I mean by the message being confused - it's not clear what the threat is supposed to be.
- nopinsight 9y ago
  Complete defense is more expensive than targeted offence. Can we afford to and do we wish to have a number of anti-drones flying everywhere in public space and our own homes?
- soVeryTired 9y ago
  I don't think "people will invent countermeasures" is a particularly good argument. We have countermeasures against chemical weapons - but not everyone has them all the time. We would still be better off if we hadn't opened that pandora's box. For an eight minute video, it does a decent job of exploring the consequences of a weapon that can kill remotely, with minimal collateral damage, and with no supervision. They touch on potential use against terrorists, political opponents, and activists. The main point being that once you release thing into the wild, you have little control over how it will be used.
  - folknor 9y ago
    > We would still be better off if we hadn't opened that pandora's box. I don't think that's a reasonable assumption to make. I have no evidence either way, but I can easily instantly theorize that lots of medical advancements might have come from the same people that got their feet wet in chemical weapon technology, in later stages of their careers/life. Again, I don't really have an opinion, and haven't really thought it through - I'm just saying your blanket statement needs a citation.
dr0l3 9y ago
Magnificent piece of technology! Except it gets defeated by a 2$ skimask. This kind of fearmongering isn't really useful as a way of creating political change. If you want people to participate in a reasonable and constructive dialog then convince them to do so with reasonble and constructive arguments.
- jononor 9y ago
  Do you know that person identification based on gait detection is actively developed and getting decent results? Same with voice. Also in a really grim scenario, could threaten/kill anyone that hides their face, be it with a skimask, burka or whatever. Or even require positive identification, that your face is found in the database.
hans_mueller 9y ago
technically there is nothing new in this clip. People have been singled out and murdered for since we exist and sometimes nobody is singled out but everybody killed with a bomb. nonetheless there is something undeniably creepy and unsettling about being killed by a soulless machine.
EGreg 9y ago
I have been warning about this for years: https://www.schneier.com/blog/archives/2015/08/shooting_down_d.html#c6702487 https://www.schneier.com/blog/archives/2015/08/shooting_down...
jstewartmobile 9y ago
In a world of fighter-jets, satellites, and nukes, under the right circumstances this is a development which could cut both ways. From Wikipedia: "Quigley concludes that the characteristics of weapons are the main predictor of democracy.[0] Democracy tends to emerge only when the best weapons available are easy for individuals to buy and use." [0] http://www.carrollquigley.net/pdf/Weapons%20Systems%20and%20Political%20Stability.pdf http://www.carrollquigley.net/pdf/Weapons%20Systems%20and%20...
- georgecmu 9y ago
  Do Somalia, Afghanistan, and Iraq qualify as natural experiments that test this theory?
  - jstewartmobile 9y ago
    I doubt it. Jets, drones, and satellites vs peasants with automatics is a fairly stark disparity. I'm making more of a long-term guess here. Unlike uranium enrichment or jet construction, I suspect that state-of-the-art drone manufacturing may eventually reach a level of affordability to where drone-facilitated hits, feuds, and drive-bys become a thing on domestic soil. edit: And key part of the wiki quote is "the best weapons available". If the best weapon available is a six-shooter, and it has a price that almost everyone can afford, you've got an "equalizer". Democracy! If the best weapon available is a nuke, and it takes billions in supply chains and hundreds of millions in materials and know-how to make one, that is a lot of extra leverage for the guys at the top.
mnemotronic 9y ago
Thanks for the requirements and use case. How about a Gofundme to start on development? I also see a group providing drone attack services along the lines of booter/stresser DDoS providers (distributed denial of services). On the other hand, I think it will be a long time before visual face recognition will ever get this good, especially out in the real world with bad lighting, hats, eyeglasses etc. But let's say the world of the video authors comes true. What do we do? Defenses: 1. Microwaves can fry electronics. A system wouldn't need fancy targeting; just blast powerful microwaves in all directions. Downside: A. Microwaves this strong are bad for people too. B. All it takes is 1 anti-microwave drone to slip through and disable the system. C. Asymmetrical cost of offense vs. defense 2. Lasers can fry the optical sensors used by the drones Downside: A. Probably would need a fancy targeting system? B. Same problems as microwave defense 1B and 1C. 3. Something along the lines of WW2 barrage balloons. A network of suspended lines designed to entangle the drones. The lines wouldn't have to be very big; a piece of strong thread could get caught in the rotor of a drone and disable it. Downside: A. Wind gusts B. How do you suspend the thread? C. Do you carry one with you when you go out to Starbucks? D. The thread would have to be small enough to not be detected by the drone's camera but big enough to cripple it. 4. Ski mask. If the drone is programmed to attack specific individuals using facial recognition, take away it's targeting requirement. Downside: A. Maybe drone deployers don't care about a specific target -- just body count. 5. Play dead. It's assumed the drones wouldn't waste resources on targets that have already been attacked, so just emulate a victim. Downside: A. Big assumption 6. Wax museum defense. Have a mannequin with target's facial features. Drone attacks wax dummy. Downside: A. See 3C, 4A 7. Obscure the target. Throw a blanket over the target. Downside: A. Time to deploy. B. Darn. Left my blanket in the back in the ....POP!

215 points by georgecmu 9y ago

I find it Strange that nobody generally comments on this including all the previous postings.. is it because people think it is inevitable and there is nothing they can do to stop it? or is it because people think it is fantasy and could never happen?

trhway 9y ago
why stop it? even at this early stage it seems to work really well - in a hall full of people it correctly identified its target (until of course presenter cheated). Also compare to a rifle bullet, this thing is much easier to shoot down. It also fits the progress we've been making - we removed civilians from the battlefield or at least we made it a war crime, and now we need to remove soldiers from it too (especially considering that they will become inferior - slower reaction, no 360 vision, slower speed, can't fly, can't really network and exchange full tactical info, etc. - to drones anyway). Let drones fight each other. For non-military setting it was already addressed in South Park for example - autonomous police drones will control the airspace. I think in the future any UFO will be just taken down like a car without license plates.
- PhasmaFelis 9y ago
  > we removed civilians from the battlefield or at least we made it a war crime In the Middle East, the US likes to blow up buildings that they think might have terrorists in them with a drone-launched missile, then have the drone circle back and hit the rubble with another missile a few minutes later to kill any neighbors or EMTs coming to help the victims. > Let drones fight each other. Are you imagining a sci-fi future where nations politely agree to decide their disagreements with bloodless robot duels? Wars are won by destroying the enemy's will to fight, not by gentlemen's agreements. You seem to think that war can be made safe and civilized somehow. It reminds me of the 1897 New York Times writer who thought that machine guns would lead to peace on earth because leaders would hesitate to send their soldiers against such deadly weapons.
  - trhway 9y ago
    >Are you imagining a sci-fi future where nations politely agree to decide their disagreements with bloodless robot duels? Wars are won by destroying the enemy's will to fight, not by gentlemen's agreements. Where is no "politely agree" nor "gentlemen's agreements" in international relationships, only sheer power. MAD is an example of power forcing to not fight a war at all. Falkland war was fought without bombing Buenos Aires or any other action outside of immediate war zone. Ukraine/Russia war in Donbass was limited to that area because of possible repercussions to either side for letting it spill out (and for the same reason both sides in general, with some exceptions, didn't indiscriminantly target civilians). US and other developed countries limit sending its soldiers in harms ways and hitting enemy civilians because of internal and external political repercussions. So it is always a power calculation, and as a result in this more and more interconnected and densely populated world we have 2 tendencies - restricting conflicts to specific localities and minimizing human involvement. Those 2 tendencies ultimately lead to drone wars confined to specific areas (preferably in space or in cyberspace - notice how much of adversarial activity happens more and more in cyberspace, yet nobody is going for real life action over it). Attempt to bomb each other's cities will be (and pretty much is already today) unthinkable like in case of Falkland war. Of course, when there is no containing power in sight, we immediately have what Saudis do in Yemen right now - it isn't even a war, it is just pure genocide.
daxorid 9y ago
Another option is that some of us really look forward to real-life sentry gun turrets; we actually want this future.
- deleted 9y ago
  [deleted]
- jabretti 9y ago
  Let's sit back and take a chill pill. Will this capability exist? Yes, probably. Am I concerned that this capability will be deployed against me? No, not really. Theoretically it could, but then again, so could a dude with a gun. I live in a democratic first-world country, the capabilities of my own government are likely to exceed the capabilities of those who want to kill me. And theoretically maybe my government might want to kill me... but it could already do that anyway, using a dude with a gun.
  - jrochkind1 9y ago
    It's nice to live in the country that's at the top of the global political order, it's just _our_ "enemies" that have to worry about this sort of thing. For now. Things change.
- sorokod 9y ago
  Systems such as Phalanx CISW with fully automatic mode have been available for ages.
  - daxorid 9y ago
    Yes, but to clarify, I meant access by civilians, given the availability of cheap FLIR systems, stepper motor rigs, and OpenCV processing.
  - gaius 9y ago
    Systems such as Phalanx CISW with fully automatic mode have been available for ages Phalanx doesn't work tho' :-) In 1991 Iraqi forces launched a Silkworm missile against USS Missouri. The ship launched decoys... and its Phalanx guns promptly targeted those decoys. Fortunately there was a Royal Navy ship, HMS Gloucester, in the vicinity which dealt with the problem with a good British weapon, the Sea Dart missile. This was the first real "drone on drone" combat, if you will.
redeploy-test 9y ago
I noticed that too. I believe the answer is the repression of fear. The human mind will avoid awareness of a possibility that would involve extreme psychological distress. When something is "too terrible to imagine" ... people will completely not acknowledge it. This applies to many cognitive situations, including an individual's predictions about the future. It also applies to determining the intent of other people, and even a person's self image and memories. What is frightening is that the same principle really makes mass atrocities possible. Psychology and history show clearly that people would rather become docile and distracted than face the possibility that they may be murdered by an overwhelming force. BUT WHO CARES WHAT I THINK. MY IP IS SHADOWBANNED FROM HN.

neuro_imager 9y ago

Unfortunately I don't see a way this can be avoided.There has never been a weapon humanity imagined, designed, and built that it has not used. AI will be no different. A ban would be impossible to implement. What exactly would you ban? AI? Drones? facial recognition? Once you have a certain level of capability in these technologies, weaponising these things is the easy part.

vadimberman 9y ago
> There has never been a weapon humanity imagined, designed, and built that it has not used. What about the neutron bomb? But overall, yes, I agree, a rifle hanging on the wall will eventually go off.
- tehlike 9y ago
  Matter of time.
  - zachrose 9y ago
    This is a temporal version of the No True Scotsman fallacy.
    - reificator 9y ago
      The "No Temporal Scotsman" fallacy if you will.
      - aoeuasdf1 9y ago
        The "Non-Timely Scotsman" fallacy?
      - afarrell 9y ago
        Outlander fans will be very sad.
- mikeash 9y ago

boznz 9y ago

I thought it was real for a few seconds.. Wow Scary!

kirykl 9y ago

Software eating society. Suddenly our entire criminal justice paradigm breaks down without corporeal law breakers. Makes the 'corporation as a person' question seem sophomoric

free_everybody 9y ago

This is terrifying!! AHH!!!!

alexS 9y ago

Maybe we can petition.

petre 9y ago
Or start building anti drone capabilities.

hitekker 9y ago

Terrifying and fascinating, as all good wake-up calls should be. In the future, miniature exploding drones could indeed autonomously, distinguish between cranium and body, plot a flight route and divebomb at their victim. But, I don't think these skull-crackers will be fully autonomous, even with significant advances and development. Given the chance of collateral damage and the "fuzziness" of algorithms, the kill command or sequence will be still left to a human operator, like how America does so in the skies of Pakistan or with the machine-gun robots of the Middle-East[1]. In my mind, the the computer would display potential targets via live capture, and the human in the chair would cycle through them, and after filtering out the mistakes (a human dummy, a picture on the wall), type "y". Militaries would have this check less out of "consider the ethics and morality!" and more out of rational avoid-friendly-fire and ensure-maximum accuracy-of-the-payload. One exception to this approach could be indiscriminate killing; where the designers intentionally program the drones to lobotomize any target within a geographic range or a time, as long as that target exceeds a certain threshold. This approach is guaranteed to result in a lot of false positives, but would still be remarkably efficient for the purpose of terror or localized warfare. [2] On a side note, YouTube recommended a follow up video, depicting a real-life swarm, howling and circling around a target. https://www.youtube.com/watch?v=PYLP0pAGbE0 https://www.youtube.com/watch?v=PYLP0pAGbE0 [1]: https://en.wikipedia.org/wiki/Foster-Miller_TALON https://en.wikipedia.org/wiki/Foster-Miller_TALON [2]: Even remote controlled drones strikes are pretty iffy on who they kill, so it's not like unlikely that less savory actors won't mind blanketing swaths of countries with skull-crackers. https://www.nytimes.com/2015/04/24/world/asia/drone-strikes-reveal-uncomfortable-truth-us-is-often-unsure-about-who-will-die.html https://www.nytimes.com/2015/04/24/world/asia/drone-strikes-...

briga 9y ago
I think with modern advances in facial recognition technologies, a human participant wont be necessary. Have you seen how accurate these models are nowadays? Many big tech companies have already released tools that make it possible for just about anyone who can write code to implement sophisticated CNN models for facial recognition. These technologies are more accessible currently than they've ever been.
mtgx 9y ago

deleted 9y ago

[deleted]

narrator 9y ago

It's like WWI all over again. In WWI, people went happily off to war thinking it's going to be nice and heroic and adventurous like the good old days and they all get slaughtered in a pathetic pointless way in a blink of an eye by newly developed automated war machinery. The first AI War is going to be a really brutal wakeup call. The part in sci-fi where anyone actually has to aim to kill an enemy was a big romantic lie. The robots don't miss and they draw and shoot before the humans even know they're there. William Gibson, to his credit, did anticipate this with the Slamhound assassin robots in Count Zero.

reificator 9y ago
> The first AI War is going to be a really brutal wakeup call. I know. I thought I was decent at strategy games, but AI War shook that confidence to the bone. I don't think I've ever won a game. http://store.steampowered.com/app/40400/AI_War_Fleet_Command/ http://store.steampowered.com/app/40400/AI_War_Fleet_Command... On a serious note, you're absolutely right. Sci-Fi has been lying to us for a very long time and I don't think people are ready to face what the reality will be. > The robots don't miss and they draw and shoot before the humans even know they're there. Maybe, though if those robots are based on neural networks maybe we'll luck out and get a situation like this: https://xkcd.com/1897/ https://xkcd.com/1897/ To download your file: Please click all the squares containing traitorous human scum.
- RobertoG 9y ago
  "Sci-Fi has been lying to us for a very long time [..]" That's totally unfair. I think you have been reading the wrong science fiction.
int_19h 9y ago
You don't even need robots for that. You just need an advanced scope on a rifle that can still be transported by a human, and the human is also the one that identifies targets. Check this out: https://www.youtube.com/watch?v=q0oGZ4TZr5k https://www.youtube.com/watch?v=q0oGZ4TZr5k Now, we're not yet at the point where it's cost-effective to give every infantryman a thing like this - the scope alone costs more than most high-end sniper rifles. But costs will go down...

nopinsight 9y ago

People who can should apply to work on one of the AI Safety Research projects: https://futureoflife.org/ai-safety-research/ https://futureoflife.org/ai-safety-research/ The list covers a number of professors in top graduate programs and several angles should be tackled simultaneously to maximize our chances.

Pinckney 9y ago
This seems to be mostly about skynet scenarios, which is an entirely different problem.
- nopinsight 9y ago
  Yes, I am aware. But weaponized drones and robots will give much more power to a future skynet or a lesser version that autonomously acts with minimal guidance which may unintentionally divert from human goals. This greatly increases the priority of AI Safety research. Some people I know kept arguing we can simply unplug the skynet and be mostly safe (despite some damage). These AI weapons make it clear that we will not be able to largely limit the damage to the virtual world.
  - rl3 9y ago
    >But weaponized drones and robots will give much more power to a future skynet ... Nah, not really. In the face of a proper hard-takeoff superintelligence, Skynet will look positively incompetent by comparison. There's enough manufacturing equipment connected to the internet today that a superintelligence could physically manifest whatever it wishes. That's not to mention all the human labor it could coerce, trick or pay. Put simply: as soon as the thing copied itself into the internet, that's game over. Better hope it likes us.

posnet 9y ago

Daniel Suarez, the author of Daemon, has another book called Kill Decision, which explores a lot of these ideas.

mtgx 9y ago
He's also had a great video on this topic for a few years: https://www.youtube.com/watch?v=pMYYx_im5QI https://www.youtube.com/watch?v=pMYYx_im5QI

p1xelHer0 9y ago

I, too, look forward to season 4 of Black Mirror.

redeploy-test 9y ago
It seems dangerous to joke that the idea of weapons like that are just science fiction. It isn't far-fetched at all, and technology like that could be here in a short matter of years or perhaps months.
dag11 9y ago
Interestingly, this was almost the exact plot to the the latest season finale.
- tialaramex 9y ago
  I've observed elsewhere that the reason this is different is also the reason they couldn't use footage from "Hated In The Nation" (the Black Mirror episode you're thinking of) Hated In The Nation hypothesises that a lone nutter repurposes existing non-lethal technology to kill people. No explosives, just drill your way into the brain through the ear because the robots are replacements for flying insects. A law saying "Don't make killer robots" is useless in the Black Mirror scenario, nobody made killer robots except the lone nutjob they're already hunting for murder. So for this campaign it's an unhelpful message, it suggests their campaign would be futile.

jrochkind1 9y ago

I for real find it terrifying. If we were sane, we'd be pushing for international treaties against this same as chemical weapons, instead we're making it happen and selling the tech to people.

int_19h 9y ago
International treaties around weapons are hard to enact lately - it seems that most of the useful stuff that we have is mostly created in WW1-WW2 era. For more recent treaties, the usual pattern is that one is enacted, but US, Russia and China (and sometimes a few others who happen to be major users) simply refuse to be a party. Most other countries sign up, but it mostly doesn't matter, because they either don't do it on a large scale anyway for other reasons, or they ignore it when it's suddenly convenient. Examples: https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Convention_on_Cluster_Munitions https://en.wikipedia.org/wiki/Convention_on_Cluster_Munition...

speedplane 9y ago

Best and most telling part of the video: "trust me, these were all bad guys"

mickduprez 9y ago

personal EMP??

bawana 9y ago

Cannot be stopped. Whatever human mind imagines can be made. Only defense is counter drones. Everyone will buy one with next cell phone.

l0b0 9y ago

Just the availability of this kind of tech would scare anybody sane shitless. From that day, you'd better hope you don't ever piss off anybody within a 1000 km radius enough to find your photo and address online (easy in most countries) and drive to within drone distance of your house. Want to obliterate anyone's family completely? Done. Kill anyone in session at the government of a country - any country? Done.

anonymous5133 9y ago
That technology already exists and is being tested in a military setting https://www.youtube.com/watch?v=CGAk5gRD-t0 https://www.youtube.com/watch?v=CGAk5gRD-t0. Not sure if those ones have explosive on it but I'm sure it isn't harder to implement it.
nine_k 9y ago
Try flying a drone next to a major governmental building. In the best case, you'll be politely ordered to stop and leave. In worse cases, e.g. if you tried to conceal yourself, the drone might be shot down, and you face charges.

greggman 9y ago

Is this a leak of Black Mirror Season 4?

nathantross 9y ago

crazy

XorNot 9y ago

The message of this film feels extremely confused - probably because it's trying to single-mindedly push an agenda rather then explore the problem space. In a world where apparently drone-swarms are this cheap...what, no one ever developed counter-drones? It's the reason grey goo isn't a realistic scenario for nanotechnology: the goo has to replicate, whereas all the counter-goo has to do is kill it. Same issue: a world where the technology to build an incredibly cheap drone that size which can hunt humans with facial recognition based on specific parameters exists...is also a world in which an anti-drone can use the exact same technology and just ram into unidentified drone vehicles to disable them. If the swarms are cheap enough for terrorists, then the counter-swarms are monumentally cheaper. EDIT: It's worth noting you could argue other scenarios - like only governments building these, but that's what I mean by the message being confused - it's not clear what the threat is supposed to be.

nopinsight 9y ago
Complete defense is more expensive than targeted offence. Can we afford to and do we wish to have a number of anti-drones flying everywhere in public space and our own homes?
soVeryTired 9y ago
I don't think "people will invent countermeasures" is a particularly good argument. We have countermeasures against chemical weapons - but not everyone has them all the time. We would still be better off if we hadn't opened that pandora's box. For an eight minute video, it does a decent job of exploring the consequences of a weapon that can kill remotely, with minimal collateral damage, and with no supervision. They touch on potential use against terrorists, political opponents, and activists. The main point being that once you release thing into the wild, you have little control over how it will be used.
- folknor 9y ago
  > We would still be better off if we hadn't opened that pandora's box. I don't think that's a reasonable assumption to make. I have no evidence either way, but I can easily instantly theorize that lots of medical advancements might have come from the same people that got their feet wet in chemical weapon technology, in later stages of their careers/life. Again, I don't really have an opinion, and haven't really thought it through - I'm just saying your blanket statement needs a citation.

dr0l3 9y ago

Magnificent piece of technology! Except it gets defeated by a 2$ skimask. This kind of fearmongering isn't really useful as a way of creating political change. If you want people to participate in a reasonable and constructive dialog then convince them to do so with reasonble and constructive arguments.

jononor 9y ago
Do you know that person identification based on gait detection is actively developed and getting decent results? Same with voice. Also in a really grim scenario, could threaten/kill anyone that hides their face, be it with a skimask, burka or whatever. Or even require positive identification, that your face is found in the database.

hans_mueller 9y ago

technically there is nothing new in this clip. People have been singled out and murdered for since we exist and sometimes nobody is singled out but everybody killed with a bomb. nonetheless there is something undeniably creepy and unsettling about being killed by a soulless machine.

EGreg 9y ago

I have been warning about this for years: https://www.schneier.com/blog/archives/2015/08/shooting_down_d.html#c6702487 https://www.schneier.com/blog/archives/2015/08/shooting_down...

jstewartmobile 9y ago

In a world of fighter-jets, satellites, and nukes, under the right circumstances this is a development which could cut both ways. From Wikipedia: "Quigley concludes that the characteristics of weapons are the main predictor of democracy.[0] Democracy tends to emerge only when the best weapons available are easy for individuals to buy and use." [0] http://www.carrollquigley.net/pdf/Weapons%20Systems%20and%20Political%20Stability.pdf http://www.carrollquigley.net/pdf/Weapons%20Systems%20and%20...

georgecmu 9y ago
Do Somalia, Afghanistan, and Iraq qualify as natural experiments that test this theory?
- jstewartmobile 9y ago
  I doubt it. Jets, drones, and satellites vs peasants with automatics is a fairly stark disparity. I'm making more of a long-term guess here. Unlike uranium enrichment or jet construction, I suspect that state-of-the-art drone manufacturing may eventually reach a level of affordability to where drone-facilitated hits, feuds, and drive-bys become a thing on domestic soil. edit: And key part of the wiki quote is "the best weapons available". If the best weapon available is a six-shooter, and it has a price that almost everyone can afford, you've got an "equalizer". Democracy! If the best weapon available is a nuke, and it takes billions in supply chains and hundreds of millions in materials and know-how to make one, that is a lot of extra leverage for the guys at the top.

mnemotronic 9y ago

Thanks for the requirements and use case. How about a Gofundme to start on development? I also see a group providing drone attack services along the lines of booter/stresser DDoS providers (distributed denial of services). On the other hand, I think it will be a long time before visual face recognition will ever get this good, especially out in the real world with bad lighting, hats, eyeglasses etc. But let's say the world of the video authors comes true. What do we do? Defenses: 1. Microwaves can fry electronics. A system wouldn't need fancy targeting; just blast powerful microwaves in all directions. Downside: A. Microwaves this strong are bad for people too. B. All it takes is 1 anti-microwave drone to slip through and disable the system. C. Asymmetrical cost of offense vs. defense 2. Lasers can fry the optical sensors used by the drones Downside: A. Probably would need a fancy targeting system? B. Same problems as microwave defense 1B and 1C. 3. Something along the lines of WW2 barrage balloons. A network of suspended lines designed to entangle the drones. The lines wouldn't have to be very big; a piece of strong thread could get caught in the rotor of a drone and disable it. Downside: A. Wind gusts B. How do you suspend the thread? C. Do you carry one with you when you go out to Starbucks? D. The thread would have to be small enough to not be detected by the drone's camera but big enough to cripple it. 4. Ski mask. If the drone is programmed to attack specific individuals using facial recognition, take away it's targeting requirement. Downside: A. Maybe drone deployers don't care about a specific target -- just body count. 5. Play dead. It's assumed the drones wouldn't waste resources on targets that have already been attacked, so just emulate a victim. Downside: A. Big assumption 6. Wax museum defense. Have a mannequin with target's facial features. Drone attacks wax dummy. Downside: A. See 3C, 4A 7. Obscure the target. Throw a blanket over the target. Downside: A. Time to deploy. B. Darn. Left my blanket in the back in the ....POP!