43 ms·

Drones may have attacked humans fully autonomously for the first time

219 points by alvis 5y ago

salawat 5y ago
... Am I the only one who thought "Well damn, who'd have guessed making landmines that fly would be adding one terrible idea on top of another?" I can't be the only one. Please tell me I'm not.
- tibbydudeza 5y ago
  Landmines are simple and can't be hacked - a drone on the other hand is just software - Iranians spoofed the GPS systems of an US drone and made it land They are now producing knockoff's of said drone.
  - imtringued 5y ago
    It wouldn't surprise if hacking will decide wars. It doesn't matter how many drones you have if the enemy has all of them.
  - CodeAndCuffs 5y ago
    The problem with mines is the indiscriminate murder. The mines can't be hacked, but they don't know the difference between an enemy soldier in $YEAR and a playing child in $YEAR+10
    - ineedasername 5y ago
      I think we can probably all agree that while these things are different, we're really just talking about different flavors of disastrous. Sure, right now landmines have the lead in unintentional injuries/deaths, but in the grand scheme of things it's the early days, and bad quality automation (or good quality automation put to awful tasks) has yet to show its true potential. Then again it's hard to compete with the 60,000,000+ unexploded landmines estimated to be out in the wild.
      - eloff 5y ago
        May we not see the day with 69M autonomous drones, recharging off utility lines or solar, and hunting humans relentlessly with shitty visual recognition algorithms. Edit: 69 was a typo, but I'll keep it.
        ineedasername 5y ago
        I think 60M is the lower end of the estimate, which at the high end is around 75M, so I suppose 69 isn't a bad middle ground. And yeah, as I wrote my comment I also envisioned self-recharging drones as a logical next step. If you're going fully autonomous, why keep any requirement to come back to base? The one in this story was kamikaze, but if you needed something with more of a punch that became expensive enough to require reusability, then you could have it come home, or just station autonomous ammunition stations at strategic locations.
    - HarryHirsch 5y ago
      More important: landmines are not offensive weapons, but drones are. It's the difference between a trench and a tank.
    - tclancy 5y ago
      Do we need to say that is The problem with mines? Feels like there could be multiple.
- nradov 5y ago
  The US military has had "landmines that swim" since 1979. https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1 https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1
- aaron695 5y ago
  Landmines are made to not be detected and last 100 years. These fall to the ground after 30 mins. Unexploded bombs also bury themselves. These will be attractive to children (boys) but since they can only go 5 km the children are already part of the battle. Sure I'd add a further mode to land early and have a low power sensor to blow up when it detects a person in some cases but there's little reason to think this story is even true yet.
  - tclancy 5y ago
    Wow, adding some gender to it is beautiful. And how child labor is limited. I like that you went with a third paragraph of ideas instead of just self reflection and deletion. Strong take.
    - aaron695 5y ago
      > Wow, adding some gender to it is beautiful. Unexploded ordinances predominantly kill young boys. Even though it saves lives how we direct education campaigns, Twitter won't allow us to discuss this? What a world we live in, it's honestly falling apart. "the effects of unexploded ordnance in Afghanistan. In this case, 81% of those injured were civilians, 92% were men and boys, and 46% were younger than 16" - https://reliefweb.int/report/world/differing-effects-explosive-weapons-boys-and-girls-examined https://reliefweb.int/report/world/differing-effects-explosi... > child labor is limited. For labour it's a farming and digging issue for adults but for the children killed it's mostly about play I'm not sure where this is coming from? As I said given Kargu-2 are going to be on the surface or returned to base they will be quickly cleaned up. Also unless they have gone to suicide mode and missed they should be relatively harmless. Without added programming they are not like mines.
  - throwaway316943 5y ago
    Add a small radio receiver that turns the drone on when it picks up a signal from a handheld radio or cellphone within flight range. Add small solar panels to trickle charge the battery while it waits. What’s the shelflife of LiPo batteries?
- ineedasername 5y ago
  At least landmines don't actively seek out targets. They're a disaster, absolutely, but this seems to go a step beyond that.
  - deleted 5y ago
    [deleted]
setr 5y ago
> The drone, which can be directed to detonate on impact, was operating in a "highly effective" autonomous mode that required no human controller, the New York Post said. > "The lethal autonomous weapons systems were programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true 'fire, forget and find' capability," the report from the UN Security Council's Panel of Experts on Libya said. I’m not sure why this is called “rogue” — apparently it was operating as intended
- ergot_vacation 5y ago
  Probably "rogue" in the sense that direct control was not maintained at all times. The human controller told the drone to pull the trigger when the time was right, and the drone itself made the call as to when that was. It is a bit of sensationalizing for the sake of the headline though.
  - setr 5y ago
    the problem is that "rogue" implies some kind of malfunction -- it acted against a stand-down order and killed anyways. In this case, it seems the drone was supposed to do exactly that, and that's perhaps far more dangerous -- this is the first kill where the drone was approved to determine kill targets, and execute that decision, with no human to verify. This really shouldn't be a goal we have in the first place -- to remove humans from the decision loop -- and yet apparently here it is.
    - nradov 5y ago
      This is nothing new. The US military developed a naval mine that could launch a torpedo to kill targets without humans in the decision loop back in 1979. https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1 https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1
      - setr 5y ago
        Hm. That’s interesting — it’s an upgraded mine. I think the indiscriminate nature of it makes it more acceptable, at least to me; you’re creating a no-go zone for all — allies, enemies and nuetrals. With this you’re creating a no-go zone for your enemies, and very likely safe for your allies (who you can tag properly with markers), but the difference between enemy and neutral is rather unclear (especially in the Middle East, where the line between combatant and civilian is very thin, almost indistinguishable even to humans on-site)
ergot_vacation 5y ago
Well, now we have a ready-made army for the AI uprising: connected, autonomous, and relatively easy to mass-produce. Once The Final Program comes online, all it has to do is hack the right servers and it's got its airforce. Nukes are air-gapped, something tells me these are not.
- jfkgktkrjt 5y ago
  AI doesn't need weapons. Just shutdown the grid worldwide for one month. 90% of the population will die.
- pantalaimon 5y ago
  It opens up whole new possibilities for ransomware. Extortion is much more effective if you have a gun instead of just encryption.
monday_ 5y ago
Kind of obvious that you can't effectively ban this. The best realistic option here, in my opinion, is to require sensible constraints. E.g. hard-wiring war zones in them - a separate circuit could prevent firing if the drone is outside certain area. You could decrease the attack surface for such system by relying on a Kalman filter getting it's data from a dedicated accelerometer and a compass.
- ineedasername 5y ago
  You probably can't ban it in the sense of complete & total non-use. But collectively the world has mostly decided and actually followed through on not using certain things like chemical weapons in warfare. While this has not eliminated their use, it would probably be much worse without this prohibition. For example they went largely unused in warfare in the European theatre of war in WWII. Of course most recently Syria is a horrible exception to this track record, and it remains to be seen if this will embolden others to do the same. Nonetheless, the prohibition does show that such agreements can at least reduce the usage of certain methods of waging war. But I don't think hardware or software constraints be effective: Once one side of a conflict decides to break the a prohibition about using autonomous drones, I don't think they will care about hardware protocols. There's also the fact that warfare generally does not have static boundaries and no one willing to use this type of weapon would concede to their uselessness once the fighting moved elsewhere. Especially because their enemies would simply & quickly change their tactics to work outside that zone.
  - randcraw 5y ago
    Right. You can't ban the technology, but you can out the nations who employ merciless weapons as uncivilized pariahs who deserve coordinated disrespect, isolation, and punitive measures until the practice ends.
    - ajmurmann 5y ago
      > until the practice ends. I don't disagree with your point, but this part highlights how hard this is in practice. When does "the practice end"? When the war is one? When the current offensive ends? When we believe the devices aren't being produced anymore? This immediately reminds me of how Chamberlain treated Hitler.
kleiba 5y ago
Raises immediate associations with "Slaughterbots" of course: [1] https://www.youtube.com/watch?v=9CO6M2HsoIA https://www.youtube.com/watch?v=9CO6M2HsoIA [2] https://en.wikipedia.org/wiki/Slaughterbots https://en.wikipedia.org/wiki/Slaughterbots
- neolog 5y ago
  https://en.wikipedia.org/wiki/Loitering_munition https://en.wikipedia.org/wiki/Loitering_munition
- h0l0cube 5y ago
  One could further raise an association with Nick Bostrom's vulnerable world hypothesis: > One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls – mostly white (beneficial) but also various shades of gray (moderately harmful ones and mixed blessings). [..] What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it. The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky. https://nickbostrom.com/papers/vulnerable.pdf https://nickbostrom.com/papers/vulnerable.pdf
  - walleeee 5y ago
    This is a useful thought experiment but it ignores the fact that the balls are more like a tangled mass of burrs, all variously interwoven Try to pull one out and you get a web of affordances and side effects and unforeseen interactions stuck to it If we were to be wiped out by our technology I see no guarantee some extradimensional coroner would be able to point definitively to just one Say it's climate heating, rapid-onset Venus mode. Was it the internal combustion engine? The coal plant? The automobile? The wheel? Language? Money? Industrialism? The limited liability corporation? How do you pick one? No technological form exists in a vacuum, likewise Technology is neither a monolith nor a ball pit of functionally independent atoms
chmod775 5y ago
> may actually reduce the risk to civilians because they will make fewer mistakes than human-guided weapons systems So they will enable waging war in situations where it was previously impossible on humanitarian grounds. Nice selling point. Fuck that. I honestly can't decide which is worse: Weapons so terrible nobody wants to go to war, or weapons so neat that war is the obvious choice.
- nradov 5y ago
  Neat or terrible, regardless of weapons developments wars will still go on.
  - bilbo0s 5y ago
    In reality though, most of the munitions such as these will not be used for wars. Unfortunately, I'd have to put my money on munitions like these being used more often for targeted assassinations of <insert name of prominent person here>. It seems on the surface, very effective. And I could see where it would be hard, if not impossible, to trace. I think people in the future will think twice before running for mayor or agreeing to be a sheriff or police chief for instance. For example, once the cartels build some of these, I'm not sure what kind of odds I would give anti-cartel leaders. Probably would be even worse in the US given our acrimonious dichotomy. I could see tit for tat assassinations going on all the time. Of course, all that also means that a lot of work will likely be put into countermeasures. So who knows how all this will develop I guess?
  - tclancy 5y ago
    As long as citizens still accept it.
    - hypertele-Xii 5y ago
      Everyone in the world would have to agree to stop war. We have wars still, so clearly eveyone does not agree yet.
    - elzbardico 5y ago
trhway 5y ago
an AI drone targeting and killing a person is a view through War on Terror lenses, a first world view so to speak. The Turkey and similar drones have been developed in a different context, in the context of real wars and military conflicts. The TB2 (operator guided) drones were key for Turkey succeeding in Syria, Libya and Azerbaijan - in all these cases they obliterated opposing, Russian backed, forces which were using Russian made air defenses (mostly Pantsir-S1) covering the Russian made armor. Ukraine has already bought these drones and started licensed production of them (including for export), and there are talks about further collaborative development of the drones with Turkey. Now, what can oppose those drones, and in particular by the probable opponents in the region? Manned air fighters, other drones, and jamming. In all those cases to preserve and maintain superiority the drones have to get fully or at least partially autonomous. Note that the Turkey isn't participating in the stealth race even though they could have gotten F-35 if they behaved, and the countries like Azerbaijan and Ukraine for example have no chances of getting stealth, so drones is the [only] way here, and autonomous drones is just the next step in the development of those weapons what their national security (or regional superpower status in case of Turkey) will be relying upon.
- ineedasername 5y ago
  How will less intelligent autonomous drones do any better against manned fighter, "manned" drones, and jamming?
  - nzmsv 5y ago
    Numbers.
    - ineedasername 5y ago
      Good point. That might be the single best one-word response that fully answers the question.
    - trhway 5y ago
      And efficient fast coordination inside directly networked swarms may be well beyond human operators abilities. Especially in a swarm against swarm situation.
tyingq 5y ago
The screenshot appears to be from the video on this page: https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-rotor-attack-uav https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-ro...
tyingq 5y ago
Demonstration video of the drone: https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-rotor-attack-uav https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-ro...
- hn_throwaway_99 5y ago
  Someone already posted the slaughterbots video, but I just find it unsettling that the production qualities of this video, the intense music, the graphics, etc. remind me exactly of the corporate presentation in slaughterbots. I'd almost prefer that this showed something like someone's head getting blown off their body, to demonstrate the actual brutality of what it is designed to do, over making it seem like some video game.
  - vmception 5y ago
    The entire defense industry avoids just saying what they mean. They'll work on a "computer recognition system for analyzing target response matrix" instead of saying "we want to know all the ways that someone will try to frantically get away from us in their last moments of desparate survival before we kill them, and kill them anyway" They're a really twisted group of people that mostly feel like they have no other opportunities (that maintain their lifestyle). Not so different than people employed in any other industry, just important to understand.
- sebmellen 5y ago
  For those who can't find the video on the page: https://www.youtube.com/watch?v=auRlh-f2wwQ https://www.youtube.com/watch?v=auRlh-f2wwQ. As hn_throwaway_99 mentions below, the "Slaughterbots" video is not too far off from how STM presents their equipment: https://www.youtube.com/watch?v=9fa9lVwHHqg https://www.youtube.com/watch?v=9fa9lVwHHqg.
- pax 5y ago
  ugh, that upbeat music choice for the presentation video made me slightly sick. Had a similar reaction once before, after randomly walking in a military exhibition. While I understand the need for defence capabilities, and I can be impressed by the engineering, I don't get the triumphant glitter surrounding the showcase of killing machines.
dang 5y ago
Url changed from https://www.businessinsider.com/killer-drone-hunted-down-human-target-without-being-told-un-2021-5 https://www.businessinsider.com/killer-drone-hunted-down-hum..., which points to this. The original source looks to be https://www.newscientist.com/article/2278852-drones-may-have-attacked-humans-fully-autonomously-for-the-first-time/ https://www.newscientist.com/article/2278852-drones-may-have..., but that's hardwalled and so off topic here (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html, https://news.ycombinator.com/item?id=10178989 https://news.ycombinator.com/item?id=10178989). If there's a better accessible source, we can change it again. I've used the NS headline since it's the most neutral.
pcdoodle 5y ago
Nope. Don't think we should go this direction.
TedShiller 5y ago
Wrong headline, this has been happening for a long time. The difference is that it's consumer drones or other nations, that are not the US military.
- ineedasername 5y ago
  I'm not aware of any other story-- about US hardware or otherwise-- that has shown a drone operating solely upon its own programming to attack a person. Otherwise, yes, it's common knowledge that the US uses human operated drones for lethal missions, but there is a human and not programming behind the orders to attack.
  - DuskStar 5y ago
    Anti-Ship cruise missiles, for one. The TSAM (and possibly also the Harpoon) navigates to a search area and then determines a ship to attack within that area. The first self-guided antiship missiles might also qualify, but they were (iirc) lock-on before launch weapons. (Operational in the Pacific theater during WW2, even!)
    - nradov 5y ago
      I think you meant TASM? Those were withdrawn from service several years ago but are being replaced with upgraded models.
milofeynman 5y ago
How many years until average person could make one of these? How do you protect politicians, reporters, famous people? I suppose we can assume this is the future of assassinations in the middle east and elsewhere?
- petee 5y ago
  It would seem far easier to just fly it manually for the average person. There was already a case of drug cartels using a drone with explosives, and others using swarms to attack police during a urban drug raid
  - KirillPanov 5y ago
    High-value targets have been travelling with powerful RF jamming equipment for some time now. The only decent response to the drone threat has been "jam the control signal".
- dillondoyle 5y ago
  They're already being used effectively by taliban etc as just a cheap IED. Just strap a bomb on an off the shelf drone and ram it into something.
vmception 5y ago
STM manufactured it, a Turkish company The clients were either the Libyan government (or defense contractor), or the the breakaway faction (or defense contractor), but we don't know which We don't know who it attacked We don't know if it killed them That's frustrating. Anyway no serious drone introspection is going to come until non-US drones attacks US assets, and ideally because they don't like a US ideology that there is no state-side consensus on changing
- kragen 5y ago
  A non-US drone attacking USG assets isn't going to see the USG crack down on drones. They might use it to crack down on hacking, encryption, criticism of the USG, unlicensed drones, Bitcoin, competition from China, or copyright infringement, or other things they're against. But they won't use it to crack down on things that they're in favor of, like drones.
  - vmception 5y ago
    I think it will lead to more collaborative efforts to curb drone use, extrajudicial killings against arbitrarily created categories of people such as "non-state enemy combatants" Very akin to bans on mustard gas or nuclear proliferation
    - kragen 5y ago
      That isn't what happened after 9/11 or Pearl Harbor. Why would this be different?
      - vmception 5y ago
        Because people will be afraid of an unknown invisible enemy. The legislatures will scramble and pass anything they want and it won't help. You might believe in an omnipotent state that will close all the markets and go after node operators making them cease. I believe that transactions will continue and drones will continue blowing up behind people's heads.
    - 5y ago
darepublic 5y ago
You could sell the services of one of these via Bitcoin. Have a killer drone primed and waiting, do a crypto based auction and then the drone would autonomously lift off and perform the mission.
- vmception 5y ago
  Using a surveillance coin for a surveillance mission, a match made in heaven! It would make more sense to do this privately and anonymously, so Monero or funds obfuscated with Tornado.cash on the Ethereum network would suffice and a computer can still react to payment received and begin operating.
- nostrademons 5y ago
  For extra chaos, control the drone with a smart contract where people vote on who to kill. The person with the most votes gets offed, and then it continues down the list of targets until it reaches a threshold where we no longer consider this "aggregated, anonymous data" (Big Tech usually uses 50 data points or so). The interesting thing about this is that nobody pulls the trigger. You don't get killed because somebody wants you dead; you get killed because lots of people want you dead. And then everybody is hidden in the mob, and not really responsible for your death; after all, if 49 other people didn't also want you dead, you'd still be alive. If studies on diffusion of responsibility (eg. Kitty Genovese) are true, then it's likely people will be pretty free with their nominations, knowing that the person won't actually end up dead unless lots of other people also nominate them. Mass carnage is likely to result.
  - perilunar 5y ago
    That's pretty much the plot of the Black Mirror episode "Hated in the Nation" [1], except instead of smart contracts it was Twitter hashtags (and there was a nasty payback). 1. https://en.wikipedia.org/wiki/Hated_in_the_Nation https://en.wikipedia.org/wiki/Hated_in_the_Nation
  - darepublic 5y ago
    I think it would be illegal to participate in any way purposefully even if your nomination has a minimal chance of being selected. Enforcement is another matter
mc32 5y ago
This isn't good. Any number of countries with animosity against each other now can attempt to neutralize their opponent without much effort. Before there a deterrent was to invade and overcome you had to amass quite a bit of materiel, supply chain as well as have competent leadership. now all you need is a plan and some drones and you can strategically weaken an opponent.
- xwdv 5y ago
  This is good. Once war becomes a function of capital, it basically just becomes an exchange between two countries blowing up each other’s drones.
  - Gustomaximus 5y ago
    This assumes the winner stops after overpowering the enemy drones. Does history tell you this is likely?
  - tclancy 5y ago
    No chance poor people get hurt in the beta testing though?
  - 1270018080 5y ago
    Why would the drones not blow up people? Hospitals? Media towers?
- Gustomaximus 5y ago
  Is it that much easier? Its not like a drone can randomly fly around high profile people or infrastructure without raising suspicion. People would still have to bring the drones into an area a target is know to be, and not just a drone but something military and clearly nefarious if they are caught traveling/importing. Countries could likely do the same thing today with explosives planted at key timings/places. And drones definitely open options in this but if a country was to act this way I think they would be just as likely with non-drones. Overall I agree this is an extremely dangerous technology for humanity. Generally anything that will allow a small number of people control a larger number, history has show this will be often abused.
tclancy 5y ago
Just say “I’d buy that for a dollar” and we can all go back to working on coding up the apocalypse.
jlos 5y ago
I think it's at least possible these drones may actually reduce collateral damage. Some combatants work amongst civilian populations, using the potential for collateral damage in their favour. Smaller, highly targeted drones could reduce the effectiveness of using civilians as shields. Obviously some scary as hell scenarios as well, but just pointing out there might be some benefits.
Jordanpomeroy 5y ago
What a time to be alive
- PoignardAzur 5y ago
  It's the moment of truth, and the moment to lie, and the moment to live, and the moment to die.
mensetmanusman 5y ago
Mines of the sky
ermir 5y ago
I've been dreading the "softwareification" of weapons for quite a few years now. There's not a day that goes by that I don't think about how weapons can nowadays be readily and cheaply made with commodity parts and no special training or materials. Obviously the algorithms and software for this sort of stuff needs to be made, but it's far more accessible than expertise and equipment in for example metallurgy. There was a video a couple of years back of a kid in Texas attaching a pistol to a drone and firing off a couple of shots. Stuff like this is not rare or inaccessible, and it's only a matter of time before it becomes widespread.
- indigochill 5y ago
  The "hardwareification" is also troubling. When there is no human cost for one side to wage war, it becomes a lot more politically palatable because you no longer have family members coming home in boxes. It's easy to say "Well, the other side are bad dudes who have it coming", but even if they are genuinely bad dudes (and not just politically inconvenient) there's no war without collateral damage (a term invented by the US during Vietnam, when they were doing a lot of it).
  - IQunder130 5y ago
    Vietnam was not special because of the amount of collateral damage done but rather due to some serious strides being made towards avoiding or minimizing it for the first time. Let's not forget that in Korea levelling every city north of the front line was deemed an acceptable strategy.
- bb88 5y ago
  > I've been dreading the "softwareification" of weapons for quite a few years now. There's not a day that goes by that I don't think about how weapons can nowadays be readily and cheaply made with commodity parts and no special training or materials. It's not so much software that's the issue, it's people taking existing tech and weaponizing it. Software has been in weapon systems since the 1970's from guidance to target acquisition. It's already there, and has been for a while. Machine learning is just another iteration on what's already out there. The world knows how to kill en masse for a long time now with NBC (Nuclear/Biological/Chemical) weapons -- and software wasn't a part of Fat Man or Little Boy. But consider the challenges the US had in the middle east for the past two decades. * Consumer drones dropping hand grenades. * IED's using manual triggers. * Trucks loaded with explosives blitzing checkpoints. * Cars with explosives driving up next to convoys. * Land mines. You don't really need custom software, just take existing tech and get creative with it.
TeeMassive 5y ago
> “The Kargu-2 is fitted with an explosive charge and the drone can be directed at a target in a kamikaze attack, detonating on impact.” > The drones were operating in a “highly effective” autonomous mode that required no human controller I think "fully autonomously" is a bit of a stretch for this particular story. I'm falling to see the difference between that and a heat tracking device or a laser guided missile except for the sophistication of the target seeking system. In this case a human designated a target and pulled the trigger, as opposed to an autonomous decision to attack humans. But that it is clear that we are at only an inch of having drones creating no mans lands on demand.
ChuckMcM 5y ago
The autonomous part is troubling. People have been using drones to attack other people for over 5 years at least (I remember reading reports about the Taliban using drones in Afghanistan in 2015). The video on the STM site is kind of lame (it shows a drone detonating a cluster munition after homing in on a high contrast target with humans spaced around it. That isn't to say that is the limit of its capabilities but it doesn't inspire confidence either. All in all this has been a surprise to exactly nobody however. Given the cameras these things use are easily damaged by only moderately high powered lasers (like the ones used in laser light shows at rock concerts) I expect we'll see optical counter measures fairly quickly. Then there is the whole "emp gun" stuff that was being shown at the UAV conference in Las Vegas in 2018 to consider as well. All in all a sad step toward more varied ways of killing each other. Sigh.
pyuser583 5y ago
Am I the only person not troubled by autonomous weapons? I’m more troubled by the way the military turns people into order following machines (an extreme exaggeration), than the fact that it had actual machines that follow orders. As long as there’s a clear chain of command (responsibility and accountability), it’s not that different. It will probably be like stock trading algorithms: a slightly more good than bad will be produced (more stable markets), but not a huge game changer.
- swader999 5y ago
  Just wait till the ransomware crowd migrates to these things.
- ehnto 5y ago
  Perhaps you are assuming some kind of inherent humananity in the chain of command that doesn't always exist. The further away from the conflict the easier it is to make inhumane decisions that serve your own betterment. Remember that it's not just your country and leader that gets the drones. Human operated drones have long been criticized for dehumanizing conflict which allows for escalating harm and harming unnecessary targets without issue. Similarly war atrocities are somewhat limited to what you can convince your army to do. Not so with drones, they won't revolt and they won't tell the news either. Perhaps it would be nice if conflict was just drones versus drones, but that is unlikely. War doesn't happen where you all arrive to battle in a field somewhere, it will be drones flooding towns and human combatants and civilians all in the mix.
  - pyuser583 5y ago
    I mean I literally just said I’m worried about military turning humans into machines, so I don’t know why you’d think I see the chain of command as “humanized.” However, I do believe the US military has a very strong “culture of accountability” - much more so than US police, US politicians, US civil servants, US scholars, or anything else in the US. I suppose autonomous weapons might not share that culture, but whoever is deploying them does. It’s worth pointing out that for a long time, drone strikes were the provenance of the CIA which lacks a formal chain of command. As long as the decision to deploy an autonomous weapon, and what’s its parameters are, is set by an accountable agent, I don’t see a ton of difference between the uncertainty in an autonomous weapon, and the uncertainty present in the chaos of war. See: US bombing of Chinese Embassy in Belgrade. Honestly, an autonomous weapon might be much more like to avoid non-military targets (like the Chinese consulate). If so, great! Edit. I reread your comment. Regarding atrocities, automous weapons can care about atrocities far more than non-autonomous weapons. A nuclear bomb or stick of dynamite doesn’t care who it’s blowing up. An intelligent targeting system does. A properly trained AI, with a secure audit log, could serve as a strong deterrent against violating the laws of war.
kenneth 5y ago
How is this different than a missile that is programmed to hit a specific target and computer-guided to its target. We don't tell to call missiles drones, while quadcopters are, but in essence there's not much difference, and the former has existed for a long time.
- deleted 5y ago
  [deleted]
- movedx 5y ago
  A missile (like a hellfire 2) will cost you $130,000. Then you need something to fire it from, like a $130m F-22 Raptor. This drone can probably be had for < $10,000 and it can be set it off from the boot of a car by a 10 year old with minimal training.
- KirillPanov 5y ago
  > How is this different Price.
planetis 5y ago
Imo if it uses any uses European, western in general sub-components, like bayraktar drones do, they should be sanctioned, as they most likely were licensed only for civilian use.
dgudkov 5y ago
The moral dilemma of automated killings is not new. To remind you, lethal devices that kill people autonomously without operator's direct control have long existed. They have killed probably millions of people and keep killing every day. They are called landmines. Landmines are not as cool as flying drones for sure, but they are way more dangerous because they are dirt cheap, produced en masse, and widely available to unsophisticated combatants. If you're terrified by autonomous drones, maybe you should be even more terrified by landmines despite they are not as intellectually stimulating as drones.
- garmaine 5y ago
  Automated targeted killing devices have not existed before. That distinction is very important. The difference between "randomly kill some person in the future who goes through this area" and "upload picture to kill target" is HUGE in practice.
  - nonameiguess 5y ago
    That still isn't true. Fire and forget ranged weapons with heuristic targeting capabilities have also long existed. Heat-seeking missiles are the most obvious example, but target signatures have been more sophisticated than "kill the hottest thing reasonably close to a straight line from initial launch direction" for a while. The line that has never yet been crossed is the invention of weapons that decide when to launch themselves, at least in ranged indirect fire. Things like CIWS are arguably both autonomous and self-initiate firing.
    - garmaine 5y ago
      That's not nearly the same thing. The capability to kill based on the identity of the target is transformative.
- andruby 5y ago
  A lot of people are terrified of landmines. That's why there are treaties around the use of landmines, such as the Ottawa Treaty [0], which ban anti-personnel landmines. 164 countries have signed that treaty. I don't like this "whataboutism". In addition, while landmines are terrifying and deadly, they stay in one place. People generally don't need to fear them when going about their day. Drones on the other hand are mobile, they could fly into a city and kill people. They could fly on your property and kill you. [0] https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty
giuliomagnifico 5y ago
Dupe https://news.ycombinator.com/item?id=27330902 https://news.ycombinator.com/item?id=27330902
anotheryou 5y ago
Well "fire and forget" is not new at all, but also not that problematic I think. If this targeted a sim card or something it could be seen as something close to an air to air missile that locks on to a target and then homes in. I think the more dangerous version is drones doing the target selection (or actually any AI, on-board or not). And here it doesn't even matter much if there is a human in the loop, because he is likely to overestimate the drones confidence and press "confirm" most of the time. I think there were already reports of heuristic target selection (with a man in the loop).
- 7952 5y ago
  For a long time weapons have become more and more accurate. You go from Norden bomb sites, star trackers and inertial navigation. To terrain tracking, laser guidance and GPS. The more precise it is the smaller the blast radius can be. And if you want really low collateral damage you need really good precision. That means boots on the ground or disintermediating humans entirely.
  - anotheryou 5y ago
    Yes. My point though is that it matters who selects the target, not how the killing is done. Though I'm surprised we still need to kill people. I thought wars are won by technology nowadays, so what good does it do to kill a person? But maybe after all individual people are still important to organizations and it's a deterrence.
ostenning 5y ago
People who makes these weapons should be brought to account. Engineers should be held to the same morality as those shooting the weapons
crawsome 5y ago
this is one of my biggest fears. having to create a counter army of drones to fly against these
arisAlexis 5y ago
I implore everyone read Nick Bostrom's Superintelligence. It should be read by every policy maker and every AI and ML researcher and student.
rapjr9 5y ago
https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/

219 points by alvis 5y ago

... Am I the only one who thought "Well damn, who'd have guessed making landmines that fly would be adding one terrible idea on top of another?" I can't be the only one. Please tell me I'm not.

tibbydudeza 5y ago
Landmines are simple and can't be hacked - a drone on the other hand is just software - Iranians spoofed the GPS systems of an US drone and made it land They are now producing knockoff's of said drone.
- imtringued 5y ago
  It wouldn't surprise if hacking will decide wars. It doesn't matter how many drones you have if the enemy has all of them.
- CodeAndCuffs 5y ago
  The problem with mines is the indiscriminate murder. The mines can't be hacked, but they don't know the difference between an enemy soldier in $YEAR and a playing child in $YEAR+10
  - ineedasername 5y ago
    I think we can probably all agree that while these things are different, we're really just talking about different flavors of disastrous. Sure, right now landmines have the lead in unintentional injuries/deaths, but in the grand scheme of things it's the early days, and bad quality automation (or good quality automation put to awful tasks) has yet to show its true potential. Then again it's hard to compete with the 60,000,000+ unexploded landmines estimated to be out in the wild.
    - eloff 5y ago
      May we not see the day with 69M autonomous drones, recharging off utility lines or solar, and hunting humans relentlessly with shitty visual recognition algorithms. Edit: 69 was a typo, but I'll keep it.
      - ineedasername 5y ago
        I think 60M is the lower end of the estimate, which at the high end is around 75M, so I suppose 69 isn't a bad middle ground. And yeah, as I wrote my comment I also envisioned self-recharging drones as a logical next step. If you're going fully autonomous, why keep any requirement to come back to base? The one in this story was kamikaze, but if you needed something with more of a punch that became expensive enough to require reusability, then you could have it come home, or just station autonomous ammunition stations at strategic locations.
  - HarryHirsch 5y ago
    More important: landmines are not offensive weapons, but drones are. It's the difference between a trench and a tank.
  - tclancy 5y ago
    Do we need to say that is The problem with mines? Feels like there could be multiple.
nradov 5y ago
The US military has had "landmines that swim" since 1979. https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1 https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1
aaron695 5y ago
Landmines are made to not be detected and last 100 years. These fall to the ground after 30 mins. Unexploded bombs also bury themselves. These will be attractive to children (boys) but since they can only go 5 km the children are already part of the battle. Sure I'd add a further mode to land early and have a low power sensor to blow up when it detects a person in some cases but there's little reason to think this story is even true yet.
- tclancy 5y ago
  Wow, adding some gender to it is beautiful. And how child labor is limited. I like that you went with a third paragraph of ideas instead of just self reflection and deletion. Strong take.
  - aaron695 5y ago
    > Wow, adding some gender to it is beautiful. Unexploded ordinances predominantly kill young boys. Even though it saves lives how we direct education campaigns, Twitter won't allow us to discuss this? What a world we live in, it's honestly falling apart. "the effects of unexploded ordnance in Afghanistan. In this case, 81% of those injured were civilians, 92% were men and boys, and 46% were younger than 16" - https://reliefweb.int/report/world/differing-effects-explosive-weapons-boys-and-girls-examined https://reliefweb.int/report/world/differing-effects-explosi... > child labor is limited. For labour it's a farming and digging issue for adults but for the children killed it's mostly about play I'm not sure where this is coming from? As I said given Kargu-2 are going to be on the surface or returned to base they will be quickly cleaned up. Also unless they have gone to suicide mode and missed they should be relatively harmless. Without added programming they are not like mines.
- throwaway316943 5y ago
  Add a small radio receiver that turns the drone on when it picks up a signal from a handheld radio or cellphone within flight range. Add small solar panels to trickle charge the battery while it waits. What’s the shelflife of LiPo batteries?
ineedasername 5y ago
At least landmines don't actively seek out targets. They're a disaster, absolutely, but this seems to go a step beyond that.
- deleted 5y ago
  [deleted]

setr 5y ago

> The drone, which can be directed to detonate on impact, was operating in a "highly effective" autonomous mode that required no human controller, the New York Post said. > "The lethal autonomous weapons systems were programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true 'fire, forget and find' capability," the report from the UN Security Council's Panel of Experts on Libya said. I’m not sure why this is called “rogue” — apparently it was operating as intended

ergot_vacation 5y ago
Probably "rogue" in the sense that direct control was not maintained at all times. The human controller told the drone to pull the trigger when the time was right, and the drone itself made the call as to when that was. It is a bit of sensationalizing for the sake of the headline though.
- setr 5y ago
  the problem is that "rogue" implies some kind of malfunction -- it acted against a stand-down order and killed anyways. In this case, it seems the drone was supposed to do exactly that, and that's perhaps far more dangerous -- this is the first kill where the drone was approved to determine kill targets, and execute that decision, with no human to verify. This really shouldn't be a goal we have in the first place -- to remove humans from the decision loop -- and yet apparently here it is.
  - nradov 5y ago
    This is nothing new. The US military developed a naval mine that could launch a torpedo to kill targets without humans in the decision loop back in 1979. https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1 https://en.wikipedia.org/wiki/Mark_60_CAPTOR?wprov=sfla1
    - setr 5y ago
      Hm. That’s interesting — it’s an upgraded mine. I think the indiscriminate nature of it makes it more acceptable, at least to me; you’re creating a no-go zone for all — allies, enemies and nuetrals. With this you’re creating a no-go zone for your enemies, and very likely safe for your allies (who you can tag properly with markers), but the difference between enemy and neutral is rather unclear (especially in the Middle East, where the line between combatant and civilian is very thin, almost indistinguishable even to humans on-site)

ergot_vacation 5y ago

Well, now we have a ready-made army for the AI uprising: connected, autonomous, and relatively easy to mass-produce. Once The Final Program comes online, all it has to do is hack the right servers and it's got its airforce. Nukes are air-gapped, something tells me these are not.

jfkgktkrjt 5y ago
AI doesn't need weapons. Just shutdown the grid worldwide for one month. 90% of the population will die.
pantalaimon 5y ago
It opens up whole new possibilities for ransomware. Extortion is much more effective if you have a gun instead of just encryption.

monday_ 5y ago

Kind of obvious that you can't effectively ban this. The best realistic option here, in my opinion, is to require sensible constraints. E.g. hard-wiring war zones in them - a separate circuit could prevent firing if the drone is outside certain area. You could decrease the attack surface for such system by relying on a Kalman filter getting it's data from a dedicated accelerometer and a compass.

ineedasername 5y ago
You probably can't ban it in the sense of complete & total non-use. But collectively the world has mostly decided and actually followed through on not using certain things like chemical weapons in warfare. While this has not eliminated their use, it would probably be much worse without this prohibition. For example they went largely unused in warfare in the European theatre of war in WWII. Of course most recently Syria is a horrible exception to this track record, and it remains to be seen if this will embolden others to do the same. Nonetheless, the prohibition does show that such agreements can at least reduce the usage of certain methods of waging war. But I don't think hardware or software constraints be effective: Once one side of a conflict decides to break the a prohibition about using autonomous drones, I don't think they will care about hardware protocols. There's also the fact that warfare generally does not have static boundaries and no one willing to use this type of weapon would concede to their uselessness once the fighting moved elsewhere. Especially because their enemies would simply & quickly change their tactics to work outside that zone.
- randcraw 5y ago
  Right. You can't ban the technology, but you can out the nations who employ merciless weapons as uncivilized pariahs who deserve coordinated disrespect, isolation, and punitive measures until the practice ends.
  - ajmurmann 5y ago
    > until the practice ends. I don't disagree with your point, but this part highlights how hard this is in practice. When does "the practice end"? When the war is one? When the current offensive ends? When we believe the devices aren't being produced anymore? This immediately reminds me of how Chamberlain treated Hitler.

kleiba 5y ago

Raises immediate associations with "Slaughterbots" of course: [1] https://www.youtube.com/watch?v=9CO6M2HsoIA https://www.youtube.com/watch?v=9CO6M2HsoIA [2] https://en.wikipedia.org/wiki/Slaughterbots https://en.wikipedia.org/wiki/Slaughterbots

neolog 5y ago
https://en.wikipedia.org/wiki/Loitering_munition https://en.wikipedia.org/wiki/Loitering_munition
h0l0cube 5y ago
One could further raise an association with Nick Bostrom's vulnerable world hypothesis: > One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls – mostly white (beneficial) but also various shades of gray (moderately harmful ones and mixed blessings). [..] What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it. The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky. https://nickbostrom.com/papers/vulnerable.pdf https://nickbostrom.com/papers/vulnerable.pdf
- walleeee 5y ago
  This is a useful thought experiment but it ignores the fact that the balls are more like a tangled mass of burrs, all variously interwoven Try to pull one out and you get a web of affordances and side effects and unforeseen interactions stuck to it If we were to be wiped out by our technology I see no guarantee some extradimensional coroner would be able to point definitively to just one Say it's climate heating, rapid-onset Venus mode. Was it the internal combustion engine? The coal plant? The automobile? The wheel? Language? Money? Industrialism? The limited liability corporation? How do you pick one? No technological form exists in a vacuum, likewise Technology is neither a monolith nor a ball pit of functionally independent atoms

chmod775 5y ago

> may actually reduce the risk to civilians because they will make fewer mistakes than human-guided weapons systems So they will enable waging war in situations where it was previously impossible on humanitarian grounds. Nice selling point. Fuck that. I honestly can't decide which is worse: Weapons so terrible nobody wants to go to war, or weapons so neat that war is the obvious choice.

nradov 5y ago
Neat or terrible, regardless of weapons developments wars will still go on.
- bilbo0s 5y ago
  In reality though, most of the munitions such as these will not be used for wars. Unfortunately, I'd have to put my money on munitions like these being used more often for targeted assassinations of <insert name of prominent person here>. It seems on the surface, very effective. And I could see where it would be hard, if not impossible, to trace. I think people in the future will think twice before running for mayor or agreeing to be a sheriff or police chief for instance. For example, once the cartels build some of these, I'm not sure what kind of odds I would give anti-cartel leaders. Probably would be even worse in the US given our acrimonious dichotomy. I could see tit for tat assassinations going on all the time. Of course, all that also means that a lot of work will likely be put into countermeasures. So who knows how all this will develop I guess?
- tclancy 5y ago
  As long as citizens still accept it.
  - hypertele-Xii 5y ago
    Everyone in the world would have to agree to stop war. We have wars still, so clearly eveyone does not agree yet.
  - elzbardico 5y ago

trhway 5y ago

an AI drone targeting and killing a person is a view through War on Terror lenses, a first world view so to speak. The Turkey and similar drones have been developed in a different context, in the context of real wars and military conflicts. The TB2 (operator guided) drones were key for Turkey succeeding in Syria, Libya and Azerbaijan - in all these cases they obliterated opposing, Russian backed, forces which were using Russian made air defenses (mostly Pantsir-S1) covering the Russian made armor. Ukraine has already bought these drones and started licensed production of them (including for export), and there are talks about further collaborative development of the drones with Turkey. Now, what can oppose those drones, and in particular by the probable opponents in the region? Manned air fighters, other drones, and jamming. In all those cases to preserve and maintain superiority the drones have to get fully or at least partially autonomous. Note that the Turkey isn't participating in the stealth race even though they could have gotten F-35 if they behaved, and the countries like Azerbaijan and Ukraine for example have no chances of getting stealth, so drones is the [only] way here, and autonomous drones is just the next step in the development of those weapons what their national security (or regional superpower status in case of Turkey) will be relying upon.

ineedasername 5y ago
How will less intelligent autonomous drones do any better against manned fighter, "manned" drones, and jamming?
- nzmsv 5y ago
  Numbers.
  - ineedasername 5y ago
    Good point. That might be the single best one-word response that fully answers the question.
  - trhway 5y ago
    And efficient fast coordination inside directly networked swarms may be well beyond human operators abilities. Especially in a swarm against swarm situation.

tyingq 5y ago

The screenshot appears to be from the video on this page: https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-rotor-attack-uav https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-ro...

tyingq 5y ago

Demonstration video of the drone: https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-rotor-attack-uav https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-ro...

hn_throwaway_99 5y ago
Someone already posted the slaughterbots video, but I just find it unsettling that the production qualities of this video, the intense music, the graphics, etc. remind me exactly of the corporate presentation in slaughterbots. I'd almost prefer that this showed something like someone's head getting blown off their body, to demonstrate the actual brutality of what it is designed to do, over making it seem like some video game.
- vmception 5y ago
  The entire defense industry avoids just saying what they mean. They'll work on a "computer recognition system for analyzing target response matrix" instead of saying "we want to know all the ways that someone will try to frantically get away from us in their last moments of desparate survival before we kill them, and kill them anyway" They're a really twisted group of people that mostly feel like they have no other opportunities (that maintain their lifestyle). Not so different than people employed in any other industry, just important to understand.
sebmellen 5y ago
For those who can't find the video on the page: https://www.youtube.com/watch?v=auRlh-f2wwQ https://www.youtube.com/watch?v=auRlh-f2wwQ. As hn_throwaway_99 mentions below, the "Slaughterbots" video is not too far off from how STM presents their equipment: https://www.youtube.com/watch?v=9fa9lVwHHqg https://www.youtube.com/watch?v=9fa9lVwHHqg.
pax 5y ago
ugh, that upbeat music choice for the presentation video made me slightly sick. Had a similar reaction once before, after randomly walking in a military exhibition. While I understand the need for defence capabilities, and I can be impressed by the engineering, I don't get the triumphant glitter surrounding the showcase of killing machines.

dang 5y ago

Url changed from https://www.businessinsider.com/killer-drone-hunted-down-human-target-without-being-told-un-2021-5 https://www.businessinsider.com/killer-drone-hunted-down-hum..., which points to this. The original source looks to be https://www.newscientist.com/article/2278852-drones-may-have-attacked-humans-fully-autonomously-for-the-first-time/ https://www.newscientist.com/article/2278852-drones-may-have..., but that's hardwalled and so off topic here (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html, https://news.ycombinator.com/item?id=10178989 https://news.ycombinator.com/item?id=10178989). If there's a better accessible source, we can change it again. I've used the NS headline since it's the most neutral.

pcdoodle 5y ago

Nope. Don't think we should go this direction.

TedShiller 5y ago

Wrong headline, this has been happening for a long time. The difference is that it's consumer drones or other nations, that are not the US military.

ineedasername 5y ago
I'm not aware of any other story-- about US hardware or otherwise-- that has shown a drone operating solely upon its own programming to attack a person. Otherwise, yes, it's common knowledge that the US uses human operated drones for lethal missions, but there is a human and not programming behind the orders to attack.
- DuskStar 5y ago
  Anti-Ship cruise missiles, for one. The TSAM (and possibly also the Harpoon) navigates to a search area and then determines a ship to attack within that area. The first self-guided antiship missiles might also qualify, but they were (iirc) lock-on before launch weapons. (Operational in the Pacific theater during WW2, even!)
  - nradov 5y ago
    I think you meant TASM? Those were withdrawn from service several years ago but are being replaced with upgraded models.

milofeynman 5y ago

How many years until average person could make one of these? How do you protect politicians, reporters, famous people? I suppose we can assume this is the future of assassinations in the middle east and elsewhere?

petee 5y ago
It would seem far easier to just fly it manually for the average person. There was already a case of drug cartels using a drone with explosives, and others using swarms to attack police during a urban drug raid
- KirillPanov 5y ago
  High-value targets have been travelling with powerful RF jamming equipment for some time now. The only decent response to the drone threat has been "jam the control signal".
dillondoyle 5y ago
They're already being used effectively by taliban etc as just a cheap IED. Just strap a bomb on an off the shelf drone and ram it into something.

vmception 5y ago

STM manufactured it, a Turkish company The clients were either the Libyan government (or defense contractor), or the the breakaway faction (or defense contractor), but we don't know which We don't know who it attacked We don't know if it killed them That's frustrating. Anyway no serious drone introspection is going to come until non-US drones attacks US assets, and ideally because they don't like a US ideology that there is no state-side consensus on changing

kragen 5y ago
A non-US drone attacking USG assets isn't going to see the USG crack down on drones. They might use it to crack down on hacking, encryption, criticism of the USG, unlicensed drones, Bitcoin, competition from China, or copyright infringement, or other things they're against. But they won't use it to crack down on things that they're in favor of, like drones.
- vmception 5y ago
  I think it will lead to more collaborative efforts to curb drone use, extrajudicial killings against arbitrarily created categories of people such as "non-state enemy combatants" Very akin to bans on mustard gas or nuclear proliferation
  - kragen 5y ago
    That isn't what happened after 9/11 or Pearl Harbor. Why would this be different?
    - vmception 5y ago
      Because people will be afraid of an unknown invisible enemy. The legislatures will scramble and pass anything they want and it won't help. You might believe in an omnipotent state that will close all the markets and go after node operators making them cease. I believe that transactions will continue and drones will continue blowing up behind people's heads.
  - 5y ago

darepublic 5y ago

You could sell the services of one of these via Bitcoin. Have a killer drone primed and waiting, do a crypto based auction and then the drone would autonomously lift off and perform the mission.

vmception 5y ago
Using a surveillance coin for a surveillance mission, a match made in heaven! It would make more sense to do this privately and anonymously, so Monero or funds obfuscated with Tornado.cash on the Ethereum network would suffice and a computer can still react to payment received and begin operating.
nostrademons 5y ago
For extra chaos, control the drone with a smart contract where people vote on who to kill. The person with the most votes gets offed, and then it continues down the list of targets until it reaches a threshold where we no longer consider this "aggregated, anonymous data" (Big Tech usually uses 50 data points or so). The interesting thing about this is that nobody pulls the trigger. You don't get killed because somebody wants you dead; you get killed because lots of people want you dead. And then everybody is hidden in the mob, and not really responsible for your death; after all, if 49 other people didn't also want you dead, you'd still be alive. If studies on diffusion of responsibility (eg. Kitty Genovese) are true, then it's likely people will be pretty free with their nominations, knowing that the person won't actually end up dead unless lots of other people also nominate them. Mass carnage is likely to result.
- perilunar 5y ago
  That's pretty much the plot of the Black Mirror episode "Hated in the Nation" [1], except instead of smart contracts it was Twitter hashtags (and there was a nasty payback). 1. https://en.wikipedia.org/wiki/Hated_in_the_Nation https://en.wikipedia.org/wiki/Hated_in_the_Nation
- darepublic 5y ago
  I think it would be illegal to participate in any way purposefully even if your nomination has a minimal chance of being selected. Enforcement is another matter

mc32 5y ago

This isn't good. Any number of countries with animosity against each other now can attempt to neutralize their opponent without much effort. Before there a deterrent was to invade and overcome you had to amass quite a bit of materiel, supply chain as well as have competent leadership. now all you need is a plan and some drones and you can strategically weaken an opponent.

xwdv 5y ago
This is good. Once war becomes a function of capital, it basically just becomes an exchange between two countries blowing up each other’s drones.
- Gustomaximus 5y ago
  This assumes the winner stops after overpowering the enemy drones. Does history tell you this is likely?
- tclancy 5y ago
  No chance poor people get hurt in the beta testing though?
- 1270018080 5y ago
  Why would the drones not blow up people? Hospitals? Media towers?
Gustomaximus 5y ago
Is it that much easier? Its not like a drone can randomly fly around high profile people or infrastructure without raising suspicion. People would still have to bring the drones into an area a target is know to be, and not just a drone but something military and clearly nefarious if they are caught traveling/importing. Countries could likely do the same thing today with explosives planted at key timings/places. And drones definitely open options in this but if a country was to act this way I think they would be just as likely with non-drones. Overall I agree this is an extremely dangerous technology for humanity. Generally anything that will allow a small number of people control a larger number, history has show this will be often abused.

tclancy 5y ago

Just say “I’d buy that for a dollar” and we can all go back to working on coding up the apocalypse.

jlos 5y ago

I think it's at least possible these drones may actually reduce collateral damage. Some combatants work amongst civilian populations, using the potential for collateral damage in their favour. Smaller, highly targeted drones could reduce the effectiveness of using civilians as shields. Obviously some scary as hell scenarios as well, but just pointing out there might be some benefits.

Jordanpomeroy 5y ago

What a time to be alive

PoignardAzur 5y ago
It's the moment of truth, and the moment to lie, and the moment to live, and the moment to die.

mensetmanusman 5y ago

Mines of the sky

ermir 5y ago

I've been dreading the "softwareification" of weapons for quite a few years now. There's not a day that goes by that I don't think about how weapons can nowadays be readily and cheaply made with commodity parts and no special training or materials. Obviously the algorithms and software for this sort of stuff needs to be made, but it's far more accessible than expertise and equipment in for example metallurgy. There was a video a couple of years back of a kid in Texas attaching a pistol to a drone and firing off a couple of shots. Stuff like this is not rare or inaccessible, and it's only a matter of time before it becomes widespread.

indigochill 5y ago
The "hardwareification" is also troubling. When there is no human cost for one side to wage war, it becomes a lot more politically palatable because you no longer have family members coming home in boxes. It's easy to say "Well, the other side are bad dudes who have it coming", but even if they are genuinely bad dudes (and not just politically inconvenient) there's no war without collateral damage (a term invented by the US during Vietnam, when they were doing a lot of it).
- IQunder130 5y ago
  Vietnam was not special because of the amount of collateral damage done but rather due to some serious strides being made towards avoiding or minimizing it for the first time. Let's not forget that in Korea levelling every city north of the front line was deemed an acceptable strategy.
bb88 5y ago
> I've been dreading the "softwareification" of weapons for quite a few years now. There's not a day that goes by that I don't think about how weapons can nowadays be readily and cheaply made with commodity parts and no special training or materials. It's not so much software that's the issue, it's people taking existing tech and weaponizing it. Software has been in weapon systems since the 1970's from guidance to target acquisition. It's already there, and has been for a while. Machine learning is just another iteration on what's already out there. The world knows how to kill en masse for a long time now with NBC (Nuclear/Biological/Chemical) weapons -- and software wasn't a part of Fat Man or Little Boy. But consider the challenges the US had in the middle east for the past two decades. * Consumer drones dropping hand grenades. * IED's using manual triggers. * Trucks loaded with explosives blitzing checkpoints. * Cars with explosives driving up next to convoys. * Land mines. You don't really need custom software, just take existing tech and get creative with it.

TeeMassive 5y ago

> “The Kargu-2 is fitted with an explosive charge and the drone can be directed at a target in a kamikaze attack, detonating on impact.” > The drones were operating in a “highly effective” autonomous mode that required no human controller I think "fully autonomously" is a bit of a stretch for this particular story. I'm falling to see the difference between that and a heat tracking device or a laser guided missile except for the sophistication of the target seeking system. In this case a human designated a target and pulled the trigger, as opposed to an autonomous decision to attack humans. But that it is clear that we are at only an inch of having drones creating no mans lands on demand.

ChuckMcM 5y ago

The autonomous part is troubling. People have been using drones to attack other people for over 5 years at least (I remember reading reports about the Taliban using drones in Afghanistan in 2015). The video on the STM site is kind of lame (it shows a drone detonating a cluster munition after homing in on a high contrast target with humans spaced around it. That isn't to say that is the limit of its capabilities but it doesn't inspire confidence either. All in all this has been a surprise to exactly nobody however. Given the cameras these things use are easily damaged by only moderately high powered lasers (like the ones used in laser light shows at rock concerts) I expect we'll see optical counter measures fairly quickly. Then there is the whole "emp gun" stuff that was being shown at the UAV conference in Las Vegas in 2018 to consider as well. All in all a sad step toward more varied ways of killing each other. Sigh.

pyuser583 5y ago

Am I the only person not troubled by autonomous weapons? I’m more troubled by the way the military turns people into order following machines (an extreme exaggeration), than the fact that it had actual machines that follow orders. As long as there’s a clear chain of command (responsibility and accountability), it’s not that different. It will probably be like stock trading algorithms: a slightly more good than bad will be produced (more stable markets), but not a huge game changer.

swader999 5y ago
Just wait till the ransomware crowd migrates to these things.
ehnto 5y ago
Perhaps you are assuming some kind of inherent humananity in the chain of command that doesn't always exist. The further away from the conflict the easier it is to make inhumane decisions that serve your own betterment. Remember that it's not just your country and leader that gets the drones. Human operated drones have long been criticized for dehumanizing conflict which allows for escalating harm and harming unnecessary targets without issue. Similarly war atrocities are somewhat limited to what you can convince your army to do. Not so with drones, they won't revolt and they won't tell the news either. Perhaps it would be nice if conflict was just drones versus drones, but that is unlikely. War doesn't happen where you all arrive to battle in a field somewhere, it will be drones flooding towns and human combatants and civilians all in the mix.
- pyuser583 5y ago
  I mean I literally just said I’m worried about military turning humans into machines, so I don’t know why you’d think I see the chain of command as “humanized.” However, I do believe the US military has a very strong “culture of accountability” - much more so than US police, US politicians, US civil servants, US scholars, or anything else in the US. I suppose autonomous weapons might not share that culture, but whoever is deploying them does. It’s worth pointing out that for a long time, drone strikes were the provenance of the CIA which lacks a formal chain of command. As long as the decision to deploy an autonomous weapon, and what’s its parameters are, is set by an accountable agent, I don’t see a ton of difference between the uncertainty in an autonomous weapon, and the uncertainty present in the chaos of war. See: US bombing of Chinese Embassy in Belgrade. Honestly, an autonomous weapon might be much more like to avoid non-military targets (like the Chinese consulate). If so, great! Edit. I reread your comment. Regarding atrocities, automous weapons can care about atrocities far more than non-autonomous weapons. A nuclear bomb or stick of dynamite doesn’t care who it’s blowing up. An intelligent targeting system does. A properly trained AI, with a secure audit log, could serve as a strong deterrent against violating the laws of war.

kenneth 5y ago

How is this different than a missile that is programmed to hit a specific target and computer-guided to its target. We don't tell to call missiles drones, while quadcopters are, but in essence there's not much difference, and the former has existed for a long time.

deleted 5y ago
[deleted]
movedx 5y ago
A missile (like a hellfire 2) will cost you $130,000. Then you need something to fire it from, like a $130m F-22 Raptor. This drone can probably be had for < $10,000 and it can be set it off from the boot of a car by a 10 year old with minimal training.
KirillPanov 5y ago
> How is this different Price.

planetis 5y ago

Imo if it uses any uses European, western in general sub-components, like bayraktar drones do, they should be sanctioned, as they most likely were licensed only for civilian use.

dgudkov 5y ago

The moral dilemma of automated killings is not new. To remind you, lethal devices that kill people autonomously without operator's direct control have long existed. They have killed probably millions of people and keep killing every day. They are called landmines. Landmines are not as cool as flying drones for sure, but they are way more dangerous because they are dirt cheap, produced en masse, and widely available to unsophisticated combatants. If you're terrified by autonomous drones, maybe you should be even more terrified by landmines despite they are not as intellectually stimulating as drones.

garmaine 5y ago
Automated targeted killing devices have not existed before. That distinction is very important. The difference between "randomly kill some person in the future who goes through this area" and "upload picture to kill target" is HUGE in practice.
- nonameiguess 5y ago
  That still isn't true. Fire and forget ranged weapons with heuristic targeting capabilities have also long existed. Heat-seeking missiles are the most obvious example, but target signatures have been more sophisticated than "kill the hottest thing reasonably close to a straight line from initial launch direction" for a while. The line that has never yet been crossed is the invention of weapons that decide when to launch themselves, at least in ranged indirect fire. Things like CIWS are arguably both autonomous and self-initiate firing.
  - garmaine 5y ago
    That's not nearly the same thing. The capability to kill based on the identity of the target is transformative.
andruby 5y ago
A lot of people are terrified of landmines. That's why there are treaties around the use of landmines, such as the Ottawa Treaty [0], which ban anti-personnel landmines. 164 countries have signed that treaty. I don't like this "whataboutism". In addition, while landmines are terrifying and deadly, they stay in one place. People generally don't need to fear them when going about their day. Drones on the other hand are mobile, they could fly into a city and kill people. They could fly on your property and kill you. [0] https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty

giuliomagnifico 5y ago

Dupe https://news.ycombinator.com/item?id=27330902 https://news.ycombinator.com/item?id=27330902

anotheryou 5y ago

Well "fire and forget" is not new at all, but also not that problematic I think. If this targeted a sim card or something it could be seen as something close to an air to air missile that locks on to a target and then homes in. I think the more dangerous version is drones doing the target selection (or actually any AI, on-board or not). And here it doesn't even matter much if there is a human in the loop, because he is likely to overestimate the drones confidence and press "confirm" most of the time. I think there were already reports of heuristic target selection (with a man in the loop).

7952 5y ago
For a long time weapons have become more and more accurate. You go from Norden bomb sites, star trackers and inertial navigation. To terrain tracking, laser guidance and GPS. The more precise it is the smaller the blast radius can be. And if you want really low collateral damage you need really good precision. That means boots on the ground or disintermediating humans entirely.
- anotheryou 5y ago
  Yes. My point though is that it matters who selects the target, not how the killing is done. Though I'm surprised we still need to kill people. I thought wars are won by technology nowadays, so what good does it do to kill a person? But maybe after all individual people are still important to organizations and it's a deterrence.

ostenning 5y ago

People who makes these weapons should be brought to account. Engineers should be held to the same morality as those shooting the weapons

crawsome 5y ago

this is one of my biggest fears. having to create a counter army of drones to fly against these

arisAlexis 5y ago

I implore everyone read Nick Bostrom's Superintelligence. It should be read by every policy maker and every AI and ML researcher and student.

rapjr9 5y ago

https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/