5 ms·

There's another major issue with civilians and autonomous warfare machines of all sorts: Civilian control. For all of history, military power depended most of

There's another major issue with civilians and autonomous warfare machines of all sorts: Civilian control. For all of history, military power depended most of all on population size (to provide soldiers) and economic resources (to equip them). Mad wizards were limited by population. However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army. Also, note that these machines are often built for the US military by people who openly denegrate democracy and seem megalomaniacal; they have the technology and a lot of wealth. Could they build a factory somewhere and hide in their bunker until it's over?

deleted 3y ago
[deleted]
nonrandomstring 3y ago
> megalomaniacal persons with lust for power and disregard of others I think the problem with these sorts of people is that they are not actually that smart. Cognitively, as all-rounders, they lack crucial faculties of imagination, which not ironically are rooted in compassion, that lead to gross strategic errors. Landmines and impact grenades tied to a balloon are also "autonomous weapons". They are indiscriminate. The former are persistent and the later are ephemeral. The former are also reusable. Within a weapons taxonomy the most dangerous (for yourself) to manufacture are persistent, autonomous and reusable. In other words, like a knife in a fight, there's a very good likelihood your opponent will take it off you and use it against you. Most IEDs used against British and US soldiers are made from UXO. Weapons like drones are a gift to terrorists. For everything you make, one must imagine it being used on a busy city shopping street where your own family are. It's because the megalomaniacs are not actually that smart that they don't realise taking humans out of the loop is the same as relinquishing control, which is one step removed from handing control to the enemy.
- MichaelZuo 3y ago
  This seems a bit redundant to be honest. Any maniac who is actually super competent would be able to fool you into thinking otherwise. Because they can outwit you by definition. And the inverse also is tautological, of course you can spot those not that competent.
  - nonrandomstring 3y ago
    I guess there's two (non-contradictory) conversations going on at once here. My emphasis is more on what we can do, or not do, as participants in a technological society that chooses what technology to make. There is also the conversation on what to do about moustache-twisting, cat stroking, comedy Bond villains with tanks of laser-sharks. I don't think they are fooling any of us. They swagger around in plain sight, naked in their madness, and we could literally write a list of their names here on HN. Of course these strategic and psychiatric questions are related. "Dr. Strangelove" and "Don't look up" remain useful guides.
    - mistermann 3y ago
      > My emphasis is more on what we can do, or not do, as participants in a technological society that chooses what technology to make. The laws of physics do not (or are not known to at least) prevent people from desiring to fix this hilariously suboptimal "democracy" we live under and most everyone seems to hate (well, except when they're gushing about how great it is), but something does. I often wonder if propaganda and culture play a role. > I don't think they are fooling any of us. I believe the opposite - I believe at least 95% of people are adequately fooled (to sustain indefinite disinterest), and most of the remaining 5% are "conspiracy theorists" which we all know can be safely ignored.
  - BriggyDwiggs42 3y ago
    I think psycho/sociopath types are a bit like the uncanny valley, where most people can tell something is off without it being obvious what. It happens because being clever isn’t the same as having the absurd level of self control needed to fake normality.
- globalnode 3y ago
  relying on their "lack of well roundedness" isnt that great a strategy imho
- jongjong 3y ago
  The thing is it's usually going to be some freakishly weird outlier who will cause the problems. People who act within the realm of normalcy rarely pose an unexpected threat. There exist people who are both megalomaniac and rational and this is precisely the kinds of people who are typically selected for positions of power. However rare such individuals may be, countries like the US have a large pool of candidates to sift through. The people in power are capable of being perfectly rational in every aspect, except for the one aspect where they are required to be irrational. The system is essentially selecting people for their blind-spots and that's why we often see displays of extreme hypocrisy from politicians and other public figureheads. They are 100%, perfectly rational about everything except when it relates to pursuing their own self-interest, yet because they are rational in every other aspects, they know how to hide this irrationality from other people.
  - deleted 3y ago
    [deleted]
- wolverine876 3y ago
  > they are not actually that smart It includes billionaire leaders of the IT industry. Whatever their flaws, it doesn't stop them from developing, building, and scaling technology - beyond anyone else in the world.
  - Arainach 3y ago
    It is a mistake in logic to assume that because someone is successful they are talented or deserving. There is a huge luck factor and even larger "pre-existing advantages" factor i.e. rich parents, being born in the right area, etc. This is a fallacy very common in America in particular (though by no means only found there). If you have money, buy a sports team, and the managers of the team turn it into a champion, that does not inherently make you a gifted owner or manager - it means you had the capital to get in.
    - wolverine876 3y ago
      > It is a mistake in logic to assume that because someone is successful they are talented or deserving. There are other very strong indications of their capabilities. I suppose talent doesn't matter, just their ability to create outcomes.
      - trimethylpurine 3y ago
        Which is talent. Almost no one who has won the lottery was wealthy for very long. One exception is Stefan Mandel, a mathematician who bought all the tickets repeatedly. Very much a talent. Retaining a position of "luck" is itself a talent. It's a mistake to assume, but it's not a mistake to make a logical and educated guess that someone successful is likely also talented.
      - nonrandomstring 3y ago
        > just their ability to create outcomes Attila, Alexander, Hitler, Stalin ... those guys got things done. "Outcomes" need to be morally qualified.
deleted 3y ago
[deleted]
roenxi 3y ago
They don't even need to manufacture their own army. The weak point of any society is the military generals - if they try to seize then there are a tiny number of things that might stop them - their own troops, other generals and ... not sure if there is a 3rd thing. The big control in a typical democracy is that the troops of a rogue general will, presumably, disobey them and if the army fractures then the winning side will be the one with more people. Once mechanised armies are in play the numbers and assumptions around that dynamic change.
- wolverine876 3y ago
  Mature democracies don't have coups. That problem is solved.
  - roenxi 3y ago
    You need to go all the way to the end of the thought on this one. "Mature democracies don't have coups because ...". Democracies don't change human nature, there are still powerful people who want to execute coups in any given society.
    - wolverine876 3y ago
      > there are still powerful people who want to execute coups in any given society. There are still far more people in society who don't want a coup, and the desire for a coup isn't binary - democracy changes human behavior, obviously.
  - ricudis 3y ago
    Says who?
    - wolverine876 3y ago
      The evidence? Name a mature democracy where there was a coup.
      - 3y ago
bee_rider 3y ago
Plenty of coups have happened, and for most of recorded history, humanity was ruled over by some particularly fancy gangsters: monarchs, dictators, those sorts. Mad engineers could be worse, though. I don’t know. If there’s a good solution I guess somebody would propose it.
maxglute 3y ago
Switching to autonomous platforms might not be as labour saving as you think, especially distributed lethality model that focuses on swarms. It's trading crew infra for even more maintenance infra - there tend to be more ground logistics than operators.
paulddraper 3y ago
> megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army Mercenaries already exist. Commonly. Though in a developed country it's cheaper to buy lobbyists.
- wolverine876 3y ago
  Again, you can't build a human mercenary military with nearly the capabilities of any country but a small, poor one.
29athrowaway 3y ago
Guess what happens with most civilians when they are obsolete as a source of wealth and power. It does not take much imagination to see that the core skill of politicians, persuasion, will be obsolete as well. There will be no persuasion, no speeches, no campaigns. It will be just simply do as they say or perish. It's the end of democracy and the start of a new form of feudalism, but more violent. If we add brain implants to the mix, the prospect future gets even more grim. If you are important, information will be exfiltrated from your brain without your cooperation.
c_o_n_v_e_x 3y ago
>However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army. I've had a novel (as in writing a book) idea rattling around in my brain with something like this. A distributed AI living throughout the collective internet has humans unwittingly (or select few recruited to wittingly) manufacture a machine army for it.
- speed_spread 3y ago
  So, Neuromancer meets Terminator? I'd read that.
AdrianB1 3y ago
Denigrating democracy is not bad in any way; first, democracy is the least worst system that we have, that does not make it good. Second, permanent criticism is required for the health of any system that has the potential to go wrong. That being said, autonomous weapons are overall bad, but that explanation is too long for a HN comment.

deleted 3y ago

[deleted]

nonrandomstring 3y ago

> megalomaniacal persons with lust for power and disregard of others I think the problem with these sorts of people is that they are not actually that smart. Cognitively, as all-rounders, they lack crucial faculties of imagination, which not ironically are rooted in compassion, that lead to gross strategic errors. Landmines and impact grenades tied to a balloon are also "autonomous weapons". They are indiscriminate. The former are persistent and the later are ephemeral. The former are also reusable. Within a weapons taxonomy the most dangerous (for yourself) to manufacture are persistent, autonomous and reusable. In other words, like a knife in a fight, there's a very good likelihood your opponent will take it off you and use it against you. Most IEDs used against British and US soldiers are made from UXO. Weapons like drones are a gift to terrorists. For everything you make, one must imagine it being used on a busy city shopping street where your own family are. It's because the megalomaniacs are not actually that smart that they don't realise taking humans out of the loop is the same as relinquishing control, which is one step removed from handing control to the enemy.

MichaelZuo 3y ago
This seems a bit redundant to be honest. Any maniac who is actually super competent would be able to fool you into thinking otherwise. Because they can outwit you by definition. And the inverse also is tautological, of course you can spot those not that competent.
- nonrandomstring 3y ago
  I guess there's two (non-contradictory) conversations going on at once here. My emphasis is more on what we can do, or not do, as participants in a technological society that chooses what technology to make. There is also the conversation on what to do about moustache-twisting, cat stroking, comedy Bond villains with tanks of laser-sharks. I don't think they are fooling any of us. They swagger around in plain sight, naked in their madness, and we could literally write a list of their names here on HN. Of course these strategic and psychiatric questions are related. "Dr. Strangelove" and "Don't look up" remain useful guides.
  - mistermann 3y ago
    > My emphasis is more on what we can do, or not do, as participants in a technological society that chooses what technology to make. The laws of physics do not (or are not known to at least) prevent people from desiring to fix this hilariously suboptimal "democracy" we live under and most everyone seems to hate (well, except when they're gushing about how great it is), but something does. I often wonder if propaganda and culture play a role. > I don't think they are fooling any of us. I believe the opposite - I believe at least 95% of people are adequately fooled (to sustain indefinite disinterest), and most of the remaining 5% are "conspiracy theorists" which we all know can be safely ignored.
- BriggyDwiggs42 3y ago
  I think psycho/sociopath types are a bit like the uncanny valley, where most people can tell something is off without it being obvious what. It happens because being clever isn’t the same as having the absurd level of self control needed to fake normality.
globalnode 3y ago
relying on their "lack of well roundedness" isnt that great a strategy imho
jongjong 3y ago
The thing is it's usually going to be some freakishly weird outlier who will cause the problems. People who act within the realm of normalcy rarely pose an unexpected threat. There exist people who are both megalomaniac and rational and this is precisely the kinds of people who are typically selected for positions of power. However rare such individuals may be, countries like the US have a large pool of candidates to sift through. The people in power are capable of being perfectly rational in every aspect, except for the one aspect where they are required to be irrational. The system is essentially selecting people for their blind-spots and that's why we often see displays of extreme hypocrisy from politicians and other public figureheads. They are 100%, perfectly rational about everything except when it relates to pursuing their own self-interest, yet because they are rational in every other aspects, they know how to hide this irrationality from other people.
- deleted 3y ago
  [deleted]
wolverine876 3y ago
> they are not actually that smart It includes billionaire leaders of the IT industry. Whatever their flaws, it doesn't stop them from developing, building, and scaling technology - beyond anyone else in the world.
- Arainach 3y ago
  It is a mistake in logic to assume that because someone is successful they are talented or deserving. There is a huge luck factor and even larger "pre-existing advantages" factor i.e. rich parents, being born in the right area, etc. This is a fallacy very common in America in particular (though by no means only found there). If you have money, buy a sports team, and the managers of the team turn it into a champion, that does not inherently make you a gifted owner or manager - it means you had the capital to get in.
  - wolverine876 3y ago
    > It is a mistake in logic to assume that because someone is successful they are talented or deserving. There are other very strong indications of their capabilities. I suppose talent doesn't matter, just their ability to create outcomes.
    - trimethylpurine 3y ago
      Which is talent. Almost no one who has won the lottery was wealthy for very long. One exception is Stefan Mandel, a mathematician who bought all the tickets repeatedly. Very much a talent. Retaining a position of "luck" is itself a talent. It's a mistake to assume, but it's not a mistake to make a logical and educated guess that someone successful is likely also talented.
    - nonrandomstring 3y ago
      > just their ability to create outcomes Attila, Alexander, Hitler, Stalin ... those guys got things done. "Outcomes" need to be morally qualified.

deleted 3y ago

[deleted]

roenxi 3y ago

They don't even need to manufacture their own army. The weak point of any society is the military generals - if they try to seize then there are a tiny number of things that might stop them - their own troops, other generals and ... not sure if there is a 3rd thing. The big control in a typical democracy is that the troops of a rogue general will, presumably, disobey them and if the army fractures then the winning side will be the one with more people. Once mechanised armies are in play the numbers and assumptions around that dynamic change.

wolverine876 3y ago
Mature democracies don't have coups. That problem is solved.
- roenxi 3y ago
  You need to go all the way to the end of the thought on this one. "Mature democracies don't have coups because ...". Democracies don't change human nature, there are still powerful people who want to execute coups in any given society.
  - wolverine876 3y ago
    > there are still powerful people who want to execute coups in any given society. There are still far more people in society who don't want a coup, and the desire for a coup isn't binary - democracy changes human behavior, obviously.
- ricudis 3y ago
  Says who?
  - wolverine876 3y ago
    The evidence? Name a mature democracy where there was a coup.
    - 3y ago

bee_rider 3y ago

Plenty of coups have happened, and for most of recorded history, humanity was ruled over by some particularly fancy gangsters: monarchs, dictators, those sorts. Mad engineers could be worse, though. I don’t know. If there’s a good solution I guess somebody would propose it.

maxglute 3y ago

Switching to autonomous platforms might not be as labour saving as you think, especially distributed lethality model that focuses on swarms. It's trading crew infra for even more maintenance infra - there tend to be more ground logistics than operators.

paulddraper 3y ago

> megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army Mercenaries already exist. Commonly. Though in a developed country it's cheaper to buy lobbyists.

wolverine876 3y ago
Again, you can't build a human mercenary military with nearly the capabilities of any country but a small, poor one.

29athrowaway 3y ago

Guess what happens with most civilians when they are obsolete as a source of wealth and power. It does not take much imagination to see that the core skill of politicians, persuasion, will be obsolete as well. There will be no persuasion, no speeches, no campaigns. It will be just simply do as they say or perish. It's the end of democracy and the start of a new form of feudalism, but more violent. If we add brain implants to the mix, the prospect future gets even more grim. If you are important, information will be exfiltrated from your brain without your cooperation.

c_o_n_v_e_x 3y ago

>However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army. I've had a novel (as in writing a book) idea rattling around in my brain with something like this. A distributed AI living throughout the collective internet has humans unwittingly (or select few recruited to wittingly) manufacture a machine army for it.

speed_spread 3y ago
So, Neuromancer meets Terminator? I'd read that.

AdrianB1 3y ago

Denigrating democracy is not bad in any way; first, democracy is the least worst system that we have, that does not make it good. Second, permanent criticism is required for the health of any system that has the potential to go wrong. That being said, autonomous weapons are overall bad, but that explanation is too long for a HN comment.